Conveners: Roland Hedberg, Rainer Hörbe
Abstract: Continuing from the yesterday's session, the topic is covering the testing tools used to check for errors in federations and pointing exactly where the error is. Discussion on how the tools could be improved and what would they be mostly used for, based on the experience of individual identity managers.
The proposal on the table is to create database
"I am not sure it will be more than one tool. A documentation mapping tool."
Putting this into the GitHub allows you to add that. These documents that are on GitHub must follow a structure.
We are planning to do some JavaScript coding to make the keys that will be used.
Do you think it’s versatile to have it in the GitHub?
I was thinking of doing some kind of a GitHub database, a JavaScript framework pulling data via GitHub API. You are losing the capability of Git-repos, it is not much more then a web application.
I think starting with the document in Github is good and if it becomes too complicated, it still can be moved to something else.
It's not that big so it's impractical to make/do?
It won’t get too big.
The purpose of the session:
To be more specific how are you going to use the test tool? What would you like to achieve with the test tool?
If I am choosing a product I would like to run tests against?
For the purpose of this session what I would really like to have you contribute to this, and to have some kind of mailing list to help us improve this.
To be more specific how are you going to use the test tool? What would you like to achieve with the test tool?
Jim: I would like to run the tests against the product that I am evaluating.
Nick: Customer testing the capability of the service, e.g. an ADFS implementation. When it is passing these 6 tasks it's okay, but until then it would be put on hold. I would pick some test cases that are absolutely crucial.
At the federation level I’d like to use this to a customer or another party testing the capability of their peer.
Jim: Internet 2 has a reputation for providing quality product and that’s what the members are expecting.
Peter: I think it’s better to have interfaces where the handling can be done because there are options which are very capable and on the other hand everyone who make technical work can make some workflow systems, and you don’t want to make too many of them but one of them if possible.
Nick: A simple way to do that is to have configuration time parameters available because a lot of those workflow systems can accept email.
Rainer: E-mail based system would be the only option. . It was never my inner picture of a lot of people doing the testing and we always saw one person executing the test. When you want to change the configuration, to make another round, it would be nice if it was automatic. We talked about this tool where everything is automatic if you got it right. That’s a really good point where some lightweight workflow might be handy because it would easily update my metadata to the federation.
Jim: Seems to me a workflow system is different than tests and workflow tests exist.
Rainer: Yes I would say kick off the next workflow step whatever it is.
Peter: If we have it in a productive federation it would be interesting to have it productive when you want to make tests, it would be really interesting thing but before something like that can be done it has to be included in the productive federations.
Roland: In a production environment it’s very crucial to do these tests continuously. You can do all the tests you want. They will eventually break something because of changing
Peter: We need different environments, one for entering the federation and one who is already in the federation.
I was thinking of something about the logging service, in the legacy protocol we issue the ticket numbers, to login where the ticket is issued and the same mechanism can be applied. A test account for one time usage with a short timeout so you could pass all of your tests, including attribute release.
Nick: That brings up a really interesting point, are people starting to employ stronger authentication systems. A lot of that functional stuff has to be tested to function before it goes into production.
Roland: I also like the idea that the user might find something that is important and that the user can use the test and when the IDP would send a report back.
Roland: It will also evade arguments in between the IDP and GP where we would know exactly who is wrong and not let them have space to argue in between each other. The worst thing is when someone sees the logo and that there is no reliable party to accept. We have that experience where the service is provided when somewhere made the SP into the federation. We have this discussion in between the SP and the IDP yes.
Peter: There is also a lot of work here included to find out where is the problem itself, and if there is a tool that would be amazing.
Nick: One thing that's interesting about the open ID foundation is that it requires people to pass their test. There is no SAML foundation where you need to pass a test. There is no similar thing to SAML.
Rainer: There is Kantara and where the Liberty alliance that has vendors which have quite basic tests, and even they have problems to endorse these tests.
Basically the idea is that he is working and making tests and there is an interest to do those tests but that must be a different business model because this vendor rubber stamping isn’t working anymore so we strongly said we need to do these tests open source and it must be freely available.
For the use cases where somebody needs a certification like the US government, then someone will need a tender and you can get a certification.
Nick: Shal was really wanting us to invent this inside internet2, and I said Roland is doing this so it is certifiable to say this vendor has a comment for inoperability. Hey that means that you should go to this guy.
Rainer: I think if we don’t cooperate in this field we won’t achieve anything. If it is not complete, it is useless.
Peter: I think the IDP testing is a very difficult topic.
Nick: The ad-hoc tests something like this is fantastic. There is a specific pattern in the password or the user ID in which every library SP can rule out as having any kind of access as having echo assertions and something like that.
One example is where you would want to test the access control with the libraries and you have an edgy person that is supposed to grant you access to a journal.
Rainer: It would be certainly better. One of the tests that we made is the attributes as XML and it goes beyond the SP to the application, and the application can block specific user IDs from doing everything. It would be good if it would be a bit more general and not specific.
