It started 1.5 year ago when the OpenID crowed wanted a test suite for OpenID Connect.
The problem was that interop tests was done after the implementation was done. It needed to be done earlier in the development phases.
The task was picked up by Roland and Andreas.
The architecture:
- Built in Python
- Each test is implemented in a class and many classes is combined in a Conversation.
- User interface is web based
- It's script based and works from command line.
- Front-end is just calling the scripts and logs and results are returned.
- It could even be used as "unit tests" when doing your own implementations.
Now it's time to extend the framework for SAML and OAuth2.
A lot of talk with Rainer when it comes to SAML. Hannes and Derek is involved when it comes to OAuth2.
When it comes to OAuth2 there are a lot of flows but the implementation should start with the code flow first, then add the other flows later on.
When it comes to SAML this meeting is meant to be a discussion around that.
So, most trust frameworks uses Redirect bindings and POST bindings.
When it comes to logout request, SOAP bindings it mostly used but there also exists redirect and POST bindings.
Implementation should first focus on the normal tests. Then add the attributes profiles that could be added.
Implementation should be done using some shared repository where more people can add tests for different federations.
In SAML it's both the software and configuration that needs to be tested.
How should the test implements things that's not defined in the specification? Today it's just assumed and there exists common consensus on how to implement it but it's not written down.
The test should be a cloud service. You upload metadata and add some config and you can then run the tests.
The test suite should be open source.
The test data (the results) could be handled by some organization.
We should use the tests already made and extend them for new tests.
In the current implementation you can filter the requests and responses so that you can insert things to invalidate signatures and so on. Test should start with the normal flows and do test according to tests. Then starting to break the system.
There is still a need to be a bit flexible when doing the tests, most of the test servers probably have a self sign certificate so the test system should not break because of that, maybe just warn.
When testing IDP:s the current implementation records a username and passwords in the beginning of the tests and uses that in later tests. We might have to figure out a way to do long term tests. A fixed Donald Duck user?
What's should be tested:
Redirect POST SOAP ECP
-----------------------
AuthnReq yes yes no
SLO yes no yes
Attrib requests no no yes
NIDM no no no
AuthQ no no no
There is other dimensions in the tests. Do all methods but with some invalid, corrupted or manipulated data.
It would be nice that in the future it can be a cloud based test suite that can be integrated into own unit tests.
We should try to get a list of attacks from attackers and get them to run the tests.
