Nice to see someone caring for this kind of test as well (#2716). I started a similar project a couple of months ago when I was learning some Rust https://github.com/frankmorgner/pkcs11-test. Unfortunately, I never found the time to finalize my project (remove hard coded paths, add some documentation, beautify the code with more rust magic), which is the reason for this sitting around in a private repository until now. My code is fully functional, meaning that it is capable of running and performing all the mandatory tests from the standard.

As my code uses the rust module cryptoki, it cannot replicate the very exact sequence of a few PKCS#11 calls (e.g. regarding the C_FindObject*). That could only be done when handling the PKCS#11 call backs in unsafe code blocks, which I wanted to avoid in my learning project. Looking back to this now, I think this was a good decision as it heavily simplifies the coding and still allows most tests to be executed exactly as they are intended. This limitation would be one advantage of implementing this in pure C rather than in rust.

From an implementation point of view, some trickier things I found where the following:

1. implement the support for variable input/output, which are expressed with shell-variable-like expressions in the XML definition is not trivial. As far as I see, your code does not support this, right? I implemented this using a hash map with the variable name as key
   https://github.com/frankmorgner/pkcs11-test/blob/591a58ba24e3713aadccfb11c2d88bdd1a96503c/src/main.rs#L114
2. Also, when checking if a test succeeds, you currently only look at the return value rather than the returned data. In my implementation, I found that checking the actual data sometimes requires to be a little more relaxed to reflect the behavior that is propably intended (reflected by the STRICT setting in my code).
3. There is an obvious error in the official test cases, did you stumble upon this as well?
   https://github.com/frankmorgner/pkcs11-test/blob/591a58ba24e3713aadccfb11c2d88bdd1a96503c/src/test-cases/pkcs11-v3.1/mandatory/BL-M-1-31.xml#L67
4. What is still unknown to me is how it is possible to select the testing profile in OpenSC or any other PKCS#11 module.

From an organizational point of view, I think it would be good to have a testing tool in src/tools rather than to hide it in an internal (and not installed) folder for tests (if the tool is capable of handling other modules as well). I will keep my implementation as seperated project for this reason.

Regarding to the tricky parts:

1. the variable input/output is handled by the code, but using quite simple implementation storing linked list of string names and values. When parsing the input parameters for the symbolic value, it should try to find the stored value, when parsing and testing output values, it either tries to find the stored values and store the returned ones, if specified.
2. there is checking both for return values and returned data. The data are parsed and checked against values from the XML, stored for later use if needed.
3. no yet really, I've focused more on the proper parsing and mapping the XML to the structures

The subsequent calls to same functions (as C_FindObject) is somehow there, nevertheless, it would need some more love for proper checking that the sequence is actually correct one.

Hi @frankmorgner, I'm returning to this PR again as I want to see the testing tool run in OpenSC CI. From a quick look at the implementation in Rust, it's probably cleaner than the pure C implementation, primarily because of the XML parsing part, which is quite heavy in C. But on the other side, this processing in C brings some flexibility, as it allows to call the PKCS#11 functions out of the sequence order. The intention of the code in this PR is to be flexible, so it should work not only on the official test cases but also on custom test cases.

As I'm not skilled in Rust, I'm curious to know how much work would need to be done in https://github.com/frankmorgner/pkcs11-test to be finalized, but I would probably be less of a help there. However, I would like to set up a virtual environment to run the tool, which would be useful in CI. What do you think about the Rust vs C implementation? As I would like to make the code in this PR work, I don't want to work unnecessarily on something already implemented and usable, so I could at least help with some integration with the Rust project.

I'm not skilled in Rust, too. I used this project to learn, but I agree that Rust's infrastructure offers many simplifications. However, you may have noticed that my project is using a PKCS#11 wrapping library to do the heavy lifting of handling unsave c code (the testee) mapped to rust. The drawback is that I don't always have full control over the exact flow of calls (in particular of functions that are split up in multiple commands C_*Init, C_*Final), but for testing purposes, I think, this would be an acceptable limitation.

A C implementation inside OpenSC, on the other hand, would make the testing self contained and it would allow exact control of the test cases. Due to the high complexity and the limited applicability, I believe that not many developers would notice this tool, let alone using this tool elsewhere. At the very least, I would recommend extracting pkcs11test to a seperate project to make it available (and visible) to other PKCS#11 providers. (Maybe this step could also be done for p11test, but my understanding is that p11test is not a general purpose tool, but instead relies very heavily on the OpenSC implementation of the supported cards.)

My recommendation would be a) create a seperate project (possibly in the organization OpenSC). For using this inside OpenSC CI, just use the tool that you think is most appropriate.

Item 4 from above is still unsolved: I don't know how to switch to some particular (testing) profile via PKCS#11 interface. This is needed, to, for example, change some static labels to match the test case requirements.