While I was looking for ways to test for the storage errors, I noticed that the Hashicorp plugins tend to use "BATs" for acceptance testing. That way they are able to spin up docker containers as needed (artifactory) and use the vault binary directly instead of interfacing at the "Backend" layer in code, which feels like more like unit testing (unit tests with a real artifactory server).
I am not saying we can't keep the go-based acceptance tests, but it feels kinda strange to have direct code access when doing acceptance tests. Acceptance Tests are supposed to be "black-box" or completely separate from the code. We can write them in golang, but they need to be in a separate directory (package).
Since the "normal" user interface is the vault binary, it feels like a shell based testing system would be easier.
Thoughts?
While I was looking for ways to test for the storage errors, I noticed that the Hashicorp plugins tend to use "BATs" for acceptance testing. That way they are able to spin up docker containers as needed (artifactory) and use the vault binary directly instead of interfacing at the "Backend" layer in code, which feels like more like unit testing (unit tests with a real artifactory server).
I am not saying we can't keep the go-based acceptance tests, but it feels kinda strange to have direct code access when doing acceptance tests. Acceptance Tests are supposed to be "black-box" or completely separate from the code. We can write them in golang, but they need to be in a separate directory (package).
Since the "normal" user interface is the vault binary, it feels like a shell based testing system would be easier.
Thoughts?