Test Suite Requirements updates from Face to Face (#1115)

Co-authored-by: David Harris <74973295+davidharrishmc@users.noreply.github.com>

Author: jordancarlin

docs/ctp/src/req.adoc

@@ -4,45 +4,48 @@
 The test suites may be provided by multiple suppliers, but must meet a common set of requirements to interoperate transparently for the user:
 
 * All tests use a common set of DUT configuration files so the user only has to write them once:
-** <dut>.yaml for the UDB configuration including supported extensions and parameter values
-** rvmodel_macros.h for the RVMODEL DUT-Specific Macros
-** link.ld for the linker script
+** `<dut>.yaml` for the UDB configuration including supported extensions and parameter values
+** `rvmodel_macros.h` for the RVMODEL DUT-Specific Macros
+** `link.ld` for the linker script
 
 * All tests are self-checking
 ** Able to run on simulators, RTL, and hardware
 ** Expected values may be produced by the RISC-V Sail reference model or the test code itself, and must account for relevant parameters
 *** Alternate reference model may be used for development until RISC-V Sail is sufficiently mature
 ** Only test features whose behavior is specified for the given parameters (e.g. only test unimplemented instructions if Ssstrict is supported)
-** Calls RVMODEL_BOOT for DUT-specific boot code before starting the test
+** Calls `RVMODEL_BOOT` for DUT-specific boot code before starting the test
 ** Test begins in machine mode, if any privileged modes are implemented
-** Terminate with RVMODEL_HALT_PASS or RVMODEL_HALT_FAIL to indicate success
-** Print meaningful information to localize a failure with RVMODEL_IO_WRITE_STR, including:
+** Terminate with `RVMODEL_HALT_PASS` or `RVMODEL_HALT_FAIL` to indicate success
+** Print meaningful information to localize a failure with `RVMODEL_IO_WRITE_STR`, including:
 *** PC at failure point
 *** Expected and actual values of mismatching registers
 *** Name of coverpoint and bin that failed, if practical
 ** Tests should not contain loops over multiple bins.  Each bin of each coverpoint should be tested by a unique instruction so that a failing bin can be uniquely identified from the program counter.  Loops should go in the test generator instead of the test itself.
 
-* All tests contain test configuration information (YAML comments) so the framework can identify which ones to run on a DUT
-** List of required extensions and parameter values that the DUT must support to run the test
-** Whether the expected result is configuration-dependent or not (i.e. whether it needs to be regenerated for different configurations or can be run unmodified on any configuration that meets the requirements)
-** Tests are organized in a highly granular way so that each ISA feature can be tested independently of other features
+* Tests are organized in a highly granular way so that each ISA feature can be tested independently of other features
 
-* All tests are written in assembly or C and work with both GCC and Clang
+* All tests are written in assembly or C and work with the latest release of both GCC and Clang
+** The latest release is typically needed because new extensions are rapidly being added to these compilers
+** Certificates will be tied to specific compiler versions to ensure consistency
 
-* Each test runs on the RISC-V Sail reference model and Spike for at least one choice of parameters
+* Each test runs on the RISC-V Sail reference model, Spike, and Whisper for at least one choice of parameters
+** Exceptions can be made for specific tests when there are known bugs or unimplemented extensions
 
 * Tests are for certification, not verification
 ** Every normative rule is addressed by at least one coverpoint, defined in the CTP
-** Every test is directed to a specific coverpoint bin, not fuzzing in the hopes of randomly hitting a coverpoint
+*** Specific normative rules can be omitted from testing if they are not testable. In this case, the rule should still be listed in the CTP and should be explicitly marked as not tested
+** Every test is directed to a specific normative rule/coverpoint bin, not fuzzing in the hopes of randomly hitting a rule/coverpoint
 *** Exception: Multicore memory ordering is necessarily nondeterministic
 ** Tests emphasize RISC-V architectural behavior, not microarchitectural behavior such as complex pipeline hazards or timing
 ** Runtime is important
+** Size is important
 
 * All tests have SystemVerilog coverpoints
 ** Use standard syntax compatible with Questa, VCS and Xcelium (regression on at least Questa)
+** All coverpoints must use the RVVI interface
 
-* All testing collateral (tests, test generators, coverpoints, frameworks, documentation, etc.) are owned by RVI and distributed under an open-source license
-** Hosted in the https://github.com/riscv/riscv-arch-test[riscv/riscv-arch-test] GitHub repository under the stewardship of Architectural Certification Test (ACT) SIG
+* All testing collateral (tests, test generators, coverpoints, frameworks, documentation, etc.) are owned by RVI and distributed under an open-source license ** TODO: Jeff/Tom/Moiz to update this section
+** Hosted in the RISC-V architectural test repository under the stewardship of Architectural Certification Test (ACT) SIG
 ** Repository contains tests for ratified and in-development features, not just features in certificates
 ** Generators necessary to recreate the tests for a DUT are included in the repository, but, when tests are not unique to a DUT, the generated tests themselves are also included to reduce requiring users to run generators and to make the tests easier to view and understand
 
@@ -53,7 +56,5 @@ The test suites may be provided by multiple suppliers, but must meet a common se
 
 === For discussion:
 
-* Should RISC-V Sail be the only reference model used for results, so RVI only has to maintain one reference model?  Or is support for Whisper also a requirement?
-* Should all coverage be collected via RVVI and SystemVerilog coverpoints, so RVI only has to maintain one coverage flow?
 * Testing of WARL, WLRL, WPRI fields, and the necessary parameters to define what fields should be tested and behaviors should be expected
 * Regression requirements, including multiple parameters