Clarify testcase vs test chunk (#1112)

Two distinct concepts were both being called testcases. This clarifies
the terminology and lays out a clear hierarchy:

```
test suite
└── test file
    └── test chunk
        └── testcase
```

Author: jordancarlin

docs/DeveloperGuide.md

@@ -8,6 +8,7 @@ Each is described below.
 ## Table of Contents
 
 - [Certification Test Plan](#certification-test-plan)
+- [Test Hierarchy](#test-hierarchy)
 - [Table-Driven Unprivileged Coverpoints and Tests](#table-driven-unprivileged-coverpoints-and-tests)
   - [Creating New CSV Testplans](#creating-new-csv-testplans)
   - [Adding New Coverpoints](#adding-new-coverpoints)
@@ -59,6 +60,25 @@ The `generate_param_table.py` script turns these into .adoc files in ctp/src/par
 
 This script is also run automatically when making the CTP. Hence, all the developer must do is create YAML files in `coverpoints/param` for test suites with parameters.
 
+## Test Hierarchy
+
+The testgen package organizes generated tests into four levels:
+
+```
+test suite
+└── test file
+    └── test chunk
+        └── testcase
+```
+
+- **Testcase**: The smallest unit of testing. Each testcase checks a single bin of a coverpoint. In the generated assembly, a testcase corresponds to one call to `test_data.add_testcase()`, which creates a label and debug string for that specific bin. For example, testing that `add` writes to `x5` is one testcase of the `cp_rd` coverpoint.
+
+- **Test chunk** (`TestChunk`): An unsplittable group of one or more testcases. A test chunk is the building block of test files. Test chunks are never split across multiple files. Standard coverpoint generators (e.g., `cp_rd`, `cp_imm_edges`) produce one chunk per testcase via `format_single_testcase()`. Special coverpoint generators and privileged tests bundle multiple testcases into a single chunk using `test_data.begin_test_chunk()` / `test_data.end_test_chunk()`, typically because the testcases share setup code.
+
+- **Test file**: A complete `.S` assembly file that is compiled into a self-checking ELF. Each test file contains one or more test chunks. When an instruction has many testcases (e.g., hundreds of register/immediate combinations), the framework splits the chunks across multiple test files using `TESTCASES_PER_FILE` as the limit. Test files are named like `I-add-00.S`, `I-add-01.S`, etc., where the suffix indicates the file index.
+
+- **Test suite**: All test files in a given directory. Each test suite corresponds to one extension or combination of extensions (e.g., `I`, `Zcb`, `ZcbZbb`, `ExceptionsSm`) and maps to a single coverage file. Unprivileged test suites contain one or more test files per instruction. For privileged tests, a test suite typically contains a single test file covering all coverpoints for that feature.
+
 ## Table-Driven Unprivileged Coverpoints and Tests
 
 Unprivileged tests are tests that exercise individual instructions and do not trap.
@@ -137,14 +157,14 @@ The following applies to all coverpoint test generators:
 - All coverpoint generator functions must use the following signature:
 
   ```py
-  def make_cp_name(instr_name: str, instr_type: str, coverpoint: str, test_data: TestData) -> list[TestCase]:
+  def make_cp_name(instr_name: str, instr_type: str, coverpoint: str, test_data: TestData) -> list[TestChunk]:
   ```
 
   - `instr_name` is the instruction currently being tested. This allows coverpoint test generators to be reused for multiple instructions.
   - `instr_type` is the type of the instruction currently being tested. This allows the correct instruction formatter (see below) to be selected.
   - `coverpoint` is the full name of the coverpoint, including any variant suffix. Coverpoint test generators can match multiple variants of a coverpoint. This argument allows different values, registers, etc. to be selected based on the variant.
-  - `test_data` is the generation context that is passed to all parts of the test generation process and manages register allocation, test counting, and the active `TestCase`.
-  - The generator must return a list of `TestCase` objects. Each `TestCase` holds its own assembly code, data values, debug strings, and signature update count. The framework uses these to split tests across files and combine them into the final output.
+  - `test_data` is the generation context that is passed to all parts of the test generation process and manages register allocation, test counting, and the active `TestChunk`.
+  - The generator must return a list of `TestChunk` objects. Each `TestChunk` is an unsplittable group of one or more testcases. It holds its own assembly code, data values, debug strings, and signature update count. The framework uses these to split test chunks across test files and combine their data for the final output.
 
 Coverpoint test generators can largely be broken into two categories: standard and special.
 Standard generators use the [instruction formatters](#python-instruction-formatters) and can be applied to a wide range
@@ -166,7 +186,7 @@ It is also included below with many additional comments added to explain how it
 # which coverpoints they apply to.
 @add_coverpoint_generator("cp_rd")
 # Coverpoint generators all use the standard signature described above.
-def make_rd(instr_name: str, instr_type: str, coverpoint: str, test_data: TestData) -> list[TestCase]:
+def make_rd(instr_name: str, instr_type: str, coverpoint: str, test_data: TestData) -> list[TestChunk]:
     """Generate tests for destination register coverpoints."""
     # Determine which rd registers to test based on the coverpoint variant.
     # Multiple variants can match to the same generator. This is useful when
@@ -182,8 +202,8 @@ def make_rd(instr_name: str, instr_type: str, coverpoint: str, test_data: TestDa
         # to make debugging easy.
         raise ValueError(f"Unknown cp_rd coverpoint variant: {coverpoint} for {instr_name}")
 
-    # Initialize a list of TestCase objects to collect results
-    test_cases: list[TestCase] = []
+    # Initialize a list of TestChunk objects to collect results
+    test_chunks: list[TestChunk] = []
 
     # Generate tests
     # A common pattern is to use a loop to iterate over some value that is being tested
@@ -200,22 +220,22 @@ def make_rd(instr_name: str, instr_type: str, coverpoint: str, test_data: TestDa
         # will get random values.
         params = generate_random_params(test_data, instr_type, rd=rd)
         desc = f"{coverpoint} (Test destination rd = x{rd})"
-        # format_single_test is the key part of standard coverpoint generators. It takes
-        # the provided instruction parameters (created above) and produces a TestCase object
+        # format_single_testcase is the key part of standard coverpoint generators. It takes
+        # the provided instruction parameters (created above) and produces a TestChunk object
         # containing the assembly code and associated data. It also calls test_data.add_testcase
         # to add a label and debugging string.
-        tc = format_single_test(instr_name, instr_type, test_data, params, desc, f"b{rd}", coverpoint)
-        # If consume_registers returned setup code (register moves), prepend it to the TestCase
+        tc = format_single_testcase(instr_name, instr_type, test_data, params, desc, f"b{rd}", coverpoint)
+        # If consume_registers returned setup code (register moves), prepend it to the TestChunk
         if asm_setup:
             tc.code = asm_setup + "\n" + tc.code
-        test_cases.append(tc)
+        test_chunks.append(tc)
         # Once registers are no longer in use, they need to be marked as available again
         # so that the register allocator knows that they can be reused.
         return_test_regs(test_data, params)
 
-    # Return the list of TestCase objects. The framework will use these to split tests
-    # across files (based on num_tests counts) and combine their data for the final output.
-    return test_cases
+    # Return the list of TestChunk objects. The framework will use these to split test chunks
+    # across test files (based on num_testcases counts) and combine their data for the final output.
+    return test_chunks
 ```
 
 Additional documentation for all of these functions (and many other helper functions) is
@@ -232,16 +252,16 @@ cases each individual instruction).
 
 Special coverpoint generators vary widely, so it is impossible to provide a complete guide,
 but they usually follow the same initial flow as a standard coverpoint and then diverge
-where the call to `format_single_test` would be. Instead of calling `format_single_test`,
-special coverpoint generators use `test_data.begin_testcase()` and `test_data.end_testcase()`
-to wrap their inline assembly in a single `TestCase`. The typical pattern is:
+where the call to `format_single_testcase` would be. Instead of calling `format_single_testcase`,
+special coverpoint generators use `test_data.begin_test_chunk()` and `test_data.end_test_chunk()`
+to wrap their inline assembly in a single `TestChunk`. The typical pattern is:
 
 ```py
-tc = test_data.begin_testcase()
+tc = test_data.begin_test_chunk()
 test_lines: list[str] = []
 # ... build assembly lines, call test_data.add_testcase(), load_int_reg(), write_sigupd(), etc. ...
 tc.code = "\n".join(test_lines)
-return [test_data.end_testcase()]
+return [test_data.end_test_chunk()]
 ```
 
 While most of this code is handwritten, you are still encouraged to use helper Python

generators/testgen/src/testgen/asm/csr.py

@@ -27,8 +27,8 @@ def gen_csr_read_sigupd(check_reg: int, csr_name: str, test_data: TestData) -> s
     Returns:
         Assembly line for the CSR read SIGUPD
     """
-    assert test_data.testcase is not None, "No active testcase — call begin_testcase() first"
-    test_data.testcase.sigupd_count += 1
+    assert test_data.test_chunk is not None, "No active test chunk — call begin_test_chunk() first"
+    test_data.test_chunk.sigupd_count += 1
     return (
         f"{INDENT}# Read {csr_name} into x{check_reg} and check against expected.\n"
         f"RVTEST_SIGUPD_CSR_READ({csr_name}, x{check_reg}, {test_data.current_testcase_label}, {test_data.current_testcase_label}_str)"
@@ -50,8 +50,8 @@ def gen_csr_write_sigupd(check_reg: int, csr_name: str, test_data: TestData) ->
     Returns:
         Assembly line for the CSR write SIGUPD
     """
-    assert test_data.testcase is not None, "No active testcase — call begin_testcase() first"
-    test_data.testcase.sigupd_count += 1
+    assert test_data.test_chunk is not None, "No active test chunk — call begin_test_chunk() first"
+    test_data.test_chunk.sigupd_count += 1
     return (
         f"{INDENT}# Write x{check_reg} to {csr_name}, read back and check against expected.\n"
         f"RVTEST_SIGUPD_CSR_WRITE({csr_name}, x{check_reg}, {test_data.current_testcase_label}, {test_data.current_testcase_label}_str)"

generators/testgen/src/testgen/asm/helpers.py

@@ -54,8 +54,8 @@ def to_hex(value: int, bits: int) -> str:
 
 def load_int_reg(name: str, reg: int, val: int, test_data: TestData) -> str:
     """Generate assembly to load an integer register with a specific value."""
-    assert test_data.testcase is not None, "No active testcase — call begin_testcase() first"
-    test_data.testcase.data_values.append(val)
+    assert test_data.test_chunk is not None, "No active test chunk — call begin_test_chunk() first"
+    test_data.test_chunk.data_values.append(val)
     return f"{INDENT}RVTEST_TESTDATA_LOAD_INT(x{test_data.int_regs.data_reg}, x{reg}) # load {name}: x{reg} = {to_hex(val, test_data.xlen)}"
 
 
@@ -70,32 +70,32 @@ def load_float_reg(
     if fp_load_type is None:
         fp_load_type = test_data.fp_load_size
 
-    assert test_data.testcase is not None, "No active testcase — call begin_testcase() first"
-    test_data.testcase.data_values.append(val)
+    assert test_data.test_chunk is not None, "No active test chunk — call begin_test_chunk() first"
+    test_data.test_chunk.data_values.append(val)
     return f"{INDENT}RVTEST_TESTDATA_LOAD_FLOAT_{fp_load_type.upper()}(x{test_data.int_regs.data_reg}, f{reg}) # load {name}: f{reg} = {to_hex(val, test_data.flen)}"
 
 
 def write_sigupd(check_reg: int, test_data: TestData, sig_type: Literal["int", "float"] = "int") -> str:
     """
     Generate assembly for SIGUPD and increment sigupd_count.
     """
-    assert test_data.testcase is not None, "No active testcase — call begin_testcase() first"
+    assert test_data.test_chunk is not None, "No active test chunk — call begin_test_chunk() first"
     sig_reg = test_data.int_regs.sig_reg
     link_reg = test_data.int_regs.link_reg
     temp_reg = test_data.int_regs.temp_reg
     fp_temp_reg = test_data.float_regs.temp_reg
     if sig_type == "int":
-        test_data.testcase.sigupd_count += 1
+        test_data.test_chunk.sigupd_count += 1
         return (
             f"{INDENT}# Check if x{check_reg} contains the expected result. x{sig_reg} is the signature ptr, "
             f"x{link_reg} is the link ptr, x{temp_reg} is a temp reg.\n"
             f"{INDENT}RVTEST_SIGUPD(x{sig_reg}, x{link_reg}, x{temp_reg}, x{check_reg}, {test_data.current_testcase_label}, {test_data.current_testcase_label}_str)"
         )
     elif sig_type == "float":
         if test_data.flen > test_data.xlen:
-            test_data.testcase.sigupd_count += 3
+            test_data.test_chunk.sigupd_count += 3
         else:
-            test_data.testcase.sigupd_count += 2
+            test_data.test_chunk.sigupd_count += 2
         return (
             f"{INDENT}# Check if f{check_reg} contains the expected result. Also checks fflags. "
             f"x{sig_reg} is the signature ptr, x{link_reg} is the link ptr, x{temp_reg} "

generators/testgen/src/testgen/constants.py

@@ -26,9 +26,8 @@ def indent_asm(line: str) -> str:
 # Test Generation Configuration
 # =============================================================================
 
-# Max testcases per file before splitting. Individual coverpoints won't be split,
-# so if one coverpoint exceeds this, the file will exceed this limit.
-# TODO: Currently only applies to unpriv tests. Should this apply to priv tests too?
+# Max testcases per test file before splitting into multiple files. Individual test
+# chunks won't be split, so if one test chunk exceeds this, the file will exceed this limit.
 TESTCASES_PER_FILE = 1000
 
 # =============================================================================