Release Testing System

The GeoDMS release testing system is a Python-based framework for running automated regression tests against different versions of the GeoDMS software. The testing framework is implemented in the GeoDMS-Test repository and allows developers to verify that new versions don't break existing functionality.

overview

The release testing system consists of:

• batch/full.py - the main test runner script
• profiler/regression.py - helper functions for experiment management (in the main GeoDMS repository)
• profiler/profiler.py - profiling utilities (in the main GeoDMS repository)

The system runs a comprehensive suite of tests including operator tests, storage tests, network model tests, land use model tests, and GUI tests.

adding a new version to the release testing system

To test a new version of the GeoDMS, you need to:

1. install the GeoDMS version

Install the GeoDMS version you want to test. The installer places the executables in:

%ProgramFiles%/ObjectVision/GeoDms{VERSION}

For example, for version 18.0.3:

C:\Program Files\ObjectVision\GeoDms18.0.3

2. run the test script

Navigate to the batch folder in the GeoDMS-Test repository and run the test script with the version parameter:

cd C:\path\to\GeoDMS-Test\batch
python full.py -version 18.0.3

re-running tests for an existing version

To re-run the tests for a version that has already been tested (e.g., after fixing a bug or updating test data):

1. remove the resultsfolder for the geodms test that you would like to rerun. For geodms 19.3.0 this means removing the folder 19_3_0_x64_SF_S1S2S3_OVSRV05 or remove the specific .bin coinciding with the tests that you specifically want to rerun.

2. run the same command again with the same version number:

python full.py -version 18.0.3

3. To preserve previous results, rename or move the existing results folder before re-running:

# Rename previous results
ren "results\18.0.3_x64_S1_S2_S3" "18.0.3_x64_S1_S2_S3_old"
# Then re-run
python full.py -version 18.0.3

optional parameters

The test script accepts several optional parameters for controlling multithreading behavior:

Parameter	Description	Default	Options
-version	GeoDMS version number	18.0.3	Any installed version (e.g., 17.4.6, 18.0.0)
-MT1	Multithreading setting 1	S1	S1 (single) or C1 (concurrent)
-MT2	Multithreading setting 2	S2	S2 (single) or C2 (concurrent)
-MT3	Multithreading setting 3	S3	S3 (single) or C3 (concurrent)

Example with all parameters:

python full.py -version 18.0.3 -MT1 C1 -MT2 C2 -MT3 C3

how the version lookup works

The get_geodms_paths function in full.py constructs the paths to the GeoDMS executables based on the version string:

def get_geodms_paths(version:str) -> dict:
    geodms_paths = {}
    geodms_paths["GeoDmsPlatform"] = "x64"
    geodms_paths["GeoDmsPath"] = f"\"{os.path.expandvars(
        f"%ProgramFiles%/ObjectVision/GeoDms{version}")}\""
    geodms_paths["GeoDmsRunPath"] = f"{geodms_paths["GeoDmsPath"]}/GeoDmsRun.exe"
    geodms_paths["GeoDmsGuiQtPath"] = f"{geodms_paths["GeoDmsPath"]}/GeoDmsGuiQt.exe"
    return geodms_paths

The function expects the version string to match the folder name created by the GeoDMS installer (e.g., "18.0.3" for GeoDms18.0.3).

prerequisites

Before running the tests, ensure:

1. Python 3.x is installed and available in your PATH

2. GeoDMS version is installed via the official installer

3. Source data directories are configured in get_local_machine_parameters():

   • GEODMS_OVERRIDABLE_RegressionTestsSourceDataDir - location of regression test data
   • GEODMS_DIRECTORIES_LOCALDATADIR - local data directory for test outputs

4. Project snapshots are available in the prj_snapshots subdirectory

configuring local machine parameters

Edit the get_local_machine_parameters() function in full.py to match your system:

def get_local_machine_parameters() -> dict:
    local_machine_parameters = {}
    local_machine_parameters["GEODMS_OVERRIDABLE_RegressionTestsSourceDataDir"] = "C:/SourceData/RegressionTests"
    local_machine_parameters["RegressionTestsAltSourceDataDir"] = "D:/SourceData"
    local_machine_parameters["GEODMS_DIRECTORIES_LOCALDATADIR"] = "C:/LocalData"
    local_machine_parameters["LocalDataDirRegression"] = f"{local_machine_parameters['GEODMS_DIRECTORIES_LOCALDATADIR']}/regression"
    local_machine_parameters["tmpFileDir"] = f"{local_machine_parameters['LocalDataDirRegression']}/log"
    return local_machine_parameters

test results

Test results are stored in a results folder with a naming convention:

{version}_{platform}_{MT1}_{MT2}_{MT3}

For example: 18.0.3_x64_S1_S2_S3

Results include:

• Log files for each test
• Indicator result files comparing expected vs actual values
• File comparison results for storage tests

available tests

The full regression test suite includes:

Test ID	Description
t010	Operator test - tests all GeoDMS operators
t050	Storage write (Shapefile polygon)
t060	BAG Snapshot (GeoPackage)
t100	Network connect
t101	Network OD PC4 dense
t102	Network OD PC6 sparse
t151	BL-RD coordinate conversion
t200	Grid Poly2Grid
t300	XML read/parse
t301	BAG residential type
t405	NetworkModel PBL (prepare, run, indicator)
t410	NetworkModel EU
t611	LUS demo 2023
t641	RSopen regression test
t710	2UP model
t720	2BURP model
t810	ValLuisa Czech
t910	CUSA2 Africa
t1630	GUI expand test
t1640	Value info test
t1642	Value info group_by
t1742	Command statistics

adding new tests

To add a new test to the regression suite, add an entry in the get_experiments() function:

regression.add_exp(
    exps, 
    name=f"{result_folder_name}__t{NUMBER}_{TEST_NAME}",
    cmd=f"{geodms_paths['GeoDmsRunPath']} /L{result_paths['results_log_folder']}/{TEST_NAME}.txt /{MT1} /{MT2} /{MT3} {path_to_dms} {item_to_calculate}",
    exp_fldr=f"{result_paths['results_folder']}",
    env=env_vars,
    log_fn=f"{result_paths['results_log_folder']}/{TEST_NAME}.txt",
    indicator_results_file=f"{result_paths['results_folder']}/{TEST_NAME}.txt"
)

troubleshooting

version not found

If you get an error that the version is not found, verify:

1. The GeoDMS is installed in the expected location
2. The version string matches the folder name exactly (e.g., "18.0.3" not "v18.0.3")

missing source data

Ensure the regression test source data is available at the configured paths. Contact the GeoDMS team for access to the test data repository.

permission errors

Run the test script with administrator privileges if you encounter permission issues writing to the output directories.

see also

• Debugging - debugging the GeoDMS
• Compiling the GeoDMS - building from source
• Unit testing - Boost test framework for unit tests
• GeoDMS-Test repository - test repository on GitHub