Outsource benchmark scenarios to separate repos

[berndflemisch]

In the meeting 2026-04-30, we decided to split up repositories. From the minutes:

Repository structure

We agreed that in the midterm we aim for a single repository with a benchmark definition and then individual repositories for each implementation of a benchmark (if people want to publish their implementation)

Main repo (current github repo)

• Includes the main transformation scripts (e.g. developed by Mahdi to get an ROCrate out of the subcrates

• Examples on how to build these

• Registry for all Benchmark repositories

• Potentially a documentation (SWIG*) routine that collects the documentation files from all repos and builds a joint summary documentation

Benchmark repo

• For each benchmark, we create a separate repo

• It includes the semantic representation of the benchmark, a documentation (markdown) and a postprocessing script the contains potential queries to the knowledge graph and plots that for relevant metrices (e.g. convergence plot for different metrics as in plate with a hole)

• There is a version, and the benchmark can be published. Each published version (e.g. in the github actions) builds an ROcrate for the benchmark definition that is uploaded to ROHub

• While the package is build, a the documentation and the postprocessing scripts are combined into a jupyter notebook, such that a user can download the notebook and visually explore the results of the benchmark obtain for different software implementations

Software repo

• For each tool (or multiple), this repo stores the implementation of a specific benchmark in a specific tool

• In the optimal case, It provides a workflow to call the tool for a specific benchmark

• It regularly performs integration tests, potentially creating new ROCrates for each run (potentially those can be published automatically), at least when major changes are done. It might also contain multiple versions (e.g. fenics and fenicx, or version 12.0 and 13.0 of a software) that can still be tested (as long as they are maintained and are used

With this issue, we document the progress of this splitting process.

[berndflemisch]

Since the proposed splitting into "benchmark" and "software" repos appears a bit too fine-granular, we would start with storing the available implementations within the associated benchmark repo.

Which would, in my understanding, mean that we outsource for the plate-with-a-hole scenario:

• docs
• implementations
• notebook
• Related CI stuff

[berndflemisch]

I have a GitHub organization https://github.com/Simulation-Benchmarks/. Are there objections against storing the benchmark repos there?

[joergfunger]

I agree to your suggestions, I would, the notebook does not have to be stored, but the python script that does the postprocessing (that is then automatically integrated into the notebook together with the documentation). And we need a testing procedure that runs all the tools with a reduced set of parameter configurations and compare some result metrics. And a subfolder with the semantic representation of the benchmark. So the subfolders that @VasiliySeibert might create could be

• docs
• semantic_description
• software_implementations
• post_processing
• test

[VasiliySeibert]

Following the meeting decision (2026-04-30), I have executed the split. Three new repositories were created under github.com/Simulation-Benchmarks:

• Main — Simulation-Benchmarks/NFDI4IngModelValidationPlatform
  Registry + general docs + examples + shared infrastructure
• Plate with Hole — Simulation-Benchmarks/linear-elastic-plate-with-hole
  Benchmark definition + implementations + post-processing
• Rotating Cylinders — Simulation-Benchmarks/rotating-cylinders
  Benchmark definition + DuMux implementation

What was moved where

Main repo now contains:

• docs/ — general documentation, getting-started guides, provenance docs, ROHub docs
• examples/ — simplified "how to build a benchmark" implementations
• src/meshhelper/ — shared Python utility
• notebooks/ — Jupyter notebooks for exploring results
• Registry table in docs/index.md linking to all benchmark repos
  Benchmark repos (plate-with-hole as the pilot) now contain:
• docs/ — benchmark-specific documentation (problem definition, PDE, parameters)
• fenics/, kratos/, extendablefem/ — software implementations
• plot_metrics.py, notebooks/RoCrate.ipynb — post-processing scripts
• tests/ — test reference data
• common/ — copied shared scripts (summarize_results.py, provenance.py, upload_provenance.py, etc.) so each repo is self-contained
• CI workflows that run the benchmark and upload RO-Crates to ROHub

Open questions

1. Folder naming convention — Do we align exactly with @joergfunger's proposed subfolder names (software_implementations, post_processing, semantic_description)? Or keep the current flatter layout?
2. Semantic description folder — Should we create a dedicated semantic_description/ folder containing parameters_*.json, create_mesh.py, generate_config.py, and the analytical solution script?
3. Common scripts — We copied common/ into each benchmark repo for self-containment. If summarize_results.py or provenance.py evolve in one repo, others will diverge. Should we later extract these into a pip-installable package from the main repo?
4. CI secrets — ROHUB_USERNAME and ROHUB_PASSWORD need to be re-configured in the new benchmark repos' GitHub Settings → Secrets.
5. ReadTheDocs — The main repo's mkdocs.yml now points to the new Simulation-Benchmarks URL. Do we need to re-register ReadTheDocs for the new main repo?
6. Old repo archival — Should we add a deprecation notice to BAMresearch/NFDI4IngModelValidationPlatform and archive it?

[joergfunger]

@VasiliySeibert thanks for the separation.
1: Folder naming conventions: I'm also fine with the flatter folder convention with having the tool implementations directly on the top level. In the current repo (and now also in the new one), there is however a lot of boilerplate code that was used at some point, put is no longer needed. I'm currently even not fully sure, if we need a general repo apart for the registry of the other repos and potentially building a joint documentation (potentially there is some stuff related to packaging the ROCrate to be done by @M-Jafarkhani ). The documentation itself should be actually part of each benchmark repo, so not as it is now for all together, but one doc per benchmark - meaning the doc has to be moved to the plate with a hole repo.
2. The semantic representation would just be a single json_ld file. There is no need to have the parameters*.json being stored, since they are directly created from that semantic representation. Also, create_mesh and generate_config are probably no longer needed. We good place the json_ld file directly at the top level, or create a folder.
3. I think also the common scripts are no longer needed that includes summarize_results.py and provenance.py
4. I guess this has to be done by @M-Jafarkhani , but i would wait until @div-tyg has updated the current version implementations
5. As discussed in 1., i would opt for a documentation per benchmark (this is the one used for building the notebook). As such, we would have to reconfigure ReadtheDocs per Benchmark repo
6. Old repo archival. My suggestion is to currently focus only on the Plate with a hole repo, and see, if we really need general.

I'll mark the #83 as duplicate, maybe we could continue the discussion on the timeline up to the workshop here. I will copy the content of that issue here again.

[joergfunger]

In order to have something for the workshop to demonstrate, there is a couple of things that we have to finalize. Due to the relatively tight schedule, and the dependencies of tasks upon each other, I want to suggest a potential timeline for the different parts.

Until May 17th, @VasiliySeibert will create a new repository under https://github.com/Simulation-Benchmarks for the plate with a hole example (as we have agreed, we will outsource each benchmark into a separate repository, currently we will put the actual definition and the implementation for different into a single repo that could later be further subdivided). This will include a generally FAIR repo (license, cff, ..), the documentation (so as in the current repo a ReadtheDocs documentation with a button that links to that documentation) and a github actions pipeline that creates from the markdown documentation and some python script (at that stage just use a dummy script that plots some arbitrary graphs) an executable jupyter notebook. This notebook should be published as an artefact in github actions. In addition, it would be nice if we could somehow have an attached link that people can directly execute that on the NFDI juypterhub (or NFDI4ING)

Until May 24th, @div-tyg will transfer the example codes (FEniCSx, Kratos, extendableFEM, Edelweiß) to that repo. (if @jpthiele is able to add the other metrics before that , this would be great, otherwise this has to be adapted in the new repo). The goal would be to setup a github actions pipeline that runs a single configuration for each implementation within a testing procedure, and in addition, runs the pipelines to create the ROCrate.

Until May 31st, @M-Jafarkhani will integrate the actual assembly of the RO-Crates with the subcrates for the results. They will also be added as artefacts, and in parallel uploaded to the github actions. The dummy plotting file (see @VasiliySeibert) will be modified to actually query the ROCrates and plot the real simulation metrics.
In addition, the semantic description of the benchmark (such that the parameter files are automatically generated from the semantic representation) will be integrated. This also ends up in an ROCrate that includes the semantic benchmark definition with the jupyter notebook (that has the markdown documentation and the postprocessing - see above) .

In June, we will then discuss the final results and have some time to polish the procedure. In addition to the developers as mentioned above, I would like @doigl and @berndflemisch to comment on the principle approach, the individual tasks or the timeline (at least with a thumbs up, if you agree).

I would only then try to adapt the approach/repo to the rotating cylinders @Sarbani-Roy , I hope that we could finish that then still before the workshop.