feat: remove obsolete lineage view and refactor fetching to use all_docs

[sh1vam31]

Description

Fixes #10748

This PR removes the obsolete and resource-heavy docs_by_id_lineage map-reduce view from the medic-client design document and replaces the lineage-fetching logic with a more efficient two-step all_docs process.

The docs_by_id_lineage view was previously one of the largest views in the system, emitting every ancestor for every record. As identified in Issue #10748, this can be replaced by direct document retrieval since the child document already contains the necessary ancestor chain.This PR removes the obsolete and resource-heavy docs_by_id_lineage map-reduce view from the medic-client design document and replaces the lineage-fetching logic with a more efficient two-step all_docs process.

The docs_by_id_lineage view was previously one of the largest views in the system, emitting every ancestor for every record. As identified in Issue #10748, this can be replaced by direct document retrieval since the child document already contains the necessary ancestor chain.

Changes

Refactoring Fetch Logic

shared-libs/lineage: Updated fetchLineageById in hydration.js to first fetch the child document via DB.get() and then retrieve all ancestors using their IDs in a single DB.allDocs() call.
shared-libs/cht-datasource: Updated getLineageDocsById in lineage.ts to implement the same two-step fetch strategy for local data contexts.
Ancestor Resolution: Maintained accurate lineage depth and order by ensuring that if an ancestor is missing, it is correctly represented as null in the resulting array.

View Removal & Cleanup
ddocs/medic-db: Deleted the docs_by_id_lineage view definition.
webapp/tests: Removed the now-obsolete docs_by_id_lineage.spec.js unit test.

Test Updates

Updated mocks and unit tests in shared-libs/lineage, shared-libs/cht-datasource, admin, and webapp to match the new fetching pattern.
Updated API integration tests in server.spec.js to ensure the hydration endpoint remains fully functional without the view dependency.

Code review checklist

• UI/UX backwards compatible: Test it works for the new design (enabled by default). And test it works in the old design, enable can_view_old_navigation permission to see the old design. Test it has appropriate design for RTL languages.
• Readable: Concise, well named, follows the style guide
• Documented: Configuration and user documentation on cht-docs
• Tested: Unit and/or e2e where appropriate
• Internationalised: All user facing text
• Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.
• AI disclosure: Please disclose use of AI per the guidelines.

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

[sh1vam31]

Hi @witash,
This PR addresses #10748 by removing the docs_by_id_lineage view and switching to a more efficient two-step fetch strategy. I've updated the relevant tests in the shared libraries and services to ensure everything remains stable. Looking forward to your feedback

[sh1vam31]

Hi @witash,

The ci-compile check failed due to a transient CI infrastructure issue, not a code problem. The shellcheck binary download from GitHub was rate-limited (HTTP 403) during the npm run lint-shell step:

RequestError: Status code '403' and message 'rate limit exceeded'

The ESLint and blank-link checks both passed successfully. This is unrelated to the code changes in this PR. Could you please re-run the failed job, or let me know if there's another way to trigger CI? I

[witash]

I consistenly get this error running npm run unit

FAILED TESTS:
  LineageModelGenerator service
    report
      ✖ hydrates lineage contacts - #3812
        Chrome Headless 135.0.0.0 (Linux x86_64)
      TypeError: Cannot read properties of undefined (reading 'then')
          at fetchDocs (_karma_webpack_/webpack:/shared-libs/lineage/src/hydration.js:387:52)
          at fetchContacts (_karma_webpack_/webpack:/shared-libs/lineage/src/hydration.js:133:12)
          at apply (_karma_webpack_/webpack:/shared-libs/lineage/src/hydration.js:324:20)
          at _ZoneDelegate.invoke (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone.js:398:28)
          at ProxyZoneSpec.onInvoke (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone-testing.js:2132:39)
          at _ZoneDelegate.invoke (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone.js:397:34)
          at ZoneImpl.run (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone.js:113:43)
          at apply (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone.js:2537:40)
          at _ZoneDelegate.invokeTask (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone.js:431:33)
          at ProxyZoneSpec.onInvokeTask (_karma_webpack_/webpack:/node_modules/zone.js/fesm2015/zone-testing.js:2163:39)

the stub at webapp/tests/karma/ts/services/lineage-model-generator.service.spec.ts:339 may be the culprit? not containing 'x' becuase it was already present so was not fetched again? Or something similar.

[witash]

The ci-compile check failed due to a transient CI infrastructure issue, not a code problem. The shellcheck binary download from GitHub was rate-limited (HTTP 403) during the npm run lint-shell step:

RequestError: Status code '403' and message 'rate limit exceeded'

The ESLint and blank-link checks both passed successfully. This is unrelated to the code changes in this PR. Could you please re-run the failed job, or let me know if there's another way to trigger CI

npm run lint-shell itself looks fine, but I get errors running eslint with npm run lint

npm run lint

> medic@5.1.0 lint
> eslint --color --cache . && ./scripts/build/blank-link-check.sh && npm run lint-shell


/home/tom-medic/medic/cht-core/api/tests/mocha/services/settings.spec.js
  9:1  warning  Unused eslint-disable directive (no problems were reported from 'n/no-missing-require')

/home/tom-medic/medic/cht-core/scripts/benchmark-hydration.js
   30:1   error  Expected a function expression                                                                                 func-style
   54:1   error  Expected a function expression                                                                                 func-style
   79:13  error  Statement inside of curly braces should be on next line                                                        @stylistic/brace-style
   79:75  error  Closing curly brace does not appear on the same line as the subsequent block                                   @stylistic/brace-style
   79:75  error  Closing curly brace should be on the same line as opening curly brace or on the line after the previous block  @stylistic/brace-style
   80:15  error  Statement inside of curly braces should be on next line                                                        @stylistic/brace-style
   80:65  error  Closing curly brace should be on the same line as opening curly brace or on the line after the previous block  @stylistic/brace-style
   84:15  error  Expected { after 'if' condition                                                                                curly
   92:1   error  Expected a function expression                                                                                 func-style
  106:1   error  This line has a length of 122. Maximum allowed is 120                                                          @stylistic/max-len
  117:1   error  This line has a length of 130. Maximum allowed is 120                                                          @stylistic/max-len
  139:1   error  Expected a function expression                                                                                 func-style
  151:1   error  Expected a function expression                                                                                 func-style
  163:1   error  Expected a function expression                                                                                 func-style
  168:1   error  Expected a function expression                                                                                 func-style
  183:1   error  Expected a function expression                                                                                 func-style
  183:19  error  Statement inside of curly braces should be on next line                                                        @stylistic/brace-style
  183:42  error  Closing curly brace should be on the same line as opening curly brace or on the line after the previous block  @stylistic/brace-style
  185:1   error  Expected a function expression                                                                                 func-style
  207:1   error  Expected a function expression                                                                                 func-style
  297:1   error  This line has a length of 157. Maximum allowed is 120                                                          @stylistic/max-len

✖ 22 problems (21 errors, 1 warning)
  8 errors and 1 warning potentially fixable with the `--fix` option.

[witash]

I tested the performance of this on a small db using GET /api/v1/{type}/{id}?with_lineage=true)
There's not much difference between using docs_by_id_lineage and two calls to all_docs, which is good; it doesn't need to be better at all, it just needs to be not too much worse.

There might be more difference if it was getting the lineage for batches of ids, instead of each id individually...but currently it does each id individually so it doesn't matter.

docs_by_id_lineage

Metric	Value
Count	10000
Min	6.0 ms
Median (p50)	11.0 ms
Mean	12.6 ms
p90	18.3 ms
p95	19.2 ms
p99	21.1 ms
Max	28.9 ms

Total: 126.0s

This branch

Metric	Value
Count	10000
Min	5.3 ms
Median (p50)	9.2 ms
Mean	11.3 ms
p90	16.7 ms
p95	17.5 ms
p99	19.7 ms
Max	32.0 ms

Total: 113.0s

[sh1vam31]

Hi @witash

I’ve just pushed a new set of fixes to address the test regressions and linting errors identified in your review.

Summary of changes:

Lineage Unit Tests: Updated both the Webapp and Admin LineageModelGenerator tests to support the new allDocs-based hydration logic. This included embedding full parent chains in mock documents and correcting the Sinon stub assertions to match the new fetch sequence.
Linting: Resolved the unused eslint-disable directive in api/tests/mocha/services/settings.spec.js and fixed the line-length errors in the Admin spec.
Benchmark Script: I noticed the lint errors in scripts/benchmark-hydration.js from your review, but that file isn’t present in my branch or the repository. It seems it might be a local script you've added for testing.
All unit tests are now passing locally (447 SUCCESS in Admin). Please let me know if there’s anything else that needs addressing before we can move toward a final review.

Thanks for your guidance.