Commit 565f4be
authored
feat(pt/dpmodel): add lmdb dataloader (#5283)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Full LMDB data support across training, validation and testing with
PyTorch-friendly dataset/dataloader integration, per-atom-count (nloc)
grouping, and deterministic distributed batching.
* **Tests**
* Extensive unit and integration tests covering LMDB reader,
dataset/dataloader, samplers, collation, and mixed-/uniform-nloc
scenarios.
* **Documentation**
* Added an example LMDB training configuration and updated test runner
to handle LMDB-grouped test runs.
* **Chores**
* Added runtime dependencies for LMDB and msgpack support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->1 parent 52792f0 commit 565f4be
18 files changed
Lines changed: 4212 additions & 111 deletions
File tree
- deepmd
- dpmodel/utils
- entrypoints
- pt
- entrypoints
- train
- utils
- utils
- examples/lmdb_downsample_data
- water_training.lmdb
- water_validation.lmdb
- source/tests
- common
- dpmodel
- consistent
- pt
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
9 | 17 | | |
10 | 18 | | |
11 | 19 | | |
| |||
44 | 52 | | |
45 | 53 | | |
46 | 54 | | |
| 55 | + | |
47 | 56 | | |
48 | 57 | | |
49 | 58 | | |
| 59 | + | |
| 60 | + | |
50 | 61 | | |
51 | 62 | | |
52 | 63 | | |
53 | 64 | | |
| 65 | + | |
54 | 66 | | |
55 | 67 | | |
56 | 68 | | |
| |||
59 | 71 | | |
60 | 72 | | |
61 | 73 | | |
| 74 | + | |
62 | 75 | | |
63 | 76 | | |
64 | 77 | | |
65 | 78 | | |
| 79 | + | |
66 | 80 | | |
67 | 81 | | |
68 | 82 | | |
| |||
0 commit comments