[OP/Base node] Blocks produced by Engine API are rolled back after reth restart

[shivanshs9]

Describe the bug

With Engine API–driven block production (forkchoiceUpdated → getPayload → newPayload → final FCU), eth_blockNumber RPC and reth logs show a higher tip than what's actually persisted by reth on the disk.

Example scenario:
Head Block from snapshot: 43949199
Produced 3-5 blocks
Verified with eth_blockNumber, the RPC tip moved
On reth restart, eth_blockNumber the tip is back to 43949199
Unable to get block info by cast block 43949200 either, seems like they're not persisted.

I tried to check the DB state after the blocks were produced from within the container, but seems like the TransactionBlocks table doesn't have the newly-produced blocks.

$  /app/base-reth-node db --datadir /data stats   --chain base   --skip-consistency-checks

| Table Name                 | # Entries  | Branch Pages | Leaf Pages | Overflow Pages | Total Size |
|----------------------------|------------|--------------|------------|----------------|------------|
| AccountChangeSets          | 447290437  | 1806312      | 5850001    | 0              | 29.2 GiB   |
| AccountsHistory            | 12684759   | 9189         | 585373     | 124363         | 2.7 GiB    |
| AccountsTrie               | 55889791   | 18798        | 3303317    | 0              | 12.7 GiB   |
| BlockBodyIndices           | 43949200   | 1183         | 264548     | 0              | 1 GiB      |
| BlockOmmers                | 0          | 0            | 0          | 0              | 0 B        |
| BlockWithdrawals           | 0          | 0            | 0          | 0              | 0 B        |
| Bytecodes                  | 8208180    | 8866         | 593277     | 27464912       | 107.1 GiB  |
| CanonicalHeaders           | 0          | 0            | 0          | 0              | 0 B        |
| ChainState                 | 2          | 0            | 1          | 0              | 4 KiB      |
| HashedAccounts             | 677101164  | 228766       | 19476385   | 0              | 75.2 GiB   |
| HashedStorages             | 6926063455 | 2811560      | 124173867  | 0              | 484.4 GiB  |
| HeaderNumbers              | 43949200   | 14825        | 935942     | 0              | 3.6 GiB    |
| HeaderTerminalDifficulties | 0          | 0            | 0          | 0              | 0 B        |
| Headers                    | 0          | 0            | 0          | 0              | 0 B        |
| Metadata                   | 0          | 0            | 0          | 0              | 0 B        |
| PlainAccountState          | 677101164  | 141412       | 13207207   | 0              | 50.9 GiB   |
| PlainStorageState          | 6926063455 | 3432001      | 145756244  | 0              | 569.1 GiB  |
| PruneCheckpoints           | 8          | 0            | 1          | 0              | 4 KiB      |
| Receipts                   | 295637417  | 82894        | 18567164   | 2882254        | 82.1 GiB   |
| StageCheckpointProgresses  | 1          | 0            | 1          | 0              | 4 KiB      |
| StageCheckpoints           | 16         | 0            | 1          | 0              | 4 KiB      |
| StorageChangeSets          | 2566968037 | 4358053      | 43891811   | 0              | 184.1 GiB  |
| StoragesHistory            | 630140995  | 1303077      | 36345825   | 777767         | 146.6 GiB  |
| StoragesTrie               | 599661412  | 4350820      | 40157536   | 0              | 169.8 GiB  |
| TransactionBlocks          | 43949199   | 1260         | 281726     | 0              | 1.1 GiB    |
| TransactionHashNumbers     | 1015581708 | 248697       | 17953712   | 0              | 69.4 GiB   |
| TransactionSenders         | 295637417  | 12338        | 2762968    | 0              | 10.6 GiB   |
| Transactions               | 0          | 0            | 0          | 0              | 0 B        |
| VersionHistory             | 17         | 0            | 1          | 0              | 4 KiB      |
| -------------------------- | ---------- | ------------ | ---------- | -------------- | ---------- |
| Tables                     |            |              |            |                | 2 TiB      |
| Freelist                   | 38205978   |              |            |                | 145.7 GiB  |

Steps to reproduce

1. Run base-reth-node in isolated network (no op-node consensus running)
2. Use pruned reth snapshot for Base mainnet
   • snapshot copied via cp --reflink=always in a XFS filesystem
   • mounted inside the container in RW mode
3. Provided below flags:
   • --db.sync-mode=durable
   • --engine.persistence-threshold=0
   • --engine.memory-block-buffer-target=0
4. Produced some blocks via custom minimal CL:
   • speaking Engine API JWT to Reth (8551), sealing via engine_forkchoiceUpdatedV3 → engine_getPayloadV4 → engine_newPayloadV4 → engine_forkchoiceUpdatedV3

Node logs

2026-03-31T15:14:00.806331Z  INFO Initialized tracing log_dir=/root/.cache/reth/logs/base
2026-03-31T15:14:00.813621Z  INFO Starting base-reth-node version="1.11.3-dev (d6324d6)"
2026-03-31T15:14:00.813663Z  INFO Opening database path="/data/db"
2026-03-31T15:14:01.094219Z  INFO starting custom Base node
2026-03-31T15:14:01.094302Z  INFO flashblocks integration is disabled
2026-03-31T15:14:01.096642Z  WARN Pruning configuration is present in the config file, but no CLI arguments are provided. Using config from file.
2026-03-31T15:14:01.096659Z  INFO Configuration loaded path="/reth.toml"
2026-03-31T15:14:01.098668Z  INFO check_consistency: Healing static file inconsistencies.
2026-03-31T15:14:01.108747Z  INFO check_consistency:check_consistency{read_only=false}: Verifying storage consistency.
2026-03-31T15:14:01.108945Z  INFO Database opened
2026-03-31T15:14:01.109109Z  INFO 
Pre-merge hard forks (block based):
- Bedrock                          @0
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Canyon                           @1704992401
- Ecotone                          @1710374401
- Fjord                            @1720627201
- Granite                          @1726070401
- Holocene                         @1736445601
- Isthmus                          @1746806401
- Jovian                           @1764691201
2026-03-31T15:14:01.109130Z  INFO Loaded storage settings settings=StorageSettings { storage_v2: false }
2026-03-31T15:14:01.109964Z  INFO Transaction pool initialized
2026-03-31T15:14:01.128962Z  INFO P2P networking initialized enode=enode://2a44fbd68a20fe4b2406ffdffde6a993292bab81b0d92da1f39dba4c980855599406cfb461d4a4cbc2e4c07167bde5eeb7eae33164976f641f18e406d326cb31@127.0.0.1:30303
2026-03-31T15:14:01.129211Z  INFO StaticFileProducer initialized
2026-03-31T15:14:01.129762Z  INFO Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: Some(Distance(1339200)), transaction_lookup: Some(Distance(1339200)), receipts: Some(Distance(1339200)), account_history: Some(Distance(1339200)), storage_history: Some(Distance(1339200)), bodies_history: Some(Distance(1339200)), receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2026-03-31T15:14:01.130224Z  INFO Consensus engine initialized
2026-03-31T15:14:01.130291Z  INFO Engine API handler initialized
2026-03-31T15:14:01.133336Z  INFO RPC auth server started url=0.0.0.0:8551
2026-03-31T15:14:01.133849Z  INFO RPC IPC server started path=/tmp/reth.ipc
2026-03-31T15:14:01.133870Z  INFO RPC HTTP server started url=0.0.0.0:8545
2026-03-31T15:14:01.133874Z  INFO RPC WS server started url=0.0.0.0:8546
2026-03-31T15:14:01.134025Z  INFO Starting consensus engine
2026-03-31T15:14:03.022628Z  INFO Received forkchoice updated message when syncing head_block_hash=0xb137373a707b043faf3450e4ec7e5ae2e0fcb807c3d0fb361524e74618d67559 safe_block_hash=0xb137373a707b043faf3450e4ec7e5ae2e0fcb807c3d0fb361524e74618d67559 finalized_block_hash=0xb137373a707b043faf3450e4ec7e5ae2e0fcb807c3d0fb361524e74618d67559
2026-03-31T15:14:04.131949Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:15:19.131922Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:16:34.132408Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:17:11.994127Z  INFO Forkchoice updated head_block_hash=0x9cb4a55618274e02e3aef446fed7d5946e172423f086319726610af381d2ed15 safe_block_hash=0x9cb4a55618274e02e3aef446fed7d5946e172423f086319726610af381d2ed15 finalized_block_hash=0x9cb4a55618274e02e3aef446fed7d5946e172423f086319726610af381d2ed15
2026-03-31T15:17:11.994160Z  INFO New payload job created id=0x03b3e064a17e8d5c parent=0x9cb4a55618274e02e3aef446fed7d5946e172423f086319726610af381d2ed15
2026-03-31T15:17:12.099700Z  INFO Block added to canonical chain number=43949200 hash=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9 peers=0 txs=1 gas_used=60.15Kgas gas_throughput=18.13Mgas/second gas_limit=30.00Mgas full=0.2% base_fee=0.01Gwei blobs=0 excess_blobs=0 elapsed=3.317ms
2026-03-31T15:17:12.099742Z  INFO Received new payload from consensus engine number=43949200 hash=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9
2026-03-31T15:17:12.101566Z  INFO Canonical chain committed number=43949200 hash=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9 elapsed=38.52µs
2026-03-31T15:17:12.101594Z  INFO Forkchoice updated head_block_hash=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9 safe_block_hash=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9 finalized_block_hash=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9
2026-03-31T15:17:28.000421Z  INFO New payload job created id=0x03a5262720ef5ec0 parent=0x286caa2420ce678cdb1201adf11835261c9c8b8e4699e9fc3b024e65ea63bdd9
2026-03-31T15:17:28.102485Z  INFO Block added to canonical chain number=43949201 hash=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4 peers=0 txs=2 gas_used=113.93Kgas gas_throughput=389.72Mgas/second gas_limit=30.00Mgas full=0.4% base_fee=0.00Gwei blobs=0 excess_blobs=0 elapsed=292.336µs
2026-03-31T15:17:28.102524Z  INFO Received new payload from consensus engine number=43949201 hash=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4
2026-03-31T15:17:28.103637Z  INFO Canonical chain committed number=43949201 hash=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4 elapsed=46.061µs
2026-03-31T15:17:28.103653Z  INFO Forkchoice updated head_block_hash=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4 safe_block_hash=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4 finalized_block_hash=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4
2026-03-31T15:17:41.622295Z  INFO New payload job created id=0x03455d61e1643fc9 parent=0x631e3147d4f144f1de0cf88ce4ac621985fcf4d2f71d50a9a609d5805555bda4
2026-03-31T15:17:41.724334Z  INFO Block added to canonical chain number=43949202 hash=0x2769123584318edcc250efd5a449e876760c238936b0e089bdef3f4195379d07 peers=0 txs=2 gas_used=96.83Kgas gas_throughput=435.57Mgas/second gas_limit=30.00Mgas full=0.3% base_fee=0.00Gwei blobs=0 excess_blobs=0 elapsed=222.306µs
2026-03-31T15:17:41.724638Z  INFO Received new payload from consensus engine number=43949202 hash=0x2769123584318edcc250efd5a449e876760c238936b0e089bdef3f4195379d07
2026-03-31T15:17:41.725608Z  INFO Canonical chain committed number=43949202 hash=0x2769123584318edcc250efd5a449e876760c238936b0e089bdef3f4195379d07 elapsed=29.195µs
2026-03-31T15:17:41.725618Z  INFO Forkchoice updated head_block_hash=0x2769123584318edcc250efd5a449e876760c238936b0e089bdef3f4195379d07 safe_block_hash=0x2769123584318edcc250efd5a449e876760c238936b0e089bdef3f4195379d07 finalized_block_hash=0x2769123584318edcc250efd5a449e876760c238936b0e089bdef3f4195379d07
2026-03-31T15:17:49.132239Z  INFO Status connected_peers=0 latest_block=43949202
2026-03-31T15:19:04.132246Z  INFO Status connected_peers=0 latest_block=43949202
2026-03-31T15:20:19.132013Z  INFO Status connected_peers=0 latest_block=43949202
2026-03-31T15:21:34.131527Z  INFO Status connected_peers=0 latest_block=43949202
2026-03-31T15:22:49.131862Z  INFO Status connected_peers=0 latest_block=43949202
2026-03-31T15:24:01.130928Z  WARN Beacon client online, but no consensus updates received for a while. This may be because of a reth error, or an error in the beacon client! Please investigate reth and beacon client logs! period=379.405892352s
2026-03-31T15:24:04.131738Z  INFO Status connected_peers=0 latest_block=43949202
2026-03-31T15:25:08.922227Z  INFO Received SIGTERM

---podman stop and podman start commands---

2026-03-31T15:25:10.813662Z  INFO Initialized tracing log_dir=/root/.cache/reth/logs/base
2026-03-31T15:25:10.814200Z  INFO Starting base-reth-node version="1.11.3-dev (d6324d6)"
2026-03-31T15:25:10.814220Z  INFO Opening database path="/data/db"
2026-03-31T15:25:10.825530Z  INFO starting custom Base node
2026-03-31T15:25:10.825600Z  INFO flashblocks integration is disabled
2026-03-31T15:25:10.828261Z  WARN Pruning configuration is present in the config file, but no CLI arguments are provided. Using config from file.
2026-03-31T15:25:10.828283Z  INFO Configuration loaded path="/reth.toml"
2026-03-31T15:25:10.830379Z  INFO check_consistency: Healing static file inconsistencies.
2026-03-31T15:25:10.841905Z  INFO check_consistency:check_consistency{read_only=false}: Verifying storage consistency.
2026-03-31T15:25:10.842112Z  INFO Database opened
2026-03-31T15:25:10.842264Z  INFO 
Pre-merge hard forks (block based):
- Bedrock                          @0
Post-merge hard forks (timestamp based):
- Regolith                         @0
- Canyon                           @1704992401
- Ecotone                          @1710374401
- Fjord                            @1720627201
- Granite                          @1726070401
- Holocene                         @1736445601
- Isthmus                          @1746806401
- Jovian                           @1764691201
2026-03-31T15:25:10.842286Z  INFO Loaded storage settings settings=StorageSettings { storage_v2: false }
2026-03-31T15:25:10.843232Z  INFO Transaction pool initialized
2026-03-31T15:25:10.858912Z  INFO P2P networking initialized enode=enode://2a44fbd68a20fe4b2406ffdffde6a993292bab81b0d92da1f39dba4c980855599406cfb461d4a4cbc2e4c07167bde5eeb7eae33164976f641f18e406d326cb31@127.0.0.1:30303
2026-03-31T15:25:10.859215Z  INFO StaticFileProducer initialized
2026-03-31T15:25:10.859813Z  INFO Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: Some(Distance(1339200)), transaction_lookup: Some(Distance(1339200)), receipts: Some(Distance(1339200)), account_history: Some(Distance(1339200)), storage_history: Some(Distance(1339200)), bodies_history: Some(Distance(1339200)), receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2026-03-31T15:25:10.860266Z  INFO Consensus engine initialized
2026-03-31T15:25:10.860330Z  INFO Engine API handler initialized
2026-03-31T15:25:10.863172Z  INFO RPC auth server started url=0.0.0.0:8551
2026-03-31T15:25:10.865574Z  INFO RPC IPC server started path=/tmp/reth.ipc
2026-03-31T15:25:10.865593Z  INFO RPC HTTP server started url=0.0.0.0:8545
2026-03-31T15:25:10.865598Z  INFO RPC WS server started url=0.0.0.0:8546
2026-03-31T15:25:10.865737Z  INFO Starting consensus engine
2026-03-31T15:25:13.861384Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:26:28.860869Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:27:43.861007Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:28:58.861837Z  INFO Status connected_peers=0 latest_block=43949199
2026-03-31T15:30:10.861718Z  WARN Post-merge network, but never seen beacon client. Please launch one to follow the chain!

Platform(s)

Linux (x86)

Container Type

Docker

What version/commit are you on?

v1.11.3-dev (d6324d6)
Actually using base-reth image from: base/base v0.6.0

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

base mainnet fork

What type of node are you running?

Pruned with custom reth.toml config

What prune config do you use, if any?

[prune]
block_interval = 5

[prune.segments.sender_recovery]
distance = 1339200

[prune.segments.transaction_lookup]
distance = 1339200

[prune.segments.receipts]
distance = 1339200

[prune.segments.account_history]
distance = 1339200

[prune.segments.storage_history]
distance = 1339200

[prune.segments.bodies_history]
distance = 1339200

[prune.segments.merkle_changesets]
distance = 10064

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[shivanshs9]

Still observing this issue, some blocks (not all) get lost after restart:

$ make block
Using RPC URL: http://localhost:8547
Current block: 44091281
$ podman stop base-reth-testnet
base-reth-testnet
# After 1 hour
$ podman start base-reth-testnet
base-reth-testnet
$ make block
Using RPC URL: http://localhost:8547
Current block: 44033523

Affecting blockscout and other indexer, since they can't find the missing blocks after restart