Use RlpArrayBufWriter for rlp encoding in stateroot computation

[bhartnett]

The RlpArrayBufWriter only uses stack memory for the rlp encoding and using it improves the performance of the stateroot computation. When returning the rlp bytes we also don't allocate any seqs.

The RlpArrayBufWriter is based on the RlpDefaultWriter but replaces the usages of seqs with ArrayBufs to avoid the heap allocations. It turns out that these writers are faster than the two pass writer for small/simple rlp encodings and so this implementation is best suited to the stateroot computation which only encodes simple objects.

Also I don't think using the RlpHashWriter to rlp and hash in place will be any faster than this because it does two passes of the data (similar to the two pass writer). It needs to do this to calculate the length prefix but it turns out that going over the data 2 times is slower (at least this was my conclusion from doing some quick benchmarks).

[bhartnett]

Benchmark results...

The in memory unit test is at least 2x faster with this change. For the full stateroot computation on startup most of the time is spent reading and writing to the database so the speedup is less noticeable but still significant.

master branch:
Aristo compute benchmark
Serial benchmark (skipLayers = false) running...
DBG 2026-04-11 23:29:28.558+08:00 Writing computeKey cache file=aristo_compute.nim:89 keys=21399 accounts=37.16%
DBG 2026-04-11 23:29:34.592+08:00 Writing computeKey cache file=aristo_compute.nim:89 keys=42798 accounts=74.03%
DBG 2026-04-11 23:29:37.998+08:00 Wrote computeKey cache file=aristo_compute.nim:302 keys=58216 accounts=100.00%
Serial benchmark (skipLayers = false) cpu time: 17.362139130999996

RlpArrayBufWriter branch:
Aristo compute benchmark
Serial benchmark (skipLayers = false) running...
DBG 2026-04-11 23:30:39.793+08:00 Writing computeKey cache file=aristo_compute.nim:98 keys=21399 accounts=37.16%
DBG 2026-04-11 23:30:42.258+08:00 Writing computeKey cache file=aristo_compute.nim:98 keys=42798 accounts=74.03%
DBG 2026-04-11 23:30:43.683+08:00 Wrote computeKey cache file=aristo_compute.nim:318 keys=58216 accounts=100.00%
Serial benchmark (skipLayers = false) cpu time: 7.142571434999997

master branch:

Memory usage - 6.1G

NTC 2026-04-11 23:53:12.103+08:00 Importing era1 archive start=5000001 dataDir=/home/user/.local/state/nimbus/mainnet era1Dir=/media/user/BlockchainData/era1/mainnet
INF 2026-04-11 23:53:58.570+08:00 Writing computeKey cache keys=2139900 accounts=6.66%
INF 2026-04-11 23:54:38.083+08:00 Writing computeKey cache keys=4279800 accounts=11.79%
INF 2026-04-11 23:55:20.008+08:00 Writing computeKey cache keys=6419700 accounts=23.22%
INF 2026-04-11 23:56:03.879+08:00 Writing computeKey cache keys=8559600 accounts=34.03%
INF 2026-04-11 23:56:47.786+08:00 Writing computeKey cache keys=10699500 accounts=44.53%
INF 2026-04-11 23:57:28.928+08:00 Writing computeKey cache keys=12839400 accounts=50.47%
INF 2026-04-11 23:58:14.659+08:00 Writing computeKey cache keys=14979300 accounts=61.03%
INF 2026-04-11 23:59:03.254+08:00 Writing computeKey cache keys=17119200 accounts=73.91%
INF 2026-04-11 23:59:44.162+08:00 Writing computeKey cache keys=19259100 accounts=84.82%
INF 2026-04-12 00:00:23.576+08:00 Writing computeKey cache keys=21399000 accounts=93.33%
INF 2026-04-12 00:00:42.728+08:00 Wrote computeKey cache keys=22487281 accounts=100.00%
INF 2026-04-12 00:00:42.786+08:00 Imported blocks blockNumber=5000002 slot=1 blocks=1 txs=88 mgas=7.986 bps=0.002 tps=0.195 mgps=0.018 avgBps=0.002 avgTps=0.195 avgMGps=0.018 elapsed=7m30s692ms
NTC 2026-04-12 00:00:42.820+08:00 Import complete blockNumber=5000002 slot=1 blocks=1 txs=88 mgas=7.986

RlpArrayBufWriter branch:

Memory usage - 6.1G

NTC 2026-04-11 23:45:14.974+08:00 Importing era1 archive start=5000001 dataDir=/home/user/.local/state/nimbus/mainnet era1Dir=/media/user/BlockchainData/era1/mainnet
INF 2026-04-11 23:45:59.009+08:00 Writing computeKey cache keys=2139900 accounts=6.66%
INF 2026-04-11 23:46:39.122+08:00 Writing computeKey cache keys=4279800 accounts=11.79%
INF 2026-04-11 23:47:13.317+08:00 Writing computeKey cache keys=6419700 accounts=23.22%
INF 2026-04-11 23:47:47.538+08:00 Writing computeKey cache keys=8559600 accounts=34.03%
INF 2026-04-11 23:48:25.844+08:00 Writing computeKey cache keys=10699500 accounts=44.53%
INF 2026-04-11 23:49:01.437+08:00 Writing computeKey cache keys=12839400 accounts=50.47%
INF 2026-04-11 23:49:38.887+08:00 Writing computeKey cache keys=14979300 accounts=61.03%
INF 2026-04-11 23:50:17.615+08:00 Writing computeKey cache keys=17119200 accounts=73.91%
INF 2026-04-11 23:50:51.567+08:00 Writing computeKey cache keys=19259100 accounts=84.82%
INF 2026-04-11 23:51:30.890+08:00 Writing computeKey cache keys=21399000 accounts=93.33%
INF 2026-04-11 23:51:50.556+08:00 Wrote computeKey cache keys=22487281 accounts=100.00%
INF 2026-04-11 23:51:50.598+08:00 Imported blocks blockNumber=5000002 slot=1 blocks=1 txs=88 mgas=7.986 bps=0.003 tps=0.222 mgps=0.020 avgBps=0.003 avgTps=0.222 avgMGps=0.020 elapsed=6m35s646ms
NTC 2026-04-11 23:51:50.650+08:00 Import complete blockNumber=5000002 slot=1 blocks=1 txs=88 mgas=7.986

[status-im-auto]

Jenkins Builds

Click to see older builds (5)

❔	Commit	#️⃣	Finished (UTC)	Duration	Platform	Result
⁉️	2aaf4c1	#1	2026-04-11 17:01:17	~22 min	unknown	📄log
⁉️	84f36ca2	#1	2026-04-11 17:01:17	~22 min	unknown	📄log
⁉️	757b7c7f	#2	2026-04-11 17:25:02	~23 min	unknown	📄log
⁉️	f1e2247	#2	2026-04-11 17:25:31	~24 min	unknown	📄log
⁉️	0d589e8	#3	2026-04-13 14:51:21	~24 min	unknown	📄log

❔	Commit	#️⃣	Finished (UTC)	Duration	Platform	Result
⁉️	b4b6fa8	#4	2026-04-14 03:46:04	~22 min	unknown	📄log
⁉️	4ee04de	#5	2026-04-14 05:22:18	~22 min	unknown	📄log