Skip to content

Commit 5cd817b

Browse files
committed
feat(indexer): set up pruning for objects_backward_history (#11300)
# Description of change Add pruning support for `objects_backward_history`. **New `ByCheckpointWithLimit` pruning strategy:** - Deletes by `superseded_at_checkpoint` range with a per-statement row limit via a CTE with `FOR UPDATE LIMIT`, same pattern as `optimistic_transactions` pruning - Handles variable batch sizes — a single checkpoint range can contain many rows depending on traffic, unlike `checkpoints`/`pruner_cp_watermark` which have O(1) rows per checkpoint - The pruner loop repeats row-limited deletes until the range is fully pruned, with pauses between batches **Changes:** - `ObjectsBackwardHistory` added to `PrunableTable` enum - `ByCheckpointWithLimit` strategy + `CheckpointRangeWithLimit` chunk variant - Row-limited delete function in `pg_indexer_store.rs` **E2e test (`backward_history_pruning`):** queries 4 object versions via `objectKeys` before and after pruning — real tombstone (v3), lamport-1 tombstone (v6), previous active from backward history (v7), current active from checkpointed_objects (v8). Verifies all 4 are findable before pruning, and only the current active survives after. ## Links to any relevant issues fixes #11168 ## How the change has been tested - [x] Basic tests (linting, compilation, formatting, unit/integration tests) - [x] Patch-specific tests (correctness, functionality coverage) - `backward_history_pruning`: e2e test with `--epochs-to-keep 1`, verifies objectKeys behavior before and after pruning - Benchmarked DELETE performance with EXPLAIN ANALYZE on 1M row table ### Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure) - [ ] Synchronization of the indexer from genesis for a network including migration objects. - [ ] Restart of indexer synchronization locally without resetting the database. - [ ] Restart of indexer synchronization on a production-like database. - [ ] Deployment of services using Docker. - [ ] Verification of API backward compatibility.
1 parent b537b2c commit 5cd817b

11 files changed

Lines changed: 502 additions & 65 deletions

File tree

crates/iota-graphql-e2e-tests/tests/consistency/consistent_view_no_filter.snap

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,13 @@ task 1, lines 12-29:
1010
//# publish
1111
created: object(1,0)
1212
mutated: object(0,1)
13-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 5380800, storage_rebate: 0, non_refundable_storage_fee: 0
13+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 5380800, storage_rebate: 0, non_refundable_storage_fee: 0
1414

1515
task 2, line 31:
1616
//# run P0::m::create --sender A --args 1 @A
1717
created: object(2,0)
1818
mutated: object(0,0)
19-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2257200, storage_rebate: 0, non_refundable_storage_fee: 0
19+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2257200, storage_rebate: 0, non_refundable_storage_fee: 0
2020

2121
task 3, line 33:
2222
//# create-checkpoint
@@ -25,7 +25,7 @@ Checkpoint created: 1
2525
task 4, line 35:
2626
//# run P0::m::update --sender A --args object(2,0) 2
2727
mutated: object(0,0), object(2,0)
28-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2257200, storage_rebate: 2257200, non_refundable_storage_fee: 0
28+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2257200, storage_rebate: 2257200, non_refundable_storage_fee: 0
2929

3030
task 5, line 37:
3131
//# create-checkpoint

crates/iota-graphql-e2e-tests/tests/consistency/consistent_view_wrapped.snap

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,15 @@ task 1, lines 13-37:
1010
//# publish
1111
created: object(1,0)
1212
mutated: object(0,1)
13-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 5783600, storage_rebate: 0, non_refundable_storage_fee: 0
13+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 5783600, storage_rebate: 0, non_refundable_storage_fee: 0
1414

1515
task 2, lines 39-41:
1616
//# programmable --sender A --inputs @A
1717
//> 0: P0::m::create_foo();
1818
//> TransferObjects([Result(0)], Input(0))
1919
created: object(2,0)
2020
mutated: object(0,0)
21-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2196400, storage_rebate: 0, non_refundable_storage_fee: 0
21+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2196400, storage_rebate: 0, non_refundable_storage_fee: 0
2222

2323
task 3, line 43:
2424
//# create-checkpoint
@@ -31,7 +31,7 @@ task 4, lines 45-47:
3131
created: object(4,0)
3232
mutated: object(0,0)
3333
wrapped: object(2,0)
34-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2196400, non_refundable_storage_fee: 0
34+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2196400, non_refundable_storage_fee: 0
3535

3636
task 5, line 49:
3737
//# create-checkpoint
@@ -40,17 +40,17 @@ Checkpoint created: 2
4040
task 6, line 51:
4141
//# transfer-object 4,0 --sender A --recipient A
4242
mutated: object(0,0), object(4,0)
43-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
43+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
4444

4545
task 7, line 53:
4646
//# transfer-object 4,0 --sender A --recipient A
4747
mutated: object(0,0), object(4,0)
48-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
48+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
4949

5050
task 8, line 55:
5151
//# transfer-object 4,0 --sender A --recipient A
5252
mutated: object(0,0), object(4,0)
53-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
53+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
5454

5555
task 9, lines 57-59:
5656
//# programmable --sender A --inputs @A object(4,0)
@@ -59,7 +59,7 @@ task 9, lines 57-59:
5959
mutated: object(0,0)
6060
unwrapped: object(2,0)
6161
deleted: object(4,0)
62-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2196400, storage_rebate: 2470000, non_refundable_storage_fee: 0
62+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2196400, storage_rebate: 2470000, non_refundable_storage_fee: 0
6363

6464
task 10, line 61:
6565
//# create-checkpoint

crates/iota-graphql-e2e-tests/tests/consistency/object_keys_tombstone.snap

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ task 1, lines 23-47:
1010
//# publish
1111
created: object(1,0)
1212
mutated: object(0,1)
13-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 5783600, storage_rebate: 0, non_refundable_storage_fee: 0
13+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 5783600, storage_rebate: 0, non_refundable_storage_fee: 0
1414

1515
task 2, lines 49-52:
1616
//# programmable --sender A --inputs @A
@@ -19,7 +19,7 @@ task 2, lines 49-52:
1919
//> TransferObjects([Result(0), Result(1)], Input(0))
2020
created: object(2,0), object(2,1)
2121
mutated: object(0,0)
22-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 3412400, storage_rebate: 0, non_refundable_storage_fee: 0
22+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 3412400, storage_rebate: 0, non_refundable_storage_fee: 0
2323

2424
task 3, line 54:
2525
//# create-checkpoint
@@ -33,22 +33,22 @@ task 4, lines 56-59:
3333
created: object(4,0), object(4,1)
3434
mutated: object(0,0)
3535
wrapped: object(2,0), object(2,1)
36-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 3959600, storage_rebate: 3412400, non_refundable_storage_fee: 0
36+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 3959600, storage_rebate: 3412400, non_refundable_storage_fee: 0
3737

3838
task 5, line 61:
3939
//# transfer-object 4,0 --sender A --recipient A
4040
mutated: object(0,0), object(4,0)
41-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
41+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
4242

4343
task 6, line 63:
4444
//# transfer-object 4,0 --sender A --recipient A
4545
mutated: object(0,0), object(4,0)
46-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
46+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
4747

4848
task 7, line 65:
4949
//# transfer-object 4,0 --sender A --recipient A
5050
mutated: object(0,0), object(4,0)
51-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
51+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
5252

5353
task 8, line 67:
5454
//# create-checkpoint
@@ -62,7 +62,7 @@ task 9, lines 69-72:
6262
mutated: object(0,0)
6363
unwrapped: object(2,0), object(2,1)
6464
deleted: object(4,0), object(4,1)
65-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 3412400, storage_rebate: 3959600, non_refundable_storage_fee: 0
65+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 3412400, storage_rebate: 3959600, non_refundable_storage_fee: 0
6666

6767
task 10, lines 74-76:
6868
//# create-checkpoint
@@ -75,22 +75,22 @@ task 11, lines 77-79:
7575
created: object(11,0)
7676
mutated: object(0,0)
7777
wrapped: object(2,0)
78-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2196400, non_refundable_storage_fee: 0
78+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2196400, non_refundable_storage_fee: 0
7979

8080
task 12, line 81:
8181
//# transfer-object 11,0 --sender A --recipient A
8282
mutated: object(0,0), object(11,0)
83-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
83+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
8484

8585
task 13, line 83:
8686
//# transfer-object 11,0 --sender A --recipient A
8787
mutated: object(0,0), object(11,0)
88-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
88+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
8989

9090
task 14, line 85:
9191
//# transfer-object 11,0 --sender A --recipient A
9292
mutated: object(0,0), object(11,0)
93-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
93+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2470000, storage_rebate: 2470000, non_refundable_storage_fee: 0
9494

9595
task 15, line 87:
9696
//# create-checkpoint
@@ -103,7 +103,7 @@ task 16, lines 89-91:
103103
mutated: object(0,0)
104104
unwrapped: object(2,0)
105105
deleted: object(11,0)
106-
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2196400, storage_rebate: 2470000, non_refundable_storage_fee: 0
106+
gas summary: computation_cost: 1000000, computation_cost_burned: 1000000, storage_cost: 2196400, storage_rebate: 2470000, non_refundable_storage_fee: 0
107107

108108
task 17, line 93:
109109
//# create-checkpoint
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
// Copyright (c) 2026 IOTA Stiftung
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//# init --protocol-version 12 --addresses P0=0x0 --accounts A --simulator --epochs-to-keep 1
5+
6+
//# publish
7+
module P0::m {
8+
public struct Foo has key, store {
9+
id: UID,
10+
value: u64,
11+
}
12+
13+
public struct Wrapper has key, store {
14+
id: UID,
15+
foo: Foo,
16+
}
17+
18+
public fun create_foo(ctx: &mut TxContext): Foo {
19+
Foo { id: object::new(ctx), value: 0 }
20+
}
21+
22+
public fun mutate_foo(foo: &mut Foo) {
23+
foo.value = foo.value + 1;
24+
}
25+
26+
public fun wrap_foo(foo: Foo, ctx: &mut TxContext): Wrapper {
27+
Wrapper { id: object::new(ctx), foo }
28+
}
29+
30+
public fun unwrap_foo(w: Wrapper): Foo {
31+
let Wrapper { id, foo } = w;
32+
object::delete(id);
33+
foo
34+
}
35+
}
36+
37+
//# programmable --sender A --inputs @A
38+
//> 0: P0::m::create_foo();
39+
//> TransferObjects([Result(0)], Input(0))
40+
41+
//# create-checkpoint
42+
43+
//# programmable --sender A --inputs @A object(2,0)
44+
//> 0: P0::m::wrap_foo(Input(1));
45+
//> TransferObjects([Result(0)], Input(0))
46+
47+
//# transfer-object 4,0 --sender A --recipient A
48+
49+
//# transfer-object 4,0 --sender A --recipient A
50+
51+
//# transfer-object 4,0 --sender A --recipient A
52+
53+
//# create-checkpoint
54+
55+
//# advance-epoch
56+
57+
//# programmable --sender A --inputs @A object(4,0)
58+
//> 0: P0::m::unwrap_foo(Input(1));
59+
//> TransferObjects([Result(0)], Input(0))
60+
61+
//# create-checkpoint
62+
63+
//# programmable --sender A --inputs object(2,0)
64+
//> 0: P0::m::mutate_foo(Input(0));
65+
66+
//# create-checkpoint
67+
68+
//# advance-epoch
69+
70+
//# run-graphql
71+
{
72+
real_tombstone_v3: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 3}]}) {
73+
nodes { status version }
74+
}
75+
lamport_v6: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 6}]}) {
76+
nodes { status version }
77+
}
78+
previous_active_v7: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 7}]}) {
79+
nodes { status version }
80+
}
81+
current_active_v8: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 8}]}) {
82+
nodes { status version }
83+
}
84+
}
85+
86+
//# programmable --sender A --inputs @A
87+
//> 0: P0::m::create_foo();
88+
//> TransferObjects([Result(0)], Input(0))
89+
90+
//# create-checkpoint
91+
92+
//# advance-epoch
93+
94+
//# programmable --sender A --inputs @A
95+
//> 0: P0::m::create_foo();
96+
//> TransferObjects([Result(0)], Input(0))
97+
98+
//# create-checkpoint
99+
100+
//# advance-epoch
101+
102+
//# run-graphql --wait-for-checkpoint-pruned 4
103+
{
104+
real_tombstone_v3: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 3}]}) {
105+
nodes { status version }
106+
}
107+
lamport_v6: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 6}]}) {
108+
nodes { status version }
109+
}
110+
previous_active_v7: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 7}]}) {
111+
nodes { status version }
112+
}
113+
current_active_v8: objects(filter: {objectKeys: [{objectId: "@{obj_2_0}", version: 8}]}) {
114+
nodes { status version }
115+
}
116+
}

0 commit comments

Comments
 (0)