feat: continuous rerand by philsippl · Pull Request #2007 · worldcoin/iris-mpc

philsippl · 2026-02-27T09:15:31Z

Spec and implementation of continuous rerandomization of the iris share databases.

Spec is provided in docs/specs/rerandomization.md.

Goals:

Continuous rerandomization of shares in persistent storage (in-memory representation stays untouched)
Rerand performed by separate server than matching server (for network performance reasons)
Minimal impact on normal server (e.g. matching, startup, ...)
Support for GPU and HNSW versions

High level design:

Rerand server operating in chunks over DB
The rerand servers can be at most 1 chunk (or 1 epoch at the boundary) apart from each other
Synchronization with matching server required:
- at startup: while the matching server is loading the db, we freeze the rerand
- when applying modifications: iris share table is always locked, when there is a writer
Rerand servers first write to "staging" schema and then apply in chunks to reduce lock times
Relies on the current system property that modifications are guaranteed to eventually arrive on all parties
- This PR also makes modification handling a bit more robust by deleting only after storing a modification and recovering potentially lost modifications.
related: support additional route ampc-common#71

dkales

A few minor comments, overall strategy seems good. Have not looked at the e2e tests yet.

dkales · 2026-03-03T15:10:07Z

@@ -138,33 +172,75 @@ pub async fn server_main(config: Config) -> Result<()> {

    sync_sqs_queues(&config, &sync_result, &aws_clients).await?;


Does the sync_sqs_queues logic not violate the assumptions on the sqs queues?

to be fair IDK if this ever happens

I think it actually is fine even in theory because of the modification sync happening before, so all nodes sync on them before. So after that, we would only delete modifications that no node has received. But this all relies on the order of the code and hopefully we anyways don't get a queue desync in prod.

@philsippl

Move delete_message in BatchProcessor::process_message from before the per-type processing branch to after it returns Ok. The previous ordering acked the SQS message before the modification was durably persisted, so a crash in that window silently dropped the message. After this change, Err from inner processing propagates without acking, so SQS visibility- timeout redelivery retries the message. The post-process DeleteMessage call is best-effort: if it transiently fails after successful processing, log + counter and continue rather than propagate. Propagating would tear down receive_batch_stream (batch.rs:116 — any Err breaks the spawn loop) and discard the assembled BatchQuery for what is a transient ack failure; the modification is already persisted and downstream appliers are idempotent. Two adjacent hardenings on the same persistence flow: - delete_message helper: receipt_handle.unwrap() -> ok_or_else with a new ReceiveRequestError::FailedToMarkRequestAsDeleted variant - modifications_sync.rs (peer-sync apply path): panic!/unwrap() on unknown modification type, missing s3_url, and JoinHandle Result converted to typed Err returns. No happy-path behavior change. Cherry-picked equivalents of changes in #2007 (continuous-rerand branch, @philsippl), translated to current main shape; the rest of #2007 remains parked on ps/cont-rerand. The best-effort delete wrap addresses a codex review point that #2007 does not. Linear: https://linear.app/worldcoin/issue/POP-3781

@philsippl

Move delete_message in BatchProcessor::process_message from before the per-type processing branch to after it returns Ok. The previous ordering acked the SQS message before the modification was durably persisted, so a crash in that window silently dropped the message. After this change, Err from inner processing propagates without acking, so SQS visibility- timeout redelivery retries the message. DeleteMessage itself uses strict `?` propagation. If the ack fails after processing has succeeded, the Err tears down receive_batch_stream (batch.rs:116) — intentional, because the modifications table is not idempotent at the DB boundary (plain INSERT, no request_id key, see iris-mpc-store/src/lib.rs:491-519). The cascade acts as a circuit breaker against duplicate-row accumulation while SQS is unhealthy. Proper modifications-table idempotency is tracked in POP-3897. Two adjacent hardenings on the same persistence flow: - delete_message helper: receipt_handle.unwrap() -> ok_or_else with a new ReceiveRequestError::FailedToMarkRequestAsDeleted variant - modifications_sync.rs (peer-sync apply path): panic!/unwrap() on unknown modification type, missing s3_url, and JoinHandle Result converted to typed Err returns. No happy-path behavior change. Cherry-picked equivalents of changes in #2007 (continuous-rerand branch, @philsippl), translated to current main shape; the rest of #2007 remains parked on ps/cont-rerand. Linear: https://linear.app/worldcoin/issue/POP-3781 Follow-up: https://linear.app/worldcoin/issue/POP-3897

@philsippl

Move delete_message in BatchProcessor::process_message from before the per-type processing branch to after it returns Ok. The previous ordering acked the SQS message before the modification was durably persisted, so a crash in that window silently dropped the message. After this change, Err from inner processing propagates without acking, so SQS visibility- timeout redelivery retries the message. DeleteMessage itself uses strict `?` propagation. If the ack fails after processing has succeeded, the Err tears down receive_batch_stream (batch.rs:116) — intentional, because the modifications table is not idempotent at the DB boundary (plain INSERT, no request_id key, see iris-mpc-store/src/lib.rs:491-519). The cascade acts as a circuit breaker against duplicate-row accumulation while SQS is unhealthy. Proper modifications-table idempotency is tracked in POP-3897. Two adjacent hardenings on the same persistence flow: - delete_message helper: receipt_handle.unwrap() -> ok_or_else with a new ReceiveRequestError::FailedToMarkRequestAsDeleted variant - modifications_sync.rs (peer-sync apply path): panic!/unwrap() on unknown modification type, missing s3_url, and JoinHandle Result converted to typed Err returns. No happy-path behavior change. Cherry-picked equivalents of changes in #2007 (continuous-rerand branch, @philsippl), translated to current main shape; the rest of #2007 remains parked on ps/cont-rerand. Linear: https://linear.app/worldcoin/issue/POP-3781 Follow-up: https://linear.app/worldcoin/issue/POP-3897

@philsippl

Move delete_message in BatchProcessor::process_message from before the per-type processing branch to after it returns Ok. The previous ordering acked the SQS message before the modification was durably persisted, so a crash in that window silently dropped the message. After this change, Err from inner processing propagates without acking, so SQS visibility- timeout redelivery retries the message. DeleteMessage itself uses strict `?` propagation. If the ack fails after processing has succeeded, the Err tears down receive_batch_stream (batch.rs:116) — intentional, because the modifications table is not idempotent at the DB boundary (plain INSERT, no request_id key, see iris-mpc-store/src/lib.rs:491-519). The cascade acts as a circuit breaker against duplicate-row accumulation while SQS is unhealthy. Proper modifications-table idempotency is tracked in POP-3897. Two adjacent hardenings on the same persistence flow: - delete_message helper: receipt_handle.unwrap() -> ok_or_else with a new ReceiveRequestError::FailedToMarkRequestAsDeleted variant - modifications_sync.rs (peer-sync apply path): panic!/unwrap() on unknown modification type, missing s3_url, and JoinHandle Result converted to typed Err returns. No happy-path behavior change. Cherry-picked equivalents of changes in #2007 (continuous-rerand branch, @philsippl), translated to current main shape; the rest of #2007 remains parked on ps/cont-rerand. Linear: https://linear.app/worldcoin/issue/POP-3781 Follow-up: https://linear.app/worldcoin/issue/POP-3897

initial implementation

d61d65b

github-actions Bot added the chore label Feb 27, 2026

semgrep-code-worldcoin Bot reviewed Feb 27, 2026

View reviewed changes

Comment thread iris-mpc-store/src/rerand.rs

semgrep-code-worldcoin Bot reviewed Feb 27, 2026

View reviewed changes

Comment thread iris-mpc-store/src/rerand.rs Outdated

simplfy and fixes

03557fd

philsippl changed the title ~~initial implementation~~ feat: continuous rerand Feb 27, 2026

philsippl added 4 commits February 27, 2026 11:11

clippy and semgrep

262af1a

fixes

b8985bf

migrations for schema

8a017c9

fix chunk sync

ed710ad

philsippl requested a review from dkales February 27, 2026 12:48

philsippl added 13 commits February 27, 2026 14:11

more fixes

b608592

modification fence

d7cfd39

up

42cbfd0

update design

82a306b

clean up

71b3dbd

simplify

7cf445d

update e2e

d7e02c9

up spec

53da85f

fmt + clippy

68a593c

Merge branch 'main' into ps/cont-rerand

7d2543b

run in ci

f0479f9

up

4ccf85e

fix mermaid

dc94b35

philsippl marked this pull request as ready for review March 1, 2026 22:05

philsippl requested a review from a team as a code owner March 1, 2026 22:05

philsippl added 2 commits March 2, 2026 10:58

fix mermaid

022d8f0

more

3a43859

philsippl requested a review from eaypek-tfh March 2, 2026 13:23

dkales reviewed Mar 3, 2026

View reviewed changes

wojciechsromek and others added 25 commits March 26, 2026 14:45

build iris-mpc image from current branch

ef509e1

deploy iris-mpc with new migrations

ca45866

support different schemas

94460c6

scale down cont rereand

9864889

do not use s3 loader in stage

7b07ce0

verification binary

4b98f95

stopping iris-mpc

b28c5f4

scale up rerand

d412c96

scale down rerand

949fe6f

scale up iris-mpc

8bfcde9

scale up rerand

05da344

iris, rerand down

ea07805

handle sigterm

6f409ad

clippy + fmt

4161e46

prevent limits on rand test

95a1a29

public ecr on images

ef4137a

prepare a test with sigterm handle

73959b6

gracefuly close rerand

1a4f04d

deploy iris + rerand

2c6f99a

service down

6f8f291

restore main iris-mpc version

a036712

stop iris

9b2a8c6

run iris

eb4c57d

pr feedback

359bc0d

rerand_enabled config and check

d66d314

carlomazzaferro mentioned this pull request May 27, 2026

[POP-3781] fix: make SQS-modification persistence flow fail-safe #2159

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: continuous rerand#2007

feat: continuous rerand#2007
philsippl wants to merge 80 commits into
mainfrom
ps/cont-rerand

philsippl commented Feb 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

dkales left a comment

Uh oh!

Uh oh!

Uh oh!

dkales Mar 3, 2026

Uh oh!

dkales Mar 3, 2026

Uh oh!

philsippl Mar 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -138,33 +172,75 @@ pub async fn server_main(config: Config) -> Result<()> {

		sync_sqs_queues(&config, &sync_result, &aws_clients).await?;

Conversation

philsippl commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkales left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dkales Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

dkales Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

philsippl Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philsippl commented Feb 27, 2026 •

edited

Loading

philsippl Mar 4, 2026 •

edited

Loading