handle reorgs for proposer preferences by james-prysm · Pull Request #16651 · OffchainLabs/prysm

james-prysm · 2026-04-09T20:53:03Z

What type of PR is this?

Bug fix

What does this PR do? Why is it needed?

The proposer preference cache was keyed by slot only (map[Slot]ProposerPreference) with first-write-wins semantics. During a reorg that changes the proposer shuffling (different RANDAO at epoch boundary), a new proposer's preferences for the same slot would be rejected because the slot already had an entry. This caused:

Gossip dedup blocking the new proposer's preferences
Bid validation using the wrong validator's fee recipient and gas limit
Payload attributes using the wrong fee recipient

Makes the proposer preferences cache reorg-safe by re-keying from slot to (slot, validatorIndex). After a reorg that changes the proposer shuffling, the correct proposer's preferences are resolved dynamically from the head state's proposer lookahead.

Reproducing reorg testing with Kurtosis

To validate this change under real reorgs, add a temporary broadcast delay flag that is NOT part of this PR. Apply the following patch locally, build new images, and run with the kurtosis config below.

1. Patch `config/features/flags.go`

Add a flag variable before DisableDutiesV2:

reorgTestBroadcastDelay = &cli.DurationFlag{
    Name:  "reorg-test-broadcast-delay",
    Usage: "(Testing): Delays P2P block broadcast by this duration while processing the block locally first.",
}

Add reorgTestBroadcastDelay, to the BeaconChainFlags slice.

2. Patch `config/features/config.go`

Add a field to the Flags struct:

ReorgTestBroadcastDelay time.Duration

Add to ConfigureBeaconChain() before Init(cfg):

if ctx.IsSet(reorgTestBroadcastDelay.Name) {
    cfg.ReorgTestBroadcastDelay = ctx.Duration(reorgTestBroadcastDelay.Name)
}

3. Patch `beacon-chain/rpc/prysm/v1alpha1/validator/proposer.go`

Add "github.com/OffchainLabs/prysm/v7/config/features" to imports.

Replace broadcastReceiveBlock with:

func (vs *Server) broadcastReceiveBlock(ctx context.Context, wg *sync.WaitGroup, block interfaces.SignedBeaconBlock, root [fieldparams.RootLength]byte) error {
    delay := features.Get().ReorgTestBroadcastDelay
    if delay > 0 {
        // Receive locally first, then delay broadcast — creates a divergent fork.
        vs.BlockNotifier.BlockFeed().Send(&feed.Event{
            Type: blockfeed.ReceivedBlock,
            Data: &blockfeed.ReceivedBlockData{SignedBlock: block},
        })
        if err := vs.BlockReceiver.ReceiveBlock(ctx, block, root, nil); err != nil {
            return errors.Wrap(err, "receive block")
        }
        time.Sleep(delay)
        if err := vs.broadcastBlock(ctx, wg, block, root); err != nil {
            return errors.Wrap(err, "broadcast block")
        }
        return nil
    }

    if err := vs.broadcastBlock(ctx, wg, block, root); err != nil {
        return errors.Wrap(err, "broadcast block")
    }
    vs.BlockNotifier.BlockFeed().Send(&feed.Event{
        Type: blockfeed.ReceivedBlock,
        Data: &blockfeed.ReceivedBlockData{SignedBlock: block},
    })
    if err := vs.BlockReceiver.ReceiveBlock(ctx, block, root, nil); err != nil {
        return errors.Wrap(err, "receive block")
    }
    return nil
}

4. Kurtosis config (`gloas-config-4node-reorg.yml`)

participants:
  el:
    el_type: geth
    el_image: ethpandaops/geth:glamsterdam-devnet-0
  cl:
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 2
    vc_extra_params:
      - "--verbosity=debug"
  - el_type: geth
    el_image: ethpandaops/geth:epbs-devnet-0
    cl_type: prysm
    cl_image: gcr.io/offchainlabs/prysm/beacon-chain:latest
    vc_image: gcr.io/offchainlabs/prysm/validator:latest
    supernode: true
    count: 2
    cl_extra_params:
      - "--reorg-test-broadcast-delay=5s"
    vc_extra_params:
      - "--verbosity=debug"

network_params:
  fulu_fork_epoch: 0
  gloas_fork_epoch: 2
  seconds_per_slot: 4
  genesis_delay: 40

additional_services:
  - dora

global_log_level: debug

dora_params:
  image: ethpandaops/dora:gloas-support

5. Run and monitor

kurtosis run --enclave gloas-reorg github.com/ethpandaops/ethereum-package \
  --args-file gloas-config-4node-reorg.yml

# Check for reorgs (delayed nodes experience them)
kurtosis service logs gloas-reorg cl-3-prysm-geth --all 2>&1 | grep "Chain reorg occurred"

# Check proposer preferences are flowing
kurtosis service logs gloas-reorg cl-1-prysm-geth --all 2>&1 | grep "Processed signed proposer"

# Key log: cache accepted a different validator's preference for a slot that already had one.
# This is the reorg-safety signal — proves the (slot, validatorIndex) keying works.
kurtosis service logs gloas-reorg cl-3-prysm-geth --all 2>&1 | grep "possible reorg"

With 4-second slots and a 5-second delay, the delayed nodes' blocks arrive after the next slot begins, reliably triggering reorgs (depth 2-3).

Logs to look for

Log message	Source	Meaning
`Chain reorg occurred`	`blockchain`	A reorg happened on this node
`New proposer preference for slot that already has a different validator (possible reorg)`	`cache`	The cache accepted a second validator's preference for the same slot — proves the reorg-safe keying works
`Processed signed proposer preferences`	`rpc/validator`	VC submitted preferences via RPC (shows broadcast/duplicate/total counts)

Which issues(s) does this PR fix?

Fixes ##16616

related specs
ethereum/consensus-specs#5196
ethereum/consensus-specs#5190
ethereum/consensus-specs#5191

Other notes for review

Acknowledgements

I have read CONTRIBUTING.md.
I have included a uniquely named changelog fragment file.
I have added a description with sufficient context for reviewers to understand this PR.
I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

Squashed cherry-pick of PR #16651 onto glamsterdam-devnet-2.

james-prysm · 2026-05-07T19:05:55Z

+	// NextSlotState may return a state at slot < boundarySlot when the
+	// dependent block was at an earlier slot (empty boundary slot); only use it
+	// if it lands exactly on the boundary, otherwise fall through to load+advance.
+	if cached := transition.NextSlotState(dependentRoot[:], boundarySlot); cached != nil && cached.Slot() == boundarySlot {


for reviewer please take a look at this to see if it makes sense, this is for getting proposer preference messages at the epoch boundary when the head state could still be behind so we need to use the next slot state cache

terencechain · 2026-05-09T02:36:18Z

+// state.block_roots[start_slot(epoch(slot)-1) - 1]. Returns
+// ErrProposerDependentRootUnderflow when the proposal epoch is < 2; the spec's
+// fallback to the genesis block root is the caller's responsibility.
+func ProposerDependentRoot(st state.ReadOnlyBeaconState, slot primitives.Slot) ([32]byte, error) {


why not make this a state getter like st.ProposerDependentRoot? you are accessing state.BlockRootAtIndex at the end anyway

thanks for the suggestion, made it a state getter in 9e3126f

terencechain · 2026-05-09T02:37:20Z

+		return cache.TrackedValidator{}, false
+	}
 	var feeRecipient primitives.ExecutionAddress
 	copy(feeRecipient[:], pref.FeeRecipient)


do we really need to copy here?

fixed in 9e3126f

terencechain · 2026-05-09T02:39:42Z

+// trackedProposer now anchors preferences on dependent_root derived from the
+// passed state (state.block_roots lookup). At slot 0 the lookup underflows so
+// proposerPreference falls back to the no-cache path; cached-preference
+// behavior is exercised end-to-end by the gossip and bid validation tests
+// under beacon-chain/sync.


are those comments helpful? they seem like mostly used by AI

no its not sorry that was a miss on personal review , fixed in 9e3126f

terencechain · 2026-05-09T02:42:10Z

+	dependentRoot [32]byte,
+	slot primitives.Slot,
+	validatorIndex primitives.ValidatorIndex,
+	feeRecipient []byte,
+	gasLimit uint64,


that's enough arguments should we just pass in the preference

terencechain · 2026-05-09T02:52:02Z

 			)
 		}

+		valIdx := msg.Message.ValidatorIndex


move this next to L94?

done 9e3126f

terencechain · 2026-05-09T02:52:15Z

+		var dependentRoot [fieldparams.RootLength]byte
+		copy(dependentRoot[:], msg.Message.DependentRoot)


do we need to copy?

fixed in 9e3126f

terencechain · 2026-05-09T02:57:01Z

+	epoch := slots.ToEpoch(slot)
+	if epoch < 2 {
+		root, err := s.cfg.beaconDB.GenesisBlockRoot(ctx)
+		if err != nil {
+			return [32]byte{}, errors.Wrap(err, "genesis block root")
+		}
+		return root, nil
+	}


can we add this to DependentRootForEpoch rather than doing it for all the callers?

changed in 9e3126f

terencechain · 2026-05-09T03:01:16Z

+
+	dependentRoot := bytesutil.ToBytes32(signedPreferences.Message.DependentRoot)
+	// [IGNORE] block with root preferences.dependent_root has been seen.
+	if !s.cfg.chain.InForkchoice(dependentRoot) && !s.cfg.beaconDB.HasBlock(ctx, dependentRoot) {


why put this here rather than part of verifier package.. and we can reuse similar method there

terencechain · 2026-05-09T03:02:34Z

+	// NextSlotState may return a state at slot < boundarySlot when the
+	// dependent block was at an earlier slot (empty boundary slot); only use it
+	// if it lands exactly on the boundary, otherwise fall through to load+advance.
+	if cached := transition.NextSlotState(dependentRoot[:], boundarySlot); cached != nil && cached.Slot() == boundarySlot {


why use the next slot state cache here, it will surely be a misse right because we are getting an old boundary slot?

i tried to isolate it more, this is for the case of getting the a proposer preference for the next epoch when our head is not up to date for slot 0 , also a part of 9e3126f

wip

5863ee0

james-prysm added bug-fix gloas labels Apr 9, 2026

github-project-automation Bot added this to Gloas Apr 9, 2026

github-project-automation Bot moved this to Unassigned in Gloas Apr 9, 2026

james-prysm and others added 3 commits April 10, 2026 12:48

Merge branch 'develop' into propose-preferences-reorgs

ed48b47

adding unit test and logging for reorg

f234fe8

Merge branch 'develop' into propose-preferences-reorgs

0d6b722

james-prysm requested a review from terencechain April 10, 2026 21:55

james-prysm marked this pull request as ready for review April 10, 2026 21:55

james-prysm and others added 10 commits April 10, 2026 16:59

changelog

cc135d8

Merge branch 'develop' into propose-preferences-reorgs

ee262c4

bad conflict resolution

a6e9f04

Merge branch 'develop' into propose-preferences-reorgs

e408137

force rebuilding proposer preferences fix

2427052

improving the force setting

662a960

Merge branch 'develop' into propose-preferences-reorgs

0901baa

Merge branch 'develop' into propose-preferences-reorgs

0d23c08

Merge branch 'develop' into propose-preferences-reorgs

6e1c388

Merge branch 'develop' into propose-preferences-reorgs

414ab83

james-prysm marked this pull request as draft April 29, 2026 07:52

james-prysm and others added 8 commits April 29, 2026 10:18

adds pr 5190 on consensus

a05bbef

wip

0311137

self review

3fb26e4

gofmt

2110320

self review

70d4ec9

moving dependent root helper so it can be reused

5b6d97d

more self review use the nextsatecache when available

a5dfcc1

Merge branch 'develop' into propose-preferences-reorgs

1a6cad3

james-prysm added a commit that referenced this pull request Apr 30, 2026

handle reorgs for proposer preferences (#16651)

822c77b

Squashed cherry-pick of PR #16651 onto glamsterdam-devnet-2.

james-prysm added a commit that referenced this pull request Apr 30, 2026

handle reorgs for proposer preferences (#16651)

87da8c3

Squashed cherry-pick of PR #16651 onto glamsterdam-devnet-2.

james-prysm and others added 4 commits May 1, 2026 06:15

Merge branch 'develop' into propose-preferences-reorgs

14b30d8

Merge branch 'develop' into propose-preferences-reorgs

2a91c3d

Merge branch 'develop' into propose-preferences-reorgs

222e9e3

fixing unit test

81a49bb

james-prysm marked this pull request as ready for review May 7, 2026 19:04

james-prysm commented May 7, 2026

View reviewed changes

james-prysm added 3 commits May 7, 2026 15:15

optimization on self review

64c05b7

deal with regression

15fad52

deal with epoch boundary bug with proposer dependent root

168995d

james-prysm commented May 7, 2026

View reviewed changes

Comment thread beacon-chain/sync/validate_execution_payload_bid.go Outdated

james-prysm and others added 6 commits May 7, 2026 15:50

updating comment

77259df

using a better way to return proposer dependent root

fd7659b

fixing comments

9b682ae

dropping guard that's not needed anymore

32d8f06

Merge branch 'develop' into propose-preferences-reorgs

cd1bd25

fixing broken conflict resolution

11b75d8

terencechain reviewed May 9, 2026

View reviewed changes

james-prysm and others added 6 commits May 11, 2026 09:50

Merge branch 'develop' into propose-preferences-reorgs

be46434

addressing terence's comments

9e3126f

fixing linting

889a437

reverting change after self review

8124b8f

self review

7d64b9b

Merge branch 'develop' into propose-preferences-reorgs

2a1f28e

		var dependentRoot [fieldparams.RootLength]byte
		copy(dependentRoot[:], msg.Message.DependentRoot)

Conversation

james-prysm commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reproducing reorg testing with Kurtosis

1. Patch config/features/flags.go

2. Patch config/features/config.go

3. Patch beacon-chain/rpc/prysm/v1alpha1/validator/proposer.go

4. Kurtosis config (gloas-config-4node-reorg.yml)

5. Run and monitor

Logs to look for

Uh oh!

james-prysm May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-prysm May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

james-prysm commented Apr 9, 2026 •

edited

Loading

1. Patch `config/features/flags.go`

2. Patch `config/features/config.go`

3. Patch `beacon-chain/rpc/prysm/v1alpha1/validator/proposer.go`

4. Kurtosis config (`gloas-config-4node-reorg.yml`)

james-prysm May 7, 2026 •

edited

Loading

james-prysm May 11, 2026 •

edited

Loading