feat(iota-core): introduce StoreObjectV2 and snapshot V2#11453
Open
bingyanglin wants to merge 2 commits intodevelopfrom
Open
feat(iota-core): introduce StoreObjectV2 and snapshot V2#11453bingyanglin wants to merge 2 commits intodevelopfrom
bingyanglin wants to merge 2 commits intodevelopfrom
Conversation
dd98261 to
e31f6a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
Make new generated formal snapshots carry per-epoch metadata and per-object checkpoint attribution, without requiring archival replay.
Storage schema (
iota-core,iota-types)StoreObjectV2addsprevious_transaction_checkpoint: CheckpointSequenceNumberto each stored object value. V1 rows lift to V2 lazily on read viaStoreObjectWrapper::migrate().SENTINEL_PREVIOUS_TRANSACTION_CHECKPOINT = u64::MAXmarks rows whose containing checkpoint is not yet known (PassthroughCache write path; V1 → V2 lifts; restore path).epoch_info: DBMap<EpochId, EpochInfoEntry>column family onCheckpointStoreTables, sibling ofepoch_last_checkpoint_map. Holds{ last_checkpoint_summary: CertifiedCheckpointSummary, first_checkpoint, end_of_epoch_tx_events: TransactionEvents }per epoch — schema confirmed by indexer team.EpochInfoEntrylives in a newiota-types::epoch_infomodule so producer (iota-core) and consumer (iota-snapshot) share the type.Execution path (
iota-core)ExecutionCacheCommit::build_db_batchgains acheckpoint_seq: CheckpointSequenceNumberparameter, threaded from the executor →WritebackCache::build_db_batch→AuthorityStore::build_db_batch→write_one_transaction_outputs→get_store_object.WritebackCache(default config) buffers tx outputs and flushes atcommit_transaction_outputs; the executor passes the real seq there, so newly-written rows on default-deployed nodes carry real values.PassthroughCachewrites synchronously before the containing checkpoint is determined; stamps the sentinel until PR 2's backfill rewrites those rows.CheckpointExecutor::write_epoch_info_entryfires inside the executor'sis_final_checkpointblock (alongsideinsert_epoch_last_checkpointandaccumulate_epoch). It runs aftercommit_transaction_outputs, so AdvanceEpoch events are durable when read.Snapshot V2 file format (
iota-snapshot)previous_transaction_checkpointtrailer. New magicREFERENCE_FILE_MAGIC_V2 = 0xCAFEBEEF(V1 was0xDEADBEEF); a V1 reader fails fast on the magic.EPOCH_INFOfile carryingEpochInfo::V1(EpochInfoV1 { entries: Vec<Option<EpochInfoEntry>> }), lengthsnapshot_epoch + 1.Noneentries indicate the producer had no row for that epoch (typical only during the PR 1 → PR 2 window). Magic0x9000C001. Sha3-anchored viaFileMetadata::sha3_digestin the MANIFEST.Manifest::V1andManifest::V2shareManifestBody; the BCS variant tag distinguishes them.LiveSetIterV2onAuthorityPerpetualTablesyields(LiveObject, CheckpointSequenceNumber)pairs so the writer can emit the per-record trailer without an extra DB lookup.Cleanup (
iota-core,iota-snapshot,iota-single-node-benchmark,iota-surfer)LiveObject::Wrappedvariant removed. Pre-merge proof:LiveSetIter::store_object_wrapper_to_live_objectonly returnsLiveObject::NormalorNone, and a workspace-wide grep confirmed zero construction sites — everyLiveObject::Wrappedreference was a destructure on input the iterator could never produce. Eightmatcharms across four crates simplified or eliminated. OrphanedWrappedObjecthelper inglobal_state_hasher.rsremoved.Links to any relevant issues
fixes #11254
How the change has been tested