Verifiable Off-chain Data Aggregation

Initial proposal: Verifiable Off-chain Data Aggregation

FIP Dicussion: https://github.com/filecoin-project/FIPs/discussions/512

FIP Discussion for on-chain verification: https://github.com/filecoin-project/FIPs/discussions/562

Solution Blueprint: https://docs.google.com/document/d/1M0uj7oUqF3b2CjBhsf_4UbMwrGk9cbGvBbPAT5PHMzE/edit

⭐

A large majority of users onboard data onto the Filecoin network via an aggregator. Today the work done by aggregators is unverifiable and unprovable. The user relies on the aggregator to perform the work correctly and at the same time, it is impossible to prove to a third party that a given piece of data was included in a deal which is a highly requested functionality for FVM.

Team

Tech Lead: Kuba

PM: Honghao Qiu

Retrieval Engineering (Boost): Will Scott, Hannah Howard, Jacob Heun

Actors Engineering: AlexNorth / Kuba

Deal Data processing: Kuba/Aggregator teams

Stakeholders

Estuary (cake) ✅
Dag House: web3.storage / nft.storage (mikeal rogers, Riba) ✅
CryptoNet Lab (Kuba, Nickle) ✅
FVM (Honghao, Aadi, Rahul) ✅
Bedrock: Boost/Indexer/Retrieval (Will, Hannah) ✅

Timeline

Formatting library	Aggregation formatting library to be used by estuary and .storage (daghouse). Batch proof library on top. (Needs to break this into smaller tasks) Note: Once format library is available, share with Boost/Indexer team to prep update	Done	By nv18 upgrade Mostly done (need more test), shared with partner teams	Kuba
Build-in actor change	Implement actor change based on alignment with Alex https://github.com/filecoin-project/builtin-actors/tree/integration/builtin-api	Done	By nv18 upgrade	Aayush
Publish FRC	Publish FRC and approved by community https://github.com/filecoin-project/FIPs/discussions/562	Done	Week of 12/5	Kuba
Verification contract	Contract to validate the proof produced by aggregators on-chain for use in “Storage Products”. Need solidity expertise to implement in Jan since Kuba might not have bandwidth available for this work	Done	Completed by Estuary	Potential help from Kevin Li, when library is available. Kuba to share spec with Kevin
Milestone	Detail	Status	ETA	DRI
Proof Library	Single proof library. Need this for verification contract work to start	Done	By nv18 upgrade Mostly done (need more test), shared with partner teams	Kuba
Estuary integration	Integrate library to provide access to inclusion proof to users	In Progress	Estuary evaluating ETA	Cake (Estuary)
.Storage integration	Steps to integrate: 1) Land the new Filecoin pipeline with Spade in a month 2) Stablize pipeline and resolve any issues with SP in production 3) Integrate inclusion proof (likely within a month of work)	To Do	Need to finish new Filecoin pipeline (Spade) first. Then Daghouse can evaluate ETA.	Mikeal (Daghouse)
Spheron decentralized bundlr integration	Decentralized bundlr to integrate with inclusion proof	In Progress	Microgrant approved	Mitra, Manank
Boost upgrade for the change	To handle subpiece indexing	In Progress	WIP PR	DVD/Will (Bedrock/Boost), we should share library format asap

Links

FIP Discussion

Solution Blueprint

Milestones

‣

🔮 Protocol Opportunities

Name	Status	Quarter	DRI	Date
Built-in actors API for user-programmed actors	✅	2022Q4	Kubuxu
Replacement Sealing Forcing Function & PoRep Security Policy (FIP-47prime)	✅	2023Q3	Kubuxu	October 1, 2022 → June 19, 2023
Verifiable Off-chain Data Aggregation	✅	2023Q2	Kubuxu
Repeatedly updatable storage ("Re-snap")	🔴	2023Q2	Kubuxu
Optimistic Commitment Translation		2023Q4	Kubuxu	August 1, 2023 → December 29, 2023
Removal of Batch Balancer through Onboarding Fees	🟡	2023Q3	Kubuxu	June 12, 2023 → July 25, 2023
L2 Deal Market	🟡	2023Q4	Kubuxu	May 8, 2023 → October 31, 2023

Tasks