Initial proposal: Verifiable Off-chain Data Aggregation
FIP Dicussion: https://github.com/filecoin-project/FIPs/discussions/512
FIP Discussion for on-chain verification: https://github.com/filecoin-project/FIPs/discussions/562
Solution Blueprint: https://docs.google.com/document/d/1M0uj7oUqF3b2CjBhsf_4UbMwrGk9cbGvBbPAT5PHMzE/edit
A large majority of users onboard data onto the Filecoin network via an
aggregator. Today the work done by aggregators is unverifiable and
unprovable.
The user relies on the aggregator to perform the work correctly and at
the same time, it is impossible to prove to a third party that a given
piece of data was included in a deal which is a highly requested
functionality for FVM.
Team
Tech Lead: Kuba
PM: Honghao Qiu
Retrieval Engineering (Boost): Will Scott, Hannah Howard, Jacob Heun
Actors Engineering: AlexNorth / Kuba
Deal Data processing: Kuba/Aggregator teams
Stakeholders
- Estuary (cake) ✅
- Dag House: web3.storage / nft.storage (mikeal rogers, Riba) ✅
- CryptoNet Lab (Kuba, Nickle) ✅
- FVM (Honghao, Aadi, Rahul) ✅
- Bedrock: Boost/Indexer/Retrieval (Will, Hannah) ✅
Timeline
Formatting library | Aggregation formatting library to be used by estuary and .storage (daghouse). Batch proof library on top.
(Needs to break this into smaller tasks)
Note:
Once format library is available, share with Boost/Indexer team to prep update | Done | By nv18 upgrade
Mostly done (need more test), shared with partner teams | Kuba |
Build-in actor change | Implement actor change based on alignment with Alex
https://github.com/filecoin-project/builtin-actors/tree/integration/builtin-api | Done | By nv18 upgrade | Aayush |
Publish FRC | Publish FRC and approved by community
https://github.com/filecoin-project/FIPs/discussions/562 | Done | Week of 12/5 | Kuba |
Verification contract | Contract to validate the proof produced by aggregators on-chain for use in “Storage Products”.
Need solidity expertise to implement in Jan since Kuba might not have bandwidth available for this work | Done | Completed by Estuary | Potential help from Kevin Li, when library is available.
Kuba to share spec with Kevin |
Milestone | Detail | Status | ETA | DRI |
Proof Library | Single proof library. Need this for verification contract work to start | Done | By nv18 upgrade
Mostly done (need more test), shared with partner teams | Kuba |
Estuary integration | Integrate library to provide access to inclusion proof to users | In Progress | Estuary evaluating ETA | Cake (Estuary) |
.Storage integration | Steps to integrate:
1) Land the new Filecoin pipeline with Spade in a month
2) Stablize pipeline and resolve any issues with SP in production
3) Integrate inclusion proof (likely within a month of work) | To Do | Need to finish new Filecoin pipeline (Spade) first.
Then Daghouse can evaluate ETA. | Mikeal (Daghouse) |
Spheron decentralized bundlr integration | Decentralized bundlr to integrate with inclusion proof | In Progress | Microgrant approved | Mitra, Manank |
Boost upgrade for the change | To handle subpiece indexing | In Progress | DVD/Will (Bedrock/Boost), we should share library format asap |
Links
Milestones
‣
🔮 Protocol Opportunities
| Name | Status | Quarter | DRI | Date | 🎒 Team backlog ㊙️ |
|---|---|---|---|---|---|
✅ | 2023Q1 |  | February 20, 2023 → February 24, 2023 | ||
✅ | 2023Q2 | | |||
✅ | 2023Q1 | | February 27, 2023 → March 10, 2023 | ||
✅ | 2023Q1 | | February 20, 2023 → February 24, 2023 | ||
✅ | | ||||
✅ | 2022Q4 | | March 19, 2023 → March 19, 2023 | ||
✅ | 2023Q2 | | February 12, 2023 → March 24, 2023 | ||
✅ | 2023Q1 | | February 12, 2023 → March 17, 2023 | ||
🔴 | 2023Q1 | | |||
🔴 | 2023Q2 | | |||
✅ | 2022Q4 | | |||
✅ | 2023Q3 | | October 1, 2022 → June 19, 2023 | ||
✅ | 2023Q2 | | |||
✅ | 2023Q2 | | |||
✅ | 2023Q1 | | February 13, 2023 → February 17, 2023 | ||
✅ | 2023Q1 | | February 20, 2023 → February 24, 2023 | ||
🔴 | 2023Q2 | | |||
✅ | 2022Q4 | | October 1, 2022 → December 22, 2022 | ||
🔴 | 2023Q1 | | |||
🔵 | 2023Q3 | | |||
🔵 | 2023Q3 | | |||
Not started | 2023Q4 | | August 1, 2023 → December 29, 2023 | ||
🟡 | 2023Q3 | | June 12, 2023 → July 25, 2023 | ||
Not started | 2023Q2 | | June 5, 2023 → June 16, 2023 | ||
🔵 | 2023Q2 | | May 15, 2023 → June 6, 2023 | ||
🔵 | 2023Q2 | | May 29, 2023 → June 17, 2023 | ||
🔵 | 2023Q3 | | May 15, 2023 → June 29, 2023 | ||
🔵 | 2023Q3 | | May 15, 2023 → June 29, 2023 | ||
🔵 | 2023Q2 | | March 15, 2023 → June 15, 2023 | ||
✅ | 2023Q2 | | May 29, 2023 → June 2, 2023 | ||
🟡 | 2023Q4 | | May 8, 2023 → October 31, 2023 | ||
✅ | 2023Q2 | | |||
✅ | 2023Q2 | | April 10, 2023 → June 19, 2023 | ||
✅ | 2023Q1 | | January 10, 2023 → March 15, 2023 | ||
✅ | 2023Q2 | | |||
✅ | 2023Q2 | | |||
✅ | 2023Q2 | | February 27, 2023 → March 10, 2023 | ||
✅ | 2023Q2 | | February 27, 2023 → March 10, 2023 | ||
✅ | | February 27, 2023 → March 3, 2023 | |||
✅ | | February 13, 2023 → February 24, 2023 |
Tasks