Simple, composable primitives for efficiency and innovation in storage applications and markets.
@Alex North @Deleted User and the Protocol Opportunities Group (with big assist from @Deleted User).
Goals
The storage/data programmability project aims to:
- Enable the development of alternative, fully-capable storage/data applications, including deal markets, including by third parties
- Support heterogenous storage application features, implementations and policies
- Guide standards for interoperable and application primitives and derivatives
- Improve support for Filecoin Plus incentivised storage, as well as alternative storage incentive schemes
With a well-conceived architecture, we should be able to:
- Develop a range of new on-chain storage applications competing on dimensions like efficiency (e.g. L2s), matching (see wish list), features (e.g perpetual deals, transfer), payments (e.g. incremental), governance
- Support on-chain applications and automation such as repair services, brokers, aggregators, collective bargaining
- Support development of storage-related derivatives such as futures, insurance
- Govern markets and other applications independently of network protocol upgrades
See also https://github.com/filecoin-project/FIPs/discussions/241 for a general motivation towards composable market primitives.
Challenges
Today (February 2022), storage powered consensus, the built-in storage deal market, and Filecoin Plus program are tightly coupled. There is no room for a competing market contract or alternative storage/data application that doesn’t sit on top of the built-in market. The built-in market actor is trusted with network-critical operations like calculating sector quality. It’s also inefficient and inflexible. Deals are expensive, incur ongoing maintenance cost, and cannot be extended, transferred, or generally inspected or manipulated in any way on-chain. Sector storage is limited to write-once semantics by the complex quality-adjusted power calculations necessary to update data.
This was fine for network liftoff, when all actor code was trusted by the network and we wanted the simplest thing that could work. But as the FVM enables user-programmable contracts, these limitations will severely limit the possibilities for innovation, if left in place.
Solutions
Our path toward lego-like building blocks comprises a few major legs.
✅ Improvements to the built-in storage market
We hope for the development of a wide variety of market mechanisms, but the built-in one will still carry momentum of usage for some time. There are some simple improvements we can make within its current design.
- ✅ Support contracts as deal clients (
FIP-0035FIP-0044) - [Maybe] Deal extension/renewal, perpetual deals (depends on resolving FIL+ QA power problems)
- ✅ Partially solved by FIP-0045 (the QA power allocation can be extended)
- [Maybe] Support transfer of deals between clients
- [Maybe] Negative pricing for verified data
✅ Decouple Filecoin Plus from from storage deal markets (FIP-0045)
Sector content is coupled to consensus power through the quality-adjusted-power mechanism, as a mechanism for incentivising storage of valuable data. The operation of this mechanism limits the utility of sector storage, generally restricting it to write-once semantics. Verified sector data cannot be replaced, even if the deal for that data has expired. It makes deal extension and transfer difficult.
- Proposal outline: https://github.com/filecoin-project/FIPs/discussions/313
- Detailed design:FIL+ indefinite term limits
- ✅ Accepted FIP-0045, implemented in Network v17
✅ Re-architect miner/market/registry interactions
The largest but most impactful objective is to refactor the interfaces and interactions between the built-in storage miner actor, storage market actor, and FIL+ verified registry in order to permit alternative actors to interact directly.
- ✅ Move the sector→deal mapping from miner state to the state of storage/data/market actors
- ✅ Allow miner to specify data commitment independent of the built-in market
- ✅ Remove network-critical calculations like deal weight from the built-in market actor to miner actor
- ✅ Remove the built-in market actor as a necessary intermediary between the miner actor and FIL+ registry
Solution ideas:
- Direct data onboarding: impacts & work outline
- Free CommD: Unconstrained data applications on Filecoin
- Architecture for programmable storage markets
- Miner market registrations for market fault processing
Define standards interfaces for market interoperability
Markets will be heterogenous, but we nevertheless want to promote interoperability so that higher level components and applications can aggregate across deals from different markets.
- ✅ Investigate and decide how to describe contract interface standards on the FVM (with FVM team)
- Design standards for markets and deals
- Implement these standards with the built-in market actor
Roadmap
A rough sequencing (last updated June 2022).
- 2022 Q1
- ✅ Conceptual design to decouple Filecoin Plus from markets
- ✅ Conceptual design to support alternative storage markets
- 2022 Q2-3
- ✅ Proposals for FVM programmability basics: calling convention, token standards etc
- ✅ (FVM without user-programmability launches)
- 2022 Q4
- ✅ Implementation of decoupling Filecoin Plus from markets
- ✅ Improvements to built-in storage market
- ✅ Define basic APIs for user actors to interact with built-in actors
- 2023 Q4
- ✅ Move deal information out of miner actor
- 2024 Q?
- Support synchronous data activation notifications to user contracts
Programmable storage markets work plan
| Title | Discuss | FIP | Issue | Current step | Starfleet owner | Notes |
|---|---|---|---|---|---|---|
Implemented | D Deleted User | |||||
PR #395 | Implemented | D Deleted User | ||||
Implemented | Expecting IPFSForce to implement this | |||||
Implemented | Lotus team w/Alex | |||||
#298 (old) | Final implementation |  Alex North | ||||
Implemented | D Deleted User | Replaced by standard message authentication | ||||
Pre-FIP discussion | ||||||
Prototype | ||||||
See also the complementary work FVM features & standards work plan
Notes
Below are notes from the Cryptonet Notebook tagged for Storage Programmability
