Alex NorthThis is a draft of https://github.com/filecoin-project/FIPs/discussions/298 rebasing it on top of FIL+ indefinite term limits, which makes all the changes to the verified registry actors.
This replaces Architecture for programmable storage markets.
TODO:
- Decide whether to support multiple notified actors (i.e. markets) per data piece
- Can we design an incremental rollout where old methods keep working initially?
Abstract
This proposal is a refactoring of the APIs and interactions between the storage miner actor, built-in storage market actor, and Filecoin Plus verified registry actor to provide a platform for alternative storage markets, incentive schemes, and other sector-content-related applications to be built on the FVM.
The thrust of the architecture is a decoupling and separation of concerns between the three actors. Notable points include:
- Both clients and miner actors can interact with the verified registry directly to allocate and claim data cap allowances. Market actors are not necessarily involved, but can act as a delegates (this property comes from FIL+ indefinite term limits).
- The built-in market actor is no longer used to calculate the sector data commitment, quality-adjusted power, or other consensus inputs. The miner actor takes all trusted computations.
- Storage provider operators specify deal metadata at proof-of-replication instead of fetching it from the market actor during sector pre-commit. The miner actor forwards it to the market. Deal metadata is really just piece metadata, which could be for a market or any other application.
- The market actor is informed of piece commitments into sectors, but has no power to deny them. Other actors could similarly be informed of changes to sector content.
This proposal requires FIP-0034 simplifying pre-commit deposit as pre-requisite.
This proposal is complemented by to miner←→market registrations: if/when we support updatable sector content we would need such a notification mechanism.
Motivation
High level motivation for why to provide a platform for alternative storage markets and other content-related contracts is given in #241.
We need to change the storage market contract architecture because the current structure does not provide the information and hooks that would be needed by new applications to implement storage-deal-like functionality. It privileges the built-in market actor as the one which the storage miner actor consults. In order to enable the development of useful storage-related applications and features, we need to break the tight coupling between these built-in actors, expose composable on-chain primitives, and make the capabilities of the built-in market actor available to user-programmable contracts.
Goals of this architecture proposal include:
- Remove any trust in market actors (including the built-in one) for network-critical things like storage power. Markets are responsible only for maintaining their own internal state and application policies.
- Prepare a path to the implementation of features like deal renewal/extension and transfer without significant changes to built-in actors’ APIs.
- Support development of alternative markets, including ability to broker deals for verified data.
- Retain the current economic structures, incentive, pledge requirements etc.
It will be more-or-less impossible to remove or change any exposed API after the first user-programmed contracts are deployed to the Filecoin mainnet, without breaking them. Thus, built-in actors should be quite conservative in what they export, as they will be bound to support it for a long time (see #401).
Specification
The following new APIs only include the methods related to storage onboarding and deals. Assume the remainder of these actors’ APIs remain as-is.
This proposal is written assuming that the changes described in FIP-0034 are already in place.
Verified registry actor
See implementation notes in FIL+ indefinite term limits
Storage miner actor
State
The miner actor no longer persists deal IDs in state at all.
type SectorPreCommitOnChainInfo struct {
Info {
SealProof abi.RegisteredSealProof
SectorNumber abi.SectorNumber
SealedCID cid.Cid // CommR
SealRandEpoch abi.ChainEpoch
// Sector pre-commit no longer references deals.
DealIDs []abi.DealID
Expiration abi.ChainEpoch
UnsealedCID cid.Cid // CommD
}
PreCommitDeposit abi.TokenAmount
PreCommitEpoch abi.ChainEpoch
}
type SectorOnChainInfo struct {
SectorNumber abi.SectorNumber
SealProof abi.RegisteredSealProof
SealedCID cid.Cid // CommR
// Sector info no longer references deals either.
DealIDs []abi.DealID
Activation abi.ChainEpoch
Expiration abi.ChainEpoch
DealWeight abi.DealWeight
VerifiedDealWeight abi.DealWeight
InitialPledge abi.TokenAmount
ExpectedDayReward abi.TokenAmount
ExpectedStoragePledge abi.TokenAmount
ReplacedSectorAge abi.ChainEpoch // Can deprecate 140 days after nv15
ReplacedDayReward abi.TokenAmount // Can deprecate 140 days after nv15
SectorKeyCID *cid.Cid
}Methods
Deal IDs are no longer specified at sector pre-commitment. Instead, the pieces of data that comprise a sector’s data are declared as a “manifest” when the replica or update is proven.
The provider must commit to the both the sealed and unsealed CIDs at pre-commit. At PoRep the miner actor then checks that the unsealed CID matches that computed from the piece CIDs (and an optional implicit zero piece for padding), and the PoRep proof verifies it corresponds to the sealed CID also committed earlier. The ReplicaUpdate flow commits and proves at the same time.
Each piece may specify a market contract address and market-specific identifier which it satisfies. The miner will synchronously notify that market address that the piece has been committed. The miner will pay no attention to the success or otherwise of that notification to the market: the sector will be committed regardless. Note that there is no requirement for the actor that is notified to be some kind of marketplace: it is just a contract that is notified when the piece is committed.
Each piece may also declare a FIL+ verified data cap allocation ID that it satisfies. The miner actor will claim that allocation directly from the verified registry actor and, if successful, calculate quality-adjusted power according to the piece’s size.
type SectorPreCommitInfo struct {
SealProof abi.RegisteredSealProof
SectorNumber abi.SectorNumber
UnsealedCID cid.Cid // CommD
SealedCID cid.Cid // CommR
SealRandEpoch abi.ChainEpoch
// No deals specified during pre-commit.
DealIDs []abi.DealID
Expiration abi.ChainEpoch
}
type PreCommitSectorBatchParams struct {
Sectors []SectorPreCommitInfo
}
func (Actor) PreCommitSectorBatch(params *PreCommitSectorBatchParams)
// Sector content is declared at PoRep
// (previously, this data was fetched from the built-in market actor).
type SectorManifest struct {
SectorNumber uint64
// Declaration of pieces that make up the sector data.
// Implicit "zero" piece to fill remaining capacity need not be specified.
Pieces []PieceManifest
}
type PieceManifest struct {
PieceCID cid.CID // CommP
Size PaddedPieceSize
// Each piece may specify a FIL+ verified data cap allocation ID.
// If the piece is a sub-piece of a larger allocation, also specifies
// the range covered and an inclusion proof in the allocation.
VerifiedDataAllocationID uint64 // Optional
VerifiedDataRangeStart uint64
VerifiedDataRangeEnd uint64
VerifiedDataInclusionProof []byte // ???
// Each piece may specify an actor and piece ID to notify on commitment.
// TODO: an array of these tuples for multiple downstream actors?
NotifyAddress address.Address // Optional
NotifyPieceID uint64 // E.g. deal ID, optional
}
type ProveCommitAggregateParams struct {
SectorNumbers bitfield.BitField
AggregateProof []byte
// Declaration of data for each non-empty sector.
Manifests []SectorManifest
}
func (Actor) ProveCommitSectorAggregate(params *ProveCommitAggregateParams)
type ReplicaUpdate struct {
SectorID abi.SectorNumber
Deadline uint64
Partition uint64
NewSealedSectorCID cid.Cid // CommR
// Deals are replaced by sector manifest.
Deals []abi.DealID
UpdateProofType abi.RegisteredUpdateProof
ReplicaProof []byte
Manifest SectorManifest
}
type ProveReplicaUpdatesParams struct {
Updates []ReplicaUpdate
}
func (Actor) ProveReplicaUpdate(params *ReplicaUpdateParams)Storage market actor
The built-in market actor loses its privileged role (except for continued use of the Cron actor), and becomes just one of many possible market mechanisms that could be implemented as user contracts.
The basic SectorDeals map in state and SectorContentAdded method described below supports deals within the current restrictions of the market actor: deals cannot exceed their sector’s committed lifetime, the data cannot be moved between sectors, deals cannot be extended. Extension methods are outlined to show how this API can be extended to break those limitations in the future.
Deal negotiation
The built-in storage market actor is pre-authorized to act as a delegate for verified registry clients, and thus allocates and claims data cap according to the verified field already present in the deal proposal.
The current deal negotiation structures do not have space to communicate an existing data cap allocation ID, but it could be specified out of band.
State
type SectorDeals struct {
SectorExpiration abi.ChainEpoch
Deals []abi.DealId
}
type State struct {
// All existing state as today.
// Maps sector IDs to deals, per storage provider.
// This supports the actor keeping track of whether deals remain satisfied
// as data moves between sectors in the future.
SectorDeals cid.CID // HAMT[address.Address]AMT[abi.SectorNumber]SectorDeals
// TODO: should we also write the deal's sector ID into the deal state?
///// Extensions
// A queue of sector IDs by their expiration epoch.
// Records sectors for which the market must check if deals have been
// re-commited to new sectors before the original one expires.
// This supports the market accepting deals that have a term longer
// than the lifespan of the sector they are originally sealed into.
// No need to track where the deal fits inside a sector's lifespan:
// the market will be explicitly notified if the sector terminates early.
SectorExpirations cid.CID // AMT[abi.ChainEpoch]bitfield.Bitfield
}Methods
The market actor becomes an example for other actors. While the deal-negotiation methods need not be standardised, the method/s by which the storage miner actor informs the market of changes to sector content must be standards.
// Publishing storage deals is unchanged.
// This method is NOT STANDARDISED across markets, as they may have
// different deal proposal structures.
func (Actor) PublishStorageDeals(params *PublishStorageDealsParams)
type PieceInfo struct {
// The piece data commitment and size, verified by the miner actor.
CID CID
Size PaddedPieceSize
// Market-specific identifier for this piece, provided by the provider.
// E.g. the deal ID which the provider claims this piece satisfies.
// TODO: should we use a more flexibile []byte key?
PieceID uint64
}
type SectorContentAddedParams struct {
Sector abi.SectorNumber
Expiration abi.ChainEpoch
Pieces []PieceInfo
}
// Called by the miner actor when non-zero data is proven in a sector,
// either at initial PoRep, or with a replica update (Snap).
// Checks that the piece CID and size match the nominated deal, and consider
// the deal active if so.
// This replaces the current ActivateDeals.
// Without extensions, rejects attempts to satisfy a deal if the deal term
// is beyond the sector expiration.
// With state.SectorExpirations state member, can support deals with arbitrary terms.
// Note: will probably implement a batch version, single use shows for simplicity.
// TODO: add a generic return value to be threaded back through the miner
// to the caller?
// STANDARD METHOD: other market actors must implement the same method
// signature if they expect to receive calls from the miner actor when
// data is committed.
func (Actor) SectorContentAdded(params *SectorContentAddedParams)
type SectorTerminatedParams struct {
Epoch abi.ChainEpoch
Sectors []abi.SectorNumber
}
// Called by the miner actor when a sector is terminated ahead of its
// scheduled expiration.
// Pending a mechanism for fine-grained event registrations, this is called
// for all unscheduled sector terminations, whether or not the sector holds
// a deal for this market (because the miner doesn't know).
// The market actor looks up the sector->deal map and marks any
// deals found as terminated (but defers further processing until cron).
// This replaces OnMinerSectorsTerminate().
func (a Actor) SectorTerminated(rt Runtime, params *OnMinerSectorsTerminate)
///// Extensions
// Note: we can immediately support deal extension up to the *original*
// expiration of the sector with no further state (but a new market method).
type SectorExpirationChangedParams {
Sector abi.SectorNumber
OldExpiration abi.ChainEpoch
NewExpiration abi.ChainEpoch
// TODO: add parameters for deals to extend at the same time?
}
// Called by the miner when a sector's expiration epoch is extended.
// Can support extending a deal up to the sector's *new* expiration.
// Note: will probably implement a batch version, single use shows for simplicity.
// STANDARD METHOD.
func (Actor) SectorExpirationChanged(params *SectorExpirationChangedParams)
// With state.SectorExpirations, can support deal extension to arbitrary terms,
// beyond a sector's currently committed lifespan, by the provider
// re-committing the same data into a new sector.
// Deal extension is a simple case of more general deal re-negotiation.
// Such methods are NOT STANDARDISED, and depend on individual market features.
// Details omitted here.
// If we make sector content updateable, a new SectorContentChanged
// method implicitly replaces any pieces not explicitly mentioned.
// In order to track pieces between sectors, either
// (a) caller must also specify the previous sector ID for a piece, or
// (b) store deal sector ID in deal state object.
// STANDARD METHOD.
func (Actor) SectorContentUpdated(params *SectorContentUpdatedParams)Control & data Flows
The standard end-to-end verified deal onboarding flow currently looks like this:
SP → Market.PublishStorageDeals
SP → Miner.PreCommitSector[Batch]
Miner → Market.VerifyDealsForActivation
SP → Miner.ProveCommitSector[Aggregate]
Miner → Market.ComputeDataCommitment
Miner → Market.ActivateDealsAfter this architecture, it looks like this:
[Optonal] Client → VerifReg.AllocateDataCap
SP → Market.PublishStorageDeals (~vo)
Market → VerifReg.AllocateDataCap (~nv)
SP → Miner.PreCommitSector[Batch]
SP → Miner.ProveCommitSector[Aggregate]
Miner → VerifReg.ClaimAllocation (~nv)
Miner → Market.SectorContentAdded (~vo)A client’s call to AllocateDataCap is necessary only if they have not delegated a market actor to allocate data cap on their behalf.
The storage provider passes the FIL+ allocation, market address, and deal IDs when proving a sector. The miner actor checks that these values match the sector content, and then passes them on as trusted values to the verified registry and market actors.
Frequent flows will be even simpler.
- For a deal that is not verified, the verified registry actor is not involved, so neither the client, market, nor the miner actor need to call to allocate or claim data cap. These calls are marked as
(~nv) - For a verified data cap without a deal, no market is involved, so the provider need not publish the deals nor call the market to inform it of the new committed content. These calls are marked as
(~vo).
Migration
A state migration is required to realise this architecture in mainnet. Details to be fleshed out, but in brief:
- Move the sector→deal mappings from miner sector info to market actor
- Leave existing QA power, pledge, penalty values in place for existing sectors
Discussion
Limitations
- Only a single “market” contract and identifier supported for each piece, notified when committed. An intermediate contract could proxy for multiple markets, but clumsily. Should we support multiple notifiers per piece?
Extensions
These APIs are designed with extension to more powerful and flexible programmable storage in mind. In general these will require new methods to be added.
- Deal extension within sector lifespan
- Extension of deal terms up to the end of a newly-extended sector commitment can be achieved after (or concurrent with) the miner calling
market.SectorExpirationChanged. New methods on miner and/or market actor would be needed to support this. - Deal extension beyond sector lifespan
- Extension of deal terms, possibly indefinitely, can be made possible by committing the same deal data into a new sector. Additional parameters need to communicate which sector the piece is moving from (or recording sector id in deal state).
- Updatable sector content
- Sector content is presently write-once, primarily due to restrictions imposed by the QA power mechanism. When we resolve those restrictions, miners can update the content in sectors after the initial seal or replica update. To support this safely, markets must be provided a notification whenever the content of a sector that is (was) supporting deals from that market changes. Something like the miner→market registrations proposal could achieve this, but requires some detailed design to find a good simplicity/efficiency trade-off.
- Transfer between providers
- New methods could support the transfer of sector content between storage providers, with the appropriate notifications to market actors. Markets would set their own policy as to whether such transfer is permitted under the deal terms.
- Fault notifications.
- Markets may wish to be notified when a sector becomes temporarily faulty. The miner→market registrations proposal may satisfy this need.
- The FIL+ premium proposal also requires this.
- Async sector content proofs, with new piece inclusion proof
- At present, a miner must activate deals synchronously with proving the sector content. This is because the unsealed sector CID (”CommD”) is not stored on chain, but is required in order to demonstrate the inclusion of any specific data within the sector. Future work may provide an inclusion proof that can be evaluated after sealing in order to demonstrate to a market (or the verified registry actor) that a provider is storing some data some time after the sector is sealed.