Logo
    💿

    Architecture for programmable storage markets – v2

    Creator
    Alex North
    Created
    Jul 20, 2022 2:52 AM
    Project
    Storage Programmability
    Stage
    Graduated from Notebook

    This is a draft of https://github.com/filecoin-project/FIPs/discussions/298 rebasing it on top of ➕FIL+ indefinite term limits, which makes all the changes to the verified registry actors.

    This replaces 💾Architecture for programmable storage markets.

    TODO:

    • Decide whether to support multiple notified actors (i.e. markets) per data piece
    • Can we design an incremental rollout where old methods keep working initially?

    Abstract

    This proposal is a refactoring of the APIs and interactions between the storage miner actor, built-in storage market actor, and Filecoin Plus verified registry actor to provide a platform for alternative storage markets, incentive schemes, and other sector-content-related applications to be built on the FVM.

    The thrust of the architecture is a decoupling and separation of concerns between the three actors. Notable points include:

    • Both clients and miner actors can interact with the verified registry directly to allocate and claim data cap allowances. Market actors are not necessarily involved, but can act as a delegates (this property comes from ➕FIL+ indefinite term limits).
    • The built-in market actor is no longer used to calculate the sector data commitment, quality-adjusted power, or other consensus inputs. The miner actor takes all trusted computations.
    • Storage provider operators specify deal metadata at proof-of-replication instead of fetching it from the market actor during sector pre-commit. The miner actor forwards it to the market. Deal metadata is really just piece metadata, which could be for a market or any other application.
    • The market actor is informed of piece commitments into sectors, but has no power to deny them. Other actors could similarly be informed of changes to sector content.

    This proposal requires FIP-0034 simplifying pre-commit deposit as pre-requisite.

    This proposal is complemented by to miner←→market registrations: if/when we support updatable sector content we would need such a notification mechanism.

    Motivation

    High level motivation for why to provide a platform for alternative storage markets and other content-related contracts is given in #241.

    We need to change the storage market contract architecture because the current structure does not provide the information and hooks that would be needed by new applications to implement storage-deal-like functionality. It privileges the built-in market actor as the one which the storage miner actor consults. In order to enable the development of useful storage-related applications and features, we need to break the tight coupling between these built-in actors, expose composable on-chain primitives, and make the capabilities of the built-in market actor available to user-programmable contracts.

    Goals of this architecture proposal include:

    • Remove any trust in market actors (including the built-in one) for network-critical things like storage power. Markets are responsible only for maintaining their own internal state and application policies.
    • Prepare a path to the implementation of features like deal renewal/extension and transfer without significant changes to built-in actors’ APIs.
    • Support development of alternative markets, including ability to broker deals for verified data.
    • Retain the current economic structures, incentive, pledge requirements etc.

    It will be more-or-less impossible to remove or change any exposed API after the first user-programmed contracts are deployed to the Filecoin mainnet, without breaking them. Thus, built-in actors should be quite conservative in what they export, as they will be bound to support it for a long time (see #401).

    Specification

    The following new APIs only include the methods related to storage onboarding and deals. Assume the remainder of these actors’ APIs remain as-is.

    This proposal is written assuming that the changes described in FIP-0034 are already in place.

    Verified registry actor

    See implementation notes in ➕FIL+ indefinite term limits

    Storage miner actor

    State

    The miner actor no longer persists deal IDs in state at all.

    Methods

    Deal IDs are no longer specified at sector pre-commitment. Instead, the pieces of data that comprise a sector’s data are declared as a “manifest” when the replica or update is proven.

    The provider must commit to the both the sealed and unsealed CIDs at pre-commit. At PoRep the miner actor then checks that the unsealed CID matches that computed from the piece CIDs (and an optional implicit zero piece for padding), and the PoRep proof verifies it corresponds to the sealed CID also committed earlier. The ReplicaUpdate flow commits and proves at the same time.

    Each piece may specify a market contract address and market-specific identifier which it satisfies. The miner will synchronously notify that market address that the piece has been committed. The miner will pay no attention to the success or otherwise of that notification to the market: the sector will be committed regardless. Note that there is no requirement for the actor that is notified to be some kind of marketplace: it is just a contract that is notified when the piece is committed.

    Each piece may also declare a FIL+ verified data cap allocation ID that it satisfies. The miner actor will claim that allocation directly from the verified registry actor and, if successful, calculate quality-adjusted power according to the piece’s size.

    Storage market actor

    The built-in market actor loses its privileged role (except for continued use of the Cron actor), and becomes just one of many possible market mechanisms that could be implemented as user contracts.

    The basic SectorDeals map in state and SectorContentAdded method described below supports deals within the current restrictions of the market actor: deals cannot exceed their sector’s committed lifetime, the data cannot be moved between sectors, deals cannot be extended. Extension methods are outlined to show how this API can be extended to break those limitations in the future.

    Deal negotiation

    The built-in storage market actor is pre-authorized to act as a delegate for verified registry clients, and thus allocates and claims data cap according to the verified field already present in the deal proposal.

    The current deal negotiation structures do not have space to communicate an existing data cap allocation ID, but it could be specified out of band.

    State

    Methods

    The market actor becomes an example for other actors. While the deal-negotiation methods need not be standardised, the method/s by which the storage miner actor informs the market of changes to sector content must be standards.

    Control & data Flows

    The standard end-to-end verified deal onboarding flow currently looks like this:

    SP → Market.PublishStorageDeals
SP → Miner.PreCommitSector[Batch]
    Miner → Market.VerifyDealsForActivation
SP → Miner.ProveCommitSector[Aggregate]
    Miner → Market.ComputeDataCommitment
    Miner → Market.ActivateDeals

    After this architecture, it looks like this:

    [Optonal] Client → VerifReg.AllocateDataCap
SP → Market.PublishStorageDeals  (~vo)
    Market → VerifReg.AllocateDataCap  (~nv)
SP → Miner.PreCommitSector[Batch]
SP → Miner.ProveCommitSector[Aggregate]
    Miner → VerifReg.ClaimAllocation  (~nv)
    Miner → Market.SectorContentAdded  (~vo)

    A client’s call to AllocateDataCap is necessary only if they have not delegated a market actor to allocate data cap on their behalf.

    The storage provider passes the FIL+ allocation, market address, and deal IDs when proving a sector. The miner actor checks that these values match the sector content, and then passes them on as trusted values to the verified registry and market actors.

    Frequent flows will be even simpler.

    • For a deal that is not verified, the verified registry actor is not involved, so neither the client, market, nor the miner actor need to call to allocate or claim data cap. These calls are marked as (~nv)
    • For a verified data cap without a deal, no market is involved, so the provider need not publish the deals nor call the market to inform it of the new committed content. These calls are marked as (~vo).

    Migration

    A state migration is required to realise this architecture in mainnet. Details to be fleshed out, but in brief:

    • Move the sector→deal mappings from miner sector info to market actor
    • Leave existing QA power, pledge, penalty values in place for existing sectors

    Discussion

    Limitations

    • Only a single “market” contract and identifier supported for each piece, notified when committed. An intermediate contract could proxy for multiple markets, but clumsily. Should we support multiple notifiers per piece?

    Extensions

    These APIs are designed with extension to more powerful and flexible programmable storage in mind. In general these will require new methods to be added.

    • Deal extension within sector lifespan
      • Extension of deal terms up to the end of a newly-extended sector commitment can be achieved after (or concurrent with) the miner calling market.SectorExpirationChanged. New methods on miner and/or market actor would be needed to support this.
    • Deal extension beyond sector lifespan
      • Extension of deal terms, possibly indefinitely, can be made possible by committing the same deal data into a new sector. Additional parameters need to communicate which sector the piece is moving from (or recording sector id in deal state).
    • Updatable sector content
      • Sector content is presently write-once, primarily due to restrictions imposed by the QA power mechanism. When we resolve those restrictions, miners can update the content in sectors after the initial seal or replica update. To support this safely, markets must be provided a notification whenever the content of a sector that is (was) supporting deals from that market changes. Something like the miner→market registrations proposal could achieve this, but requires some detailed design to find a good simplicity/efficiency trade-off.
    • Transfer between providers
      • New methods could support the transfer of sector content between storage providers, with the appropriate notifications to market actors. Markets would set their own policy as to whether such transfer is permitted under the deal terms.
    • Fault notifications.
      • Markets may wish to be notified when a sector becomes temporarily faulty. The miner→market registrations proposal may satisfy this need.
      • The FIL+ premium proposal also requires this.
    • Async sector content proofs, with new piece inclusion proof
      • At present, a miner must activate deals synchronously with proving the sector content. This is because the unsealed sector CID (”CommD”) is not stored on chain, but is required in order to demonstrate the inclusion of any specific data within the sector. Future work may provide an inclusion proof that can be evaluated after sealing in order to demonstrate to a market (or the verified registry actor) that a provider is storing some data some time after the sector is sealed.

    CryptoNet is a Protocol Labs initiative.

    type SectorPreCommitOnChainInfo struct {
	Info {
	  SealProof       abi.RegisteredSealProof
	  SectorNumber    abi.SectorNumber
	  SealedCID       cid.Cid // CommR
	  SealRandEpoch   abi.ChainEpoch
    // Sector pre-commit no longer references deals.
		DealIDs         []abi.DealID
	  Expiration      abi.ChainEpoch	
	  UnsealedCID     cid.Cid // CommD
  }
	PreCommitDeposit  abi.TokenAmount
	PreCommitEpoch    abi.ChainEpoch
}

type SectorOnChainInfo struct  {
	SectorNumber          abi.SectorNumber
	SealProof             abi.RegisteredSealProof
	SealedCID             cid.Cid // CommR
  // Sector info no longer references deals either.
	DealIDs               []abi.DealID

	Activation            abi.ChainEpoch  
	Expiration            abi.ChainEpoch  
	DealWeight            abi.DealWeight
	VerifiedDealWeight    abi.DealWeight
	InitialPledge         abi.TokenAmount 
	ExpectedDayReward     abi.TokenAmount 
	ExpectedStoragePledge abi.TokenAmount 
	ReplacedSectorAge     abi.ChainEpoch  // Can deprecate 140 days after nv15
	ReplacedDayReward     abi.TokenAmount // Can deprecate 140 days after nv15
	SectorKeyCID          *cid.Cid
}
    type SectorPreCommitInfo struct {
	SealProof       abi.RegisteredSealProof
	SectorNumber    abi.SectorNumber
	UnsealedCID	cid.Cid // CommD
	SealedCID       cid.Cid // CommR
	SealRandEpoch   abi.ChainEpoch
  // No deals specified during pre-commit.
	DealIDs         []abi.DealID
	Expiration      abi.ChainEpoch
}
type PreCommitSectorBatchParams struct {
	Sectors []SectorPreCommitInfo
}
func (Actor) PreCommitSectorBatch(params *PreCommitSectorBatchParams)

// Sector content is declared at PoRep
// (previously, this data was fetched from the built-in market actor).
type SectorManifest struct {
	SectorNumber uint64
	// Declaration of pieces that make up the sector data.
	// Implicit "zero" piece to fill remaining capacity need not be specified.
	Pieces       []PieceManifest
}
type PieceManifest struct {
	PieceCID                 cid.CID // CommP
	Size                     PaddedPieceSize
	// Each piece may specify a FIL+ verified data cap allocation ID.
  // If the piece is a sub-piece of a larger allocation, also specifies
  // the range covered and an inclusion proof in the allocation.
	VerifiedDataAllocationID uint64 // Optional
	VerifiedDataRangeStart   uint64
  VerifiedDataRangeEnd     uint64
	VerifiedDataInclusionProof []byte // ???
	// Each piece may specify an actor and piece ID to notify on commitment.
  // TODO: an array of these tuples for multiple downstream actors?
	NotifyAddress            address.Address // Optional
	NotifyPieceID            uint64 // E.g. deal ID, optional
}

type ProveCommitAggregateParams struct {
	SectorNumbers  bitfield.BitField
	AggregateProof []byte
	// Declaration of data for each non-empty sector.
	Manifests      []SectorManifest
	
}
func (Actor) ProveCommitSectorAggregate(params *ProveCommitAggregateParams)

type ReplicaUpdate struct {
	SectorID             abi.SectorNumber
	Deadline             uint64
	Partition            uint64
	NewSealedSectorCID   cid.Cid // CommR
  // Deals are replaced by sector manifest.
  Deals                []abi.DealID
	UpdateProofType      abi.RegisteredUpdateProof
	ReplicaProof         []byte
	Manifest             SectorManifest
}
type ProveReplicaUpdatesParams struct {
	Updates []ReplicaUpdate
}

func (Actor) ProveReplicaUpdate(params *ReplicaUpdateParams)
    type SectorDeals struct {
	SectorExpiration abi.ChainEpoch
	Deals            []abi.DealId
}
type State struct {
	// All existing state as today.

	// Maps sector IDs to deals, per storage provider.
	// This supports the actor keeping track of whether deals remain satisfied
	// as data moves between sectors in the future.
	SectorDeals cid.CID // HAMT[address.Address]AMT[abi.SectorNumber]SectorDeals

	// TODO: should we also write the deal's sector ID into the deal state?

	///// Extensions

	// A queue of sector IDs by their expiration epoch.
	// Records sectors for which the market must check if deals have been
	// re-commited to new sectors before the original one expires.
	// This supports the market accepting deals that have a term longer
	// than the lifespan of the sector they are originally sealed into.
	// No need to track where the deal fits inside a sector's lifespan:
	// the market will be explicitly notified if the sector terminates early.
	SectorExpirations cid.CID // AMT[abi.ChainEpoch]bitfield.Bitfield

}
    // Publishing storage deals is unchanged.
// This method is NOT STANDARDISED across markets, as they may have
// different deal proposal structures.
func (Actor) PublishStorageDeals(params *PublishStorageDealsParams)

type PieceInfo struct {
	// The piece data commitment and size, verified by the miner actor.
	CID  CID
	Size PaddedPieceSize
	// Market-specific identifier for this piece, provided by the provider.
	// E.g. the deal ID which the provider claims this piece satisfies.
	// TODO: should we use a more flexibile []byte key?
	PieceID uint64
}
type SectorContentAddedParams struct {
	Sector     abi.SectorNumber
	Expiration abi.ChainEpoch
	Pieces     []PieceInfo
}
// Called by the miner actor when non-zero data is proven in a sector,
// either at initial PoRep, or with a replica update (Snap).
// Checks that the piece CID and size match the nominated deal, and consider
// the deal active if so.
// This replaces the current ActivateDeals.
// Without extensions, rejects attempts to satisfy a deal if the deal term
// is beyond the sector expiration.
// With state.SectorExpirations state member, can support deals with arbitrary terms.
// Note: will probably implement a batch version, single use shows for simplicity.
// TODO: add a generic return value to be threaded back through the miner
// to the caller?
// STANDARD METHOD: other market actors must implement the same method
// signature if they expect to receive calls from the miner actor when
// data is committed.
func (Actor) SectorContentAdded(params *SectorContentAddedParams)

type SectorTerminatedParams struct {
	Epoch   abi.ChainEpoch
	Sectors []abi.SectorNumber
}
// Called by the miner actor when a sector is terminated ahead of its
// scheduled expiration.
// Pending a mechanism for fine-grained event registrations, this is called
// for all unscheduled sector terminations, whether or not the sector holds
// a deal for this market (because the miner doesn't know).
// The market actor looks up the sector->deal map and marks any
// deals found as terminated (but defers further processing until cron).
// This replaces OnMinerSectorsTerminate().
func (a Actor) SectorTerminated(rt Runtime, params *OnMinerSectorsTerminate)

///// Extensions

// Note: we can immediately support deal extension up to the *original* 
// expiration of the sector with no further state (but a new market method).

type SectorExpirationChangedParams {
	Sector        abi.SectorNumber
	OldExpiration abi.ChainEpoch
	NewExpiration abi.ChainEpoch
	// TODO: add parameters for deals to extend at the same time?
}
// Called by the miner when a sector's expiration epoch is extended.
// Can support extending a deal up to the sector's *new* expiration.
// Note: will probably implement a batch version, single use shows for simplicity.
// STANDARD METHOD.
func (Actor) SectorExpirationChanged(params *SectorExpirationChangedParams)

// With state.SectorExpirations, can support deal extension to arbitrary terms,
// beyond a sector's currently committed lifespan, by the provider
// re-committing the same data into a new sector.
// Deal extension is a simple case of more general deal re-negotiation.
// Such methods are NOT STANDARDISED, and depend on individual market features.
// Details omitted here.

// If we make sector content updateable, a new SectorContentChanged
// method implicitly replaces any pieces not explicitly mentioned.
// In order to track pieces between sectors, either 
// (a) caller must also specify the previous sector ID for a piece, or
// (b) store deal sector ID in deal state object.
// STANDARD METHOD.
func (Actor) SectorContentUpdated(params *SectorContentUpdatedParams)