Creator

Alex North

Created

Feb 24, 2022 2:21 AM

Project

Storage Programmability

Stage

No more action (or old)

• Background
• Toward programmability
• Goals
• Design ideas
• Network-subsidised deals
• Term policies
• Quality-adjusted power
• Sector pledge
• Enforcement by miner actor
• Failure and repair
• Internal transfer
• Term extension
• Sector extension
• Integration with the built-in storage market
• Gas costs
• [Optional] Data cap and claims as tokens
• Delegation
• [Optional (yes)] Scaling up
• [Optional] Migration of existing deals
• [Optional (not yet)] Transferrable claims
• Implementation details
• Verified registry state
• Verified registry methods
• Market actor
• Miner Actor

A proposal to implement indefinite terms for FIL+ data cap allocations, independent of storage deals, realising a controlled version of ➕FIL+ Forever.

Terminology

Allowance: data cap provided to a verifier by the root, or to a client by a verifier.

Allocation: data cap allocated by a client to a specific piece of data.

Term: period of time for which a sector/deal/allocation is active or valid.

Claim: a provider’s assertion they are storing all or part of an allocation

Background

A FIL+ data cap allocation doesn’t have any intrinsic term. An allocation is currently bounded in time only because (a) allocation claims are made via deals, (b) we don’t support deals longer than a sector’s life, and (c) a sector has a maximum lifespan . When we either decouple FIL+ from the market actor (to enable alternative markets) or enable renewal/extensibility/transfer of deals, deal terms will no longer constrain the data cap term.

By leaning in to this idea we can both (1) simplify the concepts and implementation of quality-adjusted power, resolving most of the challenges that motivate the proposal; and (2) take a simple path for implementation of 💾Architecture for programmable storage markets.

Related: ➕FIL+ term limits is a (partial) design for enforcing arbitrary explicit term limits, which this design draws some parts from.

Toward programmability

These ideas build toward the 💾Architecture for programmable storage markets, which will eventually remove the market actor from intermediating FIL+ deals. In doing so, it breaks the linkage of the verified data to the term of the deal negotiated by the client. The power and rewards associated with FIL+ data cap instead can have a term independent of any deal for a client to pay for a particular storage period.

This proposal is a mechanism for removing FIL+ term limits in the context of the existing built-in actors, especially the storage market. This makes it a change that we could release to the network prior to the larger re-architecture for programmable markets. Alternatively, we could bundle this along with a larger architectural change (probably less total effort).

Goals

• Decouple limits on the duration for which a provider can enjoy subsidised power & rewards from the terms of a storage deal, supporting arbitrarily long terms while retaining some client discretion
• Retain strict tracking and enforcement of the requirement for continual storage of committed data in order to earn rewards and avoid penalties
• Add policy limits on verified data commitment terms, set by the FIL+ notaries
• Compatibility with and a step towards architecture to support programmable storage markets
• Scalability for a large expansion in FIL+ data

Design ideas

Network-subsidised deals

Storing a piece of verified data can be viewed as a network-subsidised deal (in addition to the implicit subsidy of block reward for all storage). One party to the deal is the network itself. While a verified client must bless a particular piece of data, the network will pay for its ongoing storage. This deal is independent of any other “normal” client←→provider deal, where a client can offer to pay an additional amount.

Thus, accounting for verified pieces is performed by the verified registry actor, rather than by the built-in (or any other) storage market actor. The verified registry actor maintains a record for each allocation and claim. These are somewhat analogous to a proposal and deal in the storage market, but simpler.

A verified client makes an allocation of data cap for a particular piece of data to (initially) a particular provider. An allocation specifies:

Due to the implementation of term enforcement (details below), clients should leave a fairly large buffer between the minimum and maximum term to make the allocation practical for a provider. The maximum term may be arbitrarily large, subject to policy set in the verified registry actor (details below).

When a provider claims an allocation, this obligation and benefit is also stored in the verified registry actor. A claim records:

When a claim is made, the allocation record is removed from state. Claim records persist until the provider cleans them up (to reduce their on-chain accounting costs).

A data cap allocation and claim are independent of any storage deal. So, for example:

• A client could make a data cap allocation for a minimum term of 180 days, and not initiate any other deal with a market.
• The nominated provider can prove the data and claim the allocation, immediately gaining power boost proportional to the size of the data. This power boost lasts as long as the provider continues to provably store the data. After 180 days, the provider may allow the sector to expire, or replace the verified data, without penalty.
• A client could allocate data cap and make a storage deal for, each for 180 days.
• The provider can claim the data cap allocation and consummate the deal, earning both the increased block rewards and the deal fees.
• If the storage deal expires un-renewed, the client stops paying, but the provider can choose to continue proving the data and retain the power boost. Or, after 180 days, allow the sector to expire, or replace the verified data, without penalty.
• The storage deal could be extended, in which case the client would keep paying too.
• A client could allocate data cap for a minimum 500-day period, and make a deal for only 100 days.
• The provider earns both the quality-adjusted power and deal fees for the first 100 days.
• After the deal expires, the provider must continue proving the data for the 500-day term to avoid a termination penalty.

Term policies

The verified registry actor can enforce a policy on the minimum and maximum terms. Policy values are set by the verified registry root actor. Term policies include:

• Minimum term: the smallest value a client may specify for a verified term minimum
• Maximum term: the largest value a client may specify for a verified term maximum

For example, we might set a policy minimum of 180 days (matching the sector minimum commitment) and maximum of five or ten years, or effectively indefinite. The policy constrains the values that a client may set when allocating data cap. Policy cannot change the minimum term for an already-committed piece of verified data.

We could support per-client policies. This would let the root key set different term policies for different data cap holders.

Quality-adjusted power

The current implementation of quality-adjusted power spreads out the power and reward boost from an arbitrary deal term over a sector’s lifespan. This behaviour introduces challenges, as the new QAP can be hard to recalculate when updating sector data. A deal term rarely matches a sector’s lifespan exactly, because the client and provider can’t coordinate that well. This proposal fixes that behaviour by aligning verified data terms with sector lifespans.

The term for a verified piece starts when the data is committed. There is no separate “deal start date”, only an allocation expiration epoch by which the data must be committed to be eligible. The sector immediately gains 10x power according to the fraction of deal space occupied by verified data.

The term for a verified piece ends when the sector expires or data is replaced. The term cannot end during a sector’s life except through explicit replacement of data, in which case the sector immediately loses the power boost from the verified data.

Aligning the boosted period with the term exactly in this way removes the spreading-out of QA power. Providers gain 10x power for each byte of verified data exactly while that data is proven. In general, providers will gain slightly more power, and hence reward, for verified pieces due to the improved alignment of term with sector lifespan. This will be more pronounced where the deal term was much shorter than the lifespan of the sector into which it was committed.

Sector pledge

A sector’s pledge requirement is proportional to its share of power, just as today.

After the minimum term for a verified piece expires, the pledge requirement does not change, just as the power and reward boost are retained. The provider’s commitment to storing the verified piece extends until the sector’s scheduled expiration. If the sector fails or is manually terminated early, the termination penalty includes the boosted power.

If the verified piece is sealed into a new sector in order to continue the commitment beyond the original sector’s life, the initial pledge for the new sector is calculated using the network conditions at the time. As pledge requirements are generally expected to decrease, this will decrease the required lockup for continuing the commitment beyond the minimum term.

Enforcement by miner actor

The miner actor is trusted code and can enforce the verified minimum term and the continual storage of the data.

An allocation’s minimum term is enforced by the miner actor by requiring that the sector into which the piece is sealed must have a remaining lifespan that is sufficient to fulfil the minimum term. While slightly limited in flexibility, this avoids any need for the miner actor to track the claims that are active in each sector in order to enforce penalties at otherwise-normal sector expiration events. When a sector terminates at its scheduled expiration epoch, no penalties are due.

Similarly, an allocation’s maximum term is enforced by requiring that the sector into which the piece is sealed must have a remaining lifespan no longer than that which would reach the maximum term. That is, a sector’s expiration epoch must fall between the minimum and maximum term’s of any verified claim it holds.

The longest practical minimum term for a verified data cap allocation today is 540 days, being the maximum initial sector lifespan. There is no shortest practical maximum term, as data could be snapped-in to a sector expiring arbitrarily soon (if it meets the minimum term).

Failure and repair

There is no change to termination penalty calculations. If a sector is terminated early (continual faults, or manual termination), the provider will pay a penalty related to the expected reward earned by the sector (up to some limit).

The built-in storage market actor does not support resumption of a deal after termination, but the verified registry can support re-commitment of the same data. If the claim’s maximum term has not been reached, the provider may commit the same piece into a new sector and regain the quality-adjusted power. This resumption ignores the original expiration epoch for the allocation, but the new sector’s committed lifespan is still constrained to fit within the claim’s term.

Internal transfer

Prior to a sector’s scheduled expiration, a provider may commit the same data into a new sector and retain the boosted power and rewards. The new sector must have an expiration epoch no sooner than the sector the data is coming from, and no later than the term limit for the verified piece. Upon a transfer like this, the new sector’s power is increased and the old sector’s power decreased by the amount corresponding to the size of the verified data, thus keeping the miner’s total power constant.

The old sector’s pledge requirement remains at its original value, even though the sector’s power is reduced. This follows existing practise of maintaining sector pledge at a high water mark, and discourages moving data around between sectors solely as a mechanism to reduce total pledge. The new sector’s pledge is calculated according to the network requirements at the time.

Verified pieces may be transferred from a sector that is faulty, as a means of preserving data stored in a broken sector. After verified data is transferred out from it, the fault and termination penalties for the old sector are reduced accordingly. If the original sector is terminated, the verified data cannot be sealed into a new sector without a new data cap allocation.

Term extension

The client that initially made an allocation can increase the maximum term for that allocation or claim, up to the verified registry’s policy maximum. Increasing the term maximum doesn’t require agreement from the provider.

A verified client can extend the term for a data cap claim beyond the initial maximum by spending new data cap on it. The term maximum can then be extended by up to the policy maximum again beyond its initial maximum. [Optional] The client extending the allocation need not be the one that originally made it.

In this way, the notaries can support the indefinite extension of FIL+ power and rewards without that necessarily being the default behaviour.

Sector extension

A provider cannot extend the scheduled expiration for a sector past the maximum term of a piece of verified data that it holds. The provider must give up the verified power boost when processing such a sector extension. This does not reduce the sector’s pledge requirement, but does reduce penalties to be paid on any future fault to reflect the new, lower power.

In order to prevent a provider using a trivial extension as a means of escaping a commitment to store the piece for the sector’s full originally-scheduled lifetime, this facility to drop a verified claim can only be exercised in the final TBD epochs of the sector’s scheduled lifespan, and only for non-faulty sectors.

Integration with the built-in storage market

The workflows described above are independent of any market actor, and take place through interactions by the client and provider with the verified registry actor directly. Transitioning fully to this model thus requires reworking of deal client and provider workflows. However, we can proxy these mechanisms through the built-in storage market actor, thus leaving client and provider workflows untouched. The market actor will make some default selections for the additional capabilities.

At present, the built-in storage market actor directly manages data cap accounting (even though the client’s allowances are stored in the verified registry actor’s state). When a deal proposal is published with VerifiedDeal=true, data cap is deducted from the client’s allowance and attached to a specific piece of data and provider (like an allocation), but is revoked if the provider does not commit the data before some deadline. When a deal is activated, the allocation becomes irrevocable and the provider gains benefit (like a claim).

We can proxy explicit allocations and claims through existing workflows:

• When a verified deal proposal is published, replace the current call to VerifiedRegistry.UseBytes with a new call to allocate data cap for the piece in question. Set the allocation’s minimum term to the greater of either the deal term or the registry’s policy minimum term. Set the allocation’s maximum term to 180 days longer than its minimum. Store the allocation’s ID alongside the market actor’s deal proposals. Set the allocation’s expiration to the deal start epoch.
• When a verified deal is activated by the provider, add a new call to the verified registry actor to claim the allocation.

The market actor is also responsible for computing the verified deal weight, an input to sector power. Deal weight is calculated as deal size times duration. With data cap claim terms exactly aligned with sector lifetime, the “space-time” aspect is no long relevant, only the total deal space as a fraction of sector capacity.

• Prior to this proposal to change pre-commit deposit and defer deal weight calculations from pre-commit to post-commit, deal weight is calculated at pre-commit (via market.VerifyDealsForActivation), but cannot be done correctly for this change since the activation epoch is not known. The method could be changed to return the raw space occupied by verified deals, and require migration of sector pre-commit schema to store that.
• After the above proposal, the deal weight can be calculated correctly during market.ActivateDeals, using the sector’s now-known complete lifespan. This would be much simpler. Also, “weight” becomes a confusing concept as compared with “size” at that point, so ActivateDeals might return the verified deal size directly.

The miner actor can then compute a sector’s power as SectorSize + (9 * VerifiedSize).

Gas costs

This proxy pattern will initially raise the gas cost of deals, but this is a transitional state.

• For verified deals where the client is offering no payment above the verified rewards, no deal with the market is necessary at all. A client need only make an allocation with the verified registry. This allocation is much cheaper to make and maintain than a deal.
• For deals where additional payment is necessary, these mechanisms open up the potential to develop alternative storage markets, including ones much more efficient than the built-in one. We can expect deal costs to decrease over time.

[Optional] Data cap and claims as tokens

FIL+ data cap may be viewed as a fungible but restricted-transfer token. Proxying data cap allocations through the built-in market actor is a special case of a more general delegated authority to transfer tokens, common in, e.g. the ERC-20 specification. The built-in storage market actor is approved to lock data cap allowance tokens on allocations, which are then burnt when the allocation is claimed. This smooths the client experience when making deals, removing the need for a two-step process to both allocate data cap and create a deal.

We can and should extend this approval to third party market contracts to be developed, so that they may enjoy the same benefits. A client can approve any address to allocate its data cap tokens on its behalf. Such a delegation mechanism also opens up potential for more automated allocation workflows in the future.

Similarly, a provider’s claim on the benefits and obligations of specific data cap allocation can be viewed as a non-fungible, restricted-transfer token. We can and should implement standard NFT interfaces as they emerge on the FVM (and this use case can inform those interfaces). Initially, verified piece claims are not transferrable between providers but, as we implement transfer in the future, the token transfer concept provides a good mechanism for representing this.

As for delegated data cap allowance, we might also implement delegated authority to transfer verified piece claims. This would permit the development of marketplaces and other piece liquidity mechanisms. Transfer restrictions would still require the transferee to prove commitment of the relevant data and, usually, the client to consent to the transfer.

Delegation

Delegated authority mechanisms like this support optionally coupling FIL+ behaviour with market behaviour.

As an example, with FIL+ decoupled from markets it would be difficult to implement a negatively-priced deal (where the provider pays the client), because a provider could claim a verified piece allocation directly with the verified registry actor, and ignore the costly deal. Delegated authority allows a market actor to require commitment to a deal as a condition of making the data cap allocation, because the market itself will make the allocation only after the deal is published (at which point the market can require a deposit or pre-payment of funds).

[Optional (yes)] Scaling up

We could support data cap allocations identifying pieces that are larger than a single sector. This would allow a scalable representation within the verified registry actor.

Rather than claim an entire allocation at once, a provider claims a range of the allocation for each sector by providing a sub-piece CID and an inclusion proof into the root piece CID. The sectors with sub-pieces gain power individually.

All ranges from a single allocation must be claimed by the same provider. The client could split a dataset into shards if they wish to spread it among multiple providers.

If a sector hosting one range of an allocation is lost, the provider will pay a termination penalty but can seal the same range into a new sector to regain the same power. This maintains the incentive to host a complete replica of the allocation, which is probably what the client values.

An allocation is successfully claimed if at least one range of the allocation is committed before the allocation expires. Other ranges may be committed after the expiration epoch. This provides symmetry with resumption after faults.

This technique could mesh nicely with the supersector technique from project Neutron. A miner could then efficiently claim a contiguous range of the data cap that is many sectors large, with a higher level inclusion proof.

[Optional] Migration of existing deals

We could migrate the verified status of all existing deals into the verified registry actor. This would prevent the verified piece benefits from expiring with the corresponding deal, and permit their term to be extended up until the maximum lifetime of the sector into which those pieces are sealed (5y from commitment). We would need to select a default policy for the data cap claim term, such as:

• Minimum term: the original deal duration (which must fit inside the sector’s life)
• Maximum term: deal duration + 180 days

After migration, verified clients could raise the maximum term of each claim up to the verified registry’s policy maximum. This would require explicit messages from the clients, but no action from providers beyond extending their sectors’ expirations.

Other components and effects of such a migration include:

• Must recalculate VerifiedDealWeight for every sector with verified deals.
• Must recalculate the power for every sector with verified deals, and update the power table.
• This will be a very small shock to network power, proportional to the mismatch between verified deal terms and sector lifespans.
• Changes to deal weight and power should change pledge requirement, but is that reasonable?
• We could deprecate deal weight (space-time) in the future as only the deal size is needed.

[Optional (not yet)] Transferrable claims

We could add support for the transfer of verified piece claims to other providers. In order for the original client to retain discretion over the providers suitable for storing their data, this would involve a mechanism for the client to assent to a particular transfer: a transaction with the verified registry actor, or a signed token authorizing the transfer.

In the case of allocations that did not specify a particular provider, we could allow transfer without involving the client (the client essentially declared up front that any provider is acceptable).

Similar to transferring data between sectors at one provider, the destination provider would commit the data into a sector, and then claim a transfer from the source provider. Similar restrictions would exist on the committed term of the sector in the destination provider. The obligation and power would transfer to the new provider immediately.

The pledge requirement for the verified data in the new sector should match that required for an internal transfer.

Implementation details

The verified registry actor tracks data cap allocations in state explicitly. Verified clients allocate data cap to a specific piece of data. A verified data allocation may be claimed by a provider that has committed matching data without regard to any storage market arrangements.

A client may make multiple allocations for the same piece (for distinct replicas), but each consume additional data cap.

The sketch below includes the optional large-piece representation.

TODO: Replace verified registry state/methods with Data cap token design

Verified registry state

type Allocation struct {
	Client Address // Could drop this if it's in the map key
  Provider Address // Optional?
  Piece CID
	Size uint64
  TermMinimum uint64
	TermMaximum uint64
  Expiration Epoch
  AllOrNothing bool // Optional? Require atomic commit of entire piece
}

type Claim struct {
	Provider Address // Could drop this if it's in map key
	Client Address
	Piece CID
	Size uint64
  // This representation is for super-sector allocations.
	// It's unfortunately not compressible while retaining the association
	// between specific sectors and their range of the piece.
  // Non-overlapping ranges. Store sorted by RangeStart? Use an AMT?
	Sectors [] {
		Sector SectorID,
		RangeStart uint64, 
		RangeEnd uint64, 
	}
	TermStart ChainEpoch // Epoch when piece was committed
  TermMinimum ChainEpoch
	TermMaximum ChainEpoch
}

type VerifiedRegistryState struct {
	// Verifiers, VerifiedClients as today

	// Delegated quotas by verified client and delegate.
	Approvals HAMT[Address]AMT[Address]uint64

  // Allocations by client, then by ID.
  // Nesting by client promotes more efficient lookups at scale,
  // and supports per-client iteration for revocation.
  // Removed when claimed.
	Allocations HAMT[Address]AMT[AllocID]Allocation

	// Claims by provider ID, then Claim ID.
  // Claim ID is inherited from allocation ID.
	Claims HAMT[Address]AMT[ClaimID]Claim

	NextAllocationId uint64
}

Verified registry methods

TODO Add methods to approve a delegate market actor to allocate data cap on behalf of a client.

type AllocateDataCapParams struct {
	Allocations []Allocation
}

// Called by a verified client to allocate some of its data cap allowance
// to specific pieces and providers.
func (Actor) AllocateDataCap(params *AllocateDataCapParams)

type GetAllocationParams struct {
	Client     address.Address
	Allocation AllocationID
}
// Fetches an un-claimed, un-expired allocation.
// NOTE: probably implement a batch version
func (Actor) GetAllocation(params *GetAllocationParams) *DataCapAllocation


type RevokeExpiredAllocationsParams struct {
	AllocationIDs []AllocationID // empty for "all", or specify a set
}

// Cleans up all expired allocations for a client.
// The data cap for cleaned-up allocations is returned to the client.
// Note that un-expired allocations cannot be revoked.
func (Actor) RevokeAllocations(params *RevokeExpiredAllocations)

type SectorAllocationClaim struct {
	AllocationID AllocationID
	PieceCID     CID
	PieceSize    uint64
	SectorID     abi.SectorNumber
	RangeStart   uint64, 
	RangeEnd     uint64, 
}
type ClaimAllocationsParams struct {
	Sectors []SectorAllocationClaim
}
type SectorClaimResult {
	// The IDs of allocation claims that succeeded.
	SuccessfulClaims []AllocationID
}

// Called by storage miner actor to claim allocations for data
// provably committed to storage.
// For each allocation claim, the registry checks that the provided piece CID
// and size match that of the allocation.
func (Actor) ClaimAllocations(params *ClaimAllocationsParams) *ClaimAllocationsResult

// Called by storage miner actor to remove claims that have been completed.
func (Actor) RemoveExpiredClaims(params *RemoveExpiredClaimsParams)

Market actor

No necessary public API change for basic functionality.

Exploiting the range information for larger-than-sector allocations requires a storage deal to specify the FIL+ allocation range covered, and an inclusion proof. This is a change to market API, or possibly to miner API.

• Or we could skip that for now (while leaving the verified registry range state there for future use, other markets etc).

Change behaviour of PublishStorageDeals and ActivateDeals to allocate and claim data cap if deal verified is set true.

Changes to the return value of ActivateDeals to return the right information for the miner actor to calculate power.

Miner Actor

No public API change.

Changes to interactions with the market actor to pass any additional data for claiming data cap allocations, and to calculate sector power according to data cap allocations that exactly match sector lifespan.

Change extend sector expiration to check FIL+ allocations and fail if taking one past its maximum.

More disruptive changes will be introduced with https://github.com/filecoin-project/FIPs/discussions/298

Workflows:

• ✓ revoke expired allocations
• Needs client → allocations in registry
• ✓ commit new FIL+ data
• Remove the allocation, add claim which identifies miner and sector.
• Sector ID required in claim to prevent cheating about which sector has which claim when extending, transferring, repairing etc
• ✓ sector expires normally
• nothing to do
• the claim remains in registry, recording the sector id that did store the piece. The piece cannot be claimed by a new sector.
• ✓ Extend sector expiration
• If sector has QA power, miner calls registry to identify the sector and check the max term. Worker must specify the claim IDs (sector→claim is an off-chain index), which registry checks
• ✓ move data to different sector
• Miner sends piece CID, old and new sector ID to registry. Registry checks and swaps, then sends back the expiration bounds for the new sector.
• Allowed when source sector is faulty, to preserve data.
• ✓ extend term max
• Just update allocation in registry, wait for miner to poll it when extending sector or transferring piece.
• ✓ sector terminates (and pays penalty), but re-sealed to regain incentive
• Termination event is not propagated to registry, the miner handles it
• When resealing, worker must identify the sector that used to host the piece (off-chain index), miner actor must verify it’s terminated, then send old and new sector to registry to check and swap
• ✓ support for future transfer of alloc to a new miner (but might need client approval)
• client address stored in claim (zero it if the original allocation specified no provider?)
• otherwise similar to internal transfer, but will be a more complicated multi-miner call in order to be atomic.