Problem

At present, the miner actor records a sector→deal mapping and informs the market about changes in the state of sectors storing deals (namely sector termination). Even if we remove the sector→deal map the miner actor will still need to inform the market about sector termination.

Sector history digests can move the sector→deal mapping state off-chain and support lazy, batch processing of state changes.

Ideas

Off chain sector→deal mapping

Rather than tracking the sector->deal mapping on chain, a storage provider tracks it locally. A provider periodically submits proof that a deal’s data is contained in some sector, and that that sector has remained active since the prior proof.

A deal proof comprises:

an inclusion proof that the deal’s piece CID is contained in the CommD corresponding to a sector’s CommR, and
a history proof the sector has remained active since the last deal proof

Note these proofs do not involve SNARKs, but a SNARK computation may be applicable to proving the valid computation of many sector proofs all at once.

Sector history proofs

Each sector’s lifetime may be described as a sequence of events. Event types are:

Initial commitment = (sectorID, CommD, epoch)
Data update = (NewCommD, epoch)
Termination = (epoch)
[Faults and recoveries could also be included here]

In each sector state, store a rolling hash over the history of events. At initial commitment, the history digest is d := hash(e), where e is the initial commitment event.
At each lifecycle event replace the history digest with d' := hash(e', d), where e' is the new event.
Note that the event history is not stored on chain, only the digest. The miner operator stores the history events off-chain (they can be re-computed from chain history).
In order to prove that a sector had a specific history, a party provides the event list to a method which re-calculates the digest on-chain and compares it with the current one.

Note that if history digest matches current, no other lifecycle event can have happened, so this is an effective demonstration that the sector is still alive (as at end of last epoch).

Claiming deal fees

For each deal, a miner periodically submits a proof of the history of the sector containing a deal. Upon successful proof, the market actor releases incremental funds to the provider. The cost of this state manipulation would be borne by the miner, rather than cron. This might happen far, far less frequently than every 100 epochs; the miner can choose an appropriate period, submit the message during low-chain contention, etc.

If a deal has gone unproven for too long, a client may be able to challenge a miner to submit a proof, triggering deal termination (and loss of collateral) if the provider does not respond within some delay. Or a deal might specify a maximum period between proofs before automatic forfeit.

More generally, third parties might be incentivised to challenge deals by being able to claim some part of the provider collateral for a successful challenge. Such parties would track the deal→sector mapping off chain, and follow sector lifecycles, in order to spot when a sector has terminated while serving deals.

Moving data between sectors

The off-chain mapping supports moving piece data between sectors while continually satisfying a deal. To move a piece, a miner shows a history proof for the initial sector up to some epoch E, then an inclusion proof for the piece in a second sector at E, thereafter a history proof for the second sector.

This capability opens up deal extension, since the deal lifetime is no longer coupled to one sector's lifetime.

Moving data opens up the possibility of gaps in a deal history. Policy could determine whether gaps are tolerated, for how long, and what penalty is levied.

Proving distinct replicas

TODO: extra bookkeeping for a single miner to be able to prove multiple copies of some piece in distinct sectors.

Supersectors (Neutron)

The same technique works for supersectors. In this case a deal inclusion proof must link up to the supersector root (since individual sector CommR is no longer on chain). Supersector lifecycle events like merging would form part of sector history digests. Hopefully we can follow supersector merges without re-proving every deal for the new supersector CommR.

Aggregation of histories and proofs

Where one sector holds many deals, can we submit the sector history just once to satisfy them all
Can we aggregate histories of many sectors into a more efficient request?

Note: no mechanism to enforce zero data in CC sectors.