@Luca @Peter Rabbitson @Kubuxu
WIP: Looking for a better name
A way of proving persistent space which Implicitly proving access to data encoded into sectors
One key feature Filecoin should enable is efficient retrievals. A key insight from the growing data-owner base of filecoin, is that this efficiency is of utmost importance. This holds both for the clients themselves and for SPs who are faced with difficult choices on how to optimize their hardware for the most common use-case.
In essence, given the way our sealing process is designed today, the easiest way to ensure such a feature is via asking providers to arrange reliable access to the clear data. One key question is how to assure clients that, once they hand their data to an SP, this reliable access to their data is continuously maintained.
We propose to do so via Proof of Access, which is implicitly provided every time a WindowPost and a WinningPost is posted on-chain.
🗺️ How to get there
There are two ways we can get Proof of Access
- No new trusted setup needed
- Work is scoped sctrictly to the VM (fil-actors) and the SP software (lotus)
⇒ Fastest way to ship Data Access Proof
- SP have a 2x proving overhead wrt today
- 2x gas cost for committing a sector onchain (PoRep gas cost)
- We can minimize proving overhead for SPs
- Research work needed l
- New trusted setup needed (or Testudo?)
- More engineering work to build and audit the new circuits
⇒ As a result, this approach would require much more time to ship
In this doc, we describe the scenario of re-using existing circuits.
🚀 Impact on Filecoin
Why Proof of Access in Filecoin?
Unique features of Proof of Access vs status quo
While today efficient retrieval capability can be claimed but not proven, having a proof of access at each WindowPost and WinningPost makes efficient retrieval capability provable and publicly verifiable.
- SP can provably claim they can support Efficient Retrieval on specific sectors
- Clients have periodical (24h) guarantee that Efficient Retrieval is actually possible on their sectors
- Failing a Proof of Access results into failing to prove the sector
This means that PoRep, WindowPoSt and WinningPost would be respectively invalidated if the corresponding Proof of Access fails. This aspect, which is novel and was not achievable with previous attempts (see for example
- PoRep with Proof of Access would be a separate sector-proof type with a different on-chain flow as current PoRep (similar to Synth-PoRep, an SP can decide on a per-sector basis which PoRep/PoST regime to use)
- No need for a new trusted setup if we re-use the current circuits
- SP are able to store only one incompressible string (the replica) and secure access to a copy of the data in the clear, which can be shared among different SPs or compressed
- The change we propose would also directly enable to SnapDeal sectors with Data that currently can not be snapped (mandatory on-chain accounting making PoA work, is reusable by SnapDeal)
- The proving overhead required to SPs can be also charged to CC sectors if we want to induce both types of sector to have the same proving cost
For a more detailed description of the above, please see
🔥 Current Risks
- If we want to re-use the current circuits in order to avoid a new trusted setup and engineering effort, we would basically ask SPs to submit two proofs per sector (one for the sector key and the other one for the replica).
- We can de-risk this aspect deciding to build new circuits, but this would have other disadvantages (see )
- If we use the current circuits, SP would have to commit to both the replica and the sector key onchain
- We can partially mitigate this cost if we build new ad-hoc circuits (research is needed in order to validate this intuition)
- The PoA ( TreeK + Data ) can potentially be made optimistic (this intuition needs to be validated)
- Given each sector proof will consist of two different proofs, the on-chain footprint of each sector proof will double its size compared with today.
📦 Fil-PoA Protocol
With Fil-PoA (Filecoin Proof of Access) we want to introduce a change in the way PoRep are produced and verified in order to make efficient retrieval of filecoin data possible and easy to implement.
We do so by
- Slightly augmenting the way PoRep is computed
- Entangle Proof of Access with consensus
- Give periodical guarantee of Proof of Access (via existing PoST schedule)
Today, at PoRep step, SPs prove that the replica was built correctly in a probabilistic way.
- For CC sectors, this means that the
SectorKeyis well formed
- For sector with deals, this means that
Datais well formed
Note that we have a unique commitment
CommR onchain for each sector in the network, which in case of CC sectors “degenerate” in a commitment to
WindowPost, SP need to prove that they are still storing all the replicas in a given partition, while at
WinningPost they got challenged on a randomly selected sector for which they need to prove they are actually storing the replicated format.
We propose to slightly modify the flow above for sectors with data in the following way:
In addition to the current
CommR commitment flow, for each sector with data, an SP should add on chain
- A proof that
SectorKeyis correctly built
Each sector in a proven partition should answer challenges against both
A challenged sector should answer challenges against both
Efficient retrieval provably enabled
SectorKey implicitly proves access to a copy of clear data, encoded into the sector, while allowing an SP to not double the amount of space needed to answer challenges. Indeed:
- An SP can share a copy of the clear data and persistently store an encoded version of it.
- When challenged, he derives
Dataand answers the challenges
- Even if an SP decides to store both
SectorKeyand does not access the data,
Datacan be easily derived as
A unique proof entangling Proof of Access to the data to Consensus
Given now PoRep, WindowPost and WinningPost all consist of two components, failing any of the two would invalidate the whole proving step. This means that not (implicitly) proving data access would result into the same countermeasures which today are in place for the current PoRep, WindowPost and WinningPost respectively
Enabling SnapDeals for Sector Natively with Deals
Today SnapDeals can not be applied to sector which are natively with Deals, since in order to do so, we miss a commitment to
SectorKey onchain required to check against when proving the SnapDeals step. This limitation would be totally removed by introducing a proof and a commitment to
SectorKey for all sectors (for existing sectors natively sealed with deals, this might require re-computing the
(Optional) Equalizing proving cost of CC Sectors and Sector with Deals
If we introduce Fil-PoA for sectors with deals only, one drawback is that sectors with deals will have a 2x proving overhead compared with CC sectors. We could re-balance this by requiring CC sectors to be snapped with a pre-determined set of non-neutral elements (non-
0x00 ) . On one hand, this would make CC sectors also more expensive to prove but, on the other hand, we would minimize the operational difference between CC and Data sectors, thus reducing the expense an SP would need to pass on to a client.