Luca 🌐 Context We currently have a retrievability assurance protocol (Retriev.org, details here) which works as follows:
- There is a consortium of providers that sign up to participate in our system;
- There is a fixed set of referees for which we can say that half of them are honest (eg, 5 referees and we trust 3 of them);
- Client and Provider agree on a retrievability deal (aka the assurance policy for the retrieval service); providers lock down a collateral for each deal and client locks down a payment for the service;
- When a client does not get the requested file, it appeals to the referees. One of them tries to retrieve the file asking it again to the provider, if this works the file is forwarded to the client (the provider gets paid); If not, we try again. After some failed attempts, the provider is penalized: the collateral is taken away (and no payment is provided).
Referee Network and its Invocation
Create appeal
While the deal is active, the client can appeal to the referees (n referees in total) at any instant on time for requesting the assured storage. The appeal is considered valid if all the following is true:
- The deal is active
- For this deal, the number of appeals already created is less the the maximum allowed by the protocol (eg, 5)
- There is no active appeal already
- It includes the payment of an appeal fee. This is the amount of token that the client pays to the referees for their service.
Start and Process appeal
If an appeal is created, any referee that is free can start the appeal process.
The protocol to process an appeal is made by k (eg, 12) rounds each. In each rounds the referees try to retrieve the file in this way:
- Step 1: a retrieval leader is chosen at random.
- Step 2: the leader asks the file to the provider.
- Step 2.a: If the leader does not get the correct file in time, he sends a signed “failure msg” on chain; the other referees see the message and do nothing for this round; in the next round all starts from Step 1;
- Step 2.b: If the leader gets the file it forwards it to the client and to the other referees. Each referee does the following:
- If the correct file is received, forward it to the client;
- If the correct file is not received before the end of the round, sign a “failure vote” and broadcast it to the other referees;
- Count the number of failure votes received, if they are ≥n/2, then send a “failure msg” on chain; at the next round, start from Step 1. Otherwise, do nothing for the remaining rounds. Note that only the first “failure msg” goes through.
≥n/2
After the k rounds are over, the contract checks the total number of “failure messages”.
- If it is greater or equal to a pre-determined slashing threshold, then the provider is “slashed” (ie, the collateral is deposited to the contract owner’s account), the payment is returned to the client and the deal is deactivated.
- Otherwise nothing happen (in particular, the deal stays active).
⚠️ Warning
Given the current design, the protocol is not enforcing any retrievability “budget” nor frequency.
🎯 Goal
We want to introduce here the possibility of introducing a Retrieval Budget within the retrievability deal duration, which would be beneficial both for Client ad Provider.
- Clients will negotiate in advance how many retrievals are granted by the protocol, ensuring this budget covers his needs through time
- Providers will have a precise idea of the bandwidth requirement they will have to allocate bandwidth to serve the file in a given window of time
We stress that we envision this feature to be opt in.
📌 Protocol Overview
Step 1: Retrieval Policy Proposal
When proposing a retrievability deal, a client can specify additional asks with respect to retrievals (note that all these asks are optional)
- What is the retrieval budget he wants to have (that is, how many retrievals he want to be granted during the deal duration). This is in essence a bandwidth ask for the provider. We refer to a retrieval budget unit as a “pass”
- The frequency of passes within the deal duration
- The price which he is willing to pay for each pass (aka retrieval)
- A fault tolerance on the asks above (e.g. 10%)
- The pass issuance policy (see below)
At this point, the Client’s funds are locked into the smart contract. Deal acceptance now works the same way as in the standard retrievability deal.
Pass Issuance Policy
We envision Pass Issuance the following way
- The Client generates the passes and commits to them using a MT
- Passes are given off-chain to the Provider
- Every time there is a successful retrieval, Client sends a signature of the correspondent pass to the Provider
- Clients’ signatures are used by the Provider to redeem the payment later
Notes:
- [Pass Sharing]. The way passes are shared between Client and Provider should be decided by Client and Provider themselves. Sharing tickets upfront seems to be the easiest and most efficient option in terms of communication
Both the option are viable.
Nevertheless, an important feature of the protocol is represented by the fact that it is the Client the one who sends the signed ticket to the provider. In this way:
- If the Provider misbehaves, then the Client can The important thing in our protocol is that the client sends signed passes to the Providers
- If the Client misbehaves, the Provides can immediately stop storing and serving the file, limiting the impact of the misbehave.
Step 2: Retrievals
According to the retrievability policy, at some point the Client wants to use a pass to download some data for the Provider.
- If the retrieval happens smoothly, the Client gets the data and the Provider gets the signed pass which ca be redeemed later.
- If the Provider is not serving the data or she is not compliant with the Retrievability Policy, the Client forward a “bad pass” notification to the Referee Network and starts an Appeal
- If the Client is not giving a pass to the Provider for successfully serving the data, the provider can stop serving the data.
Failures Rates
Failure rates are part of the retrievability policy and can be used by the Referees as a reason for (partially) slashing. An high level idea on how to deal with failures at the Referees level is then following
- If according to the Referee Network the Provider’s failure rate is high, then the Provider looses its staked money
- If according to the Referee Network the Provider’s failure rate is low (below agreed rate, so bad clients) - then auditors will grant tokens to SP at inverse of that rate without using additional bandwidth. The idea here is that not serving the file to Clients but serving files to the committee is not profitable for the Provider.
In general economics we should have is:
- In case of misbehaving Clients: Providers end up at a neutral place (get money, ends up serving a multiple of overall total content)
- In case of misbehaving Providers: Clients lose money, but the Provider looses a multiple x stake
Retriev and Retrievability Passes interop
We are not envisioning Retriev to change in order to incorporate Retrievability Passes protocol. We suggest the interoperability to work as follows:
- Retriev.org deal gets packed in a new “ticket retriev deal”
- There is a "ticket” smart contract that wraps retriev deal (so it is this smart contract calling the appeal to the Retriev committee)
- A Retrievability deal has a retrieval budget to be set up by the Client
- There is a protocol which runs on top of Retriev for ticket distribution
- Some Options/ideas
- signatures
- onchain requests
- mac?
======= Notes
- WS: how exactly auditors work, referring to hackmd.io/@irenegia/retriev
- WS: flesh out pass negotiation protocol
Questions/Open points
- [Passes not being signatures]: we should think if we want Clients’ pass confirmation to be something different from signatures.
- We could enable a larger class of users for which we do not have a public key
- Can we find a way to authenticate passes ahead of time?
- [Handling APIs] We need to decide how to handle API keys?
- check what privacy pass design does
- medusa?