Filecoin Aggregation Architecture

Creator
Nicola
Created
Nov 20, 2023 3:41 PM

Project

Problem

  • It is very complex to store data smaller than 32GiB
  • There is no standardized way of doing this
  • Several projects work via centralized servers, centralized credentials (username and pass) and centralized means of payment (credit cards)

Goal

Anyone should be able to experience storing data into Filecoin using a wallet and a non-proprietary protocol.

High level plan

Design a standard interface for deal-making that operates via a layer 2.

image

Map of Efforts

Design Space (opinionated)

Here we will collect our design goals and requirements

Protocol Design requirements

  • It is not possible to do this at the layer 1 (see https://youtu.be/kSE1gZVMXHk?t=1328)
  • Aggregators shouldn’t have ad-hoc APIs (composability)
  • Aggregators take deals from a market with other aggregators (where clients can specify the aggregators)
  • …

Design goals

  • …

Architecture

Parties

  • Client
  • Storage Provider
  • Aggregator
  • Hot Stoage Market L2
  • Buffer

  • Repair & others

High level Overview

Data preparation

  1. Client prepares data locally (or delegates it - out of scope)

Buffer

  1. Client stores their data at the Buffer

Advertising

  1. Client advertises their subpiece to the Aggregators Market
  2. Aggregator advertises their subpieces to the Storage Provider

Piece offloading

  1. Storage Provider/Aggregator picks a set of subpieces from the Aggregators Market
  2. Storage Provider tries to retrieve the data from the Buffers
  3. If Storage Provider succeeds, SP generates a commitment and sends the list and the commitment to the Aggregator
  4. Aggregator verifies the commitment from and sends a confirmation to the SP

Status check

  1. The client queries the Aggregator to learn about the status of their deal

Repair

  1. If Storage Provider loses deals, Repair gets the subpiece from available SPs and restart the process from 1 as a client

High level APIs

Client

  • Put:
    • Prepares the data
    • Sends it to the buffer
    • Advertises to the aggregators market
    • Waits until the deal is made and stores the info
  • Get:
    • Get the SP from the info
    • Starts request to get the file
  • List:
    • Get the SP from the info
    • Starts request to get the list of files

Aggregators Market

  • Piece Collection
    • (Client-facing) Put / submit
    • (Client-facing) Status / getAllDeals
    • (SP-facing) AcceptCIDs

Storage Provider (or aggregator)

  • Piece offload
    • (SP-facing) PullCIDs
    • Routine checkChain / complete
    • Filecoin Deal Proposer API

Buffer

  • (Client-facing) Put
  • (SP-facing) Get

Repair

  • (Client-facing) RequestService
  • (Client-facing) Status
  • Routine Check

Plan

  • Offchain phase: The first plan is to have Repairs, Buffers and Aggregators as off-chain entities
  • Onchain phase: The second plan is to turn Repairs, Buffers and Aggregators into smart contracts

CryptoNet is a Protocol Labs initiative.