Logo

    Filecoin Aggregation Architecture

    Creator
    Nicola
    Created
    Nov 20, 2023 3:41 PM
    • Project
    • Map of Efforts
    • Design Space (opinionated)
    • Architecture
    • Parties
    • High level Overview
    • High level APIs
    • Client
    • Aggregators Market
    • Storage Provider (or aggregator)
    • Buffer
    • Repair
    • Plan

    Project

    Problem

    • It is very complex to store data smaller than 32GiB
    • There is no standardized way of doing this
    • Several projects work via centralized servers, centralized credentials (username and pass) and centralized means of payment (credit cards)

    Goal

    Anyone should be able to experience storing data into Filecoin using a wallet and a non-proprietary protocol.

    High level plan

    Design a standard interface for deal-making that operates via a layer 2.

    image

    Map of Efforts

    • Project Camel
    • DW3S Decentralization roadmap
    • FVM-enabled deal aggregation

    Design Space (opinionated)

    Here we will collect our design goals and requirements

    Protocol Design requirements

    • It is not possible to do this at the layer 1 (see https://youtu.be/kSE1gZVMXHk?t=1328)
    • Aggregators shouldn’t have ad-hoc APIs (composability)
    • Aggregators take deals from a market with other aggregators (where clients can specify the aggregators)
    • …

    Design goals

    • …

    Architecture

    Parties

    • Client
    • Storage Provider
    • Aggregator
    • Hot Stoage Market L2
    • Buffer
    • Repair & others

    High level Overview

    Data preparation

    1. Client prepares data locally (or delegates it - out of scope)

    Buffer

    1. Client stores their data at the Buffer

    Advertising

    1. Client advertises their subpiece to the Aggregators Market
    2. Aggregator advertises their subpieces to the Storage Provider

    Piece offloading

    1. Storage Provider/Aggregator picks a set of subpieces from the Aggregators Market
    2. Storage Provider tries to retrieve the data from the Buffers
    3. If Storage Provider succeeds, SP generates a commitment and sends the list and the commitment to the Aggregator
    4. Aggregator verifies the commitment from and sends a confirmation to the SP

    Status check

    1. The client queries the Aggregator to learn about the status of their deal

    Repair

    1. If Storage Provider loses deals, Repair gets the subpiece from available SPs and restart the process from 1 as a client

    High level APIs

    Client

    • Put:
      • Prepares the data
      • Sends it to the buffer
      • Advertises to the aggregators market
      • Waits until the deal is made and stores the info
    • Get:
      • Get the SP from the info
      • Starts request to get the file
    • List:
      • Get the SP from the info
      • Starts request to get the list of files

    Aggregators Market

    • Piece Collection
      • (Client-facing) Put / submit
      • (Client-facing) Status / getAllDeals
      • (SP-facing) AcceptCIDs

    Storage Provider (or aggregator)

    • Piece offload
      • (SP-facing) PullCIDs
      • Routine checkChain / complete
      • Filecoin Deal Proposer API

    Buffer

    • (Client-facing) Put
    • (SP-facing) Get

    Repair

    • (Client-facing) RequestService
    • (Client-facing) Status
    • Routine Check

    Plan

    • Offchain phase: The first plan is to have Repairs, Buffers and Aggregators as off-chain entities
    • Onchain phase: The second plan is to turn Repairs, Buffers and Aggregators into smart contracts

    CryptoNet is a Protocol Labs initiative.