Sampled Retrieval Check

Overview

The Sampled Retrieval check (sometimes referred to internally as retrieval++) tests publicly discoverable pieces on a storage provider (pieces that were not uploaded by dealbot). The intent is to measure SP retrievability against real-world tenant data, not just dealbot's own corpus.

This is distinct from the Retrieval check, which exercises pieces dealbot itself uploaded as part of a Data Storage check. The Sampled Retrieval check answers a different question: does the SP serve arbitrary pieces from its broader public corpus, with the same correctness and performance properties as dealbot's controlled pieces?

Definition of Successful Retrieval

A successful sampled retrieval requires:

1. Piece fetch — GET {spBaseUrl}/piece/{pieceCid} returns HTTP 2xx and the response bytes hash to the declared CommP (piece CID).

If the piece advertises IPFS indexing (withIPFSIndexing = true and a non-null ipfsRootCid), three additional dimensions are validated independently. Importantly, they do not gate the overall piece_fetch_status, and each is recorded as its own outcome column / metric:

1. CAR parseable: the fetched bytes parse as a CAR file.
2. IPNI: the SP is advertised as a provider for the root CID and a sample of child CIDs via filecoinpin.contact.
3. Block fetch: a sample of CIDs from the parsed CAR is re-fetched via {spBaseUrl}/ipfs/{cid}?format=raw and each response is hash-verified against its declared CID.

A piece without IPFS indexing is exercised only at step (1).

Operational timeouts exist to prevent jobs from running indefinitely. If the job exceeds SAMPLED_RETRIEVAL_JOB_TIMEOUT_SECONDS, it is aborted; a row is still emitted so that partial metrics (TTFB, bytes, response code) are not lost.

Piece Selection

Unlike the Retrieval check, dealbot does not retrieve from its own deals. Pieces are sampled from the on-chain subgraph of all FWSS-served pieces for the SP under test ¹.

Selection strategy (per scheduled job, per SP):

1. Pick a size bucket by weighted random:
2. small (1–10 MiB) — 20%
3. medium (10–50 MiB) — 50%
4. large (50–100 MiB) — 30%

Note: the bucket sizes were chosen such that the whole file will still fit into memory. In the future we may implement a streaming verification and parsing. 2. Pick a pool: - indexed (IPFS-indexed pieces) — 80% - any (all FWSS pieces) — 20%

Note: this split exists so that SPs cannot optimize only their CAR corpus and still appear healthy on this check. 3. Generate a uniform-random sampleKey and query the subgraph for the smallest Root.sampleKey ≥ $sampleKey matching the SP, payer, size range, and pool filters. If no such row exists (the random key fell above every matching sampleKey), samplePiece retries in the reverse direction (largest sampleKey < $sampleKey) so the highest keys are not a dead zone. 4. Drop the candidate if pdpPaymentEndEpoch has passed. 5. Fall back through: (same bucket, opposite pool) → (any bucket, indexed) → (any bucket, any).

Source: sampled-piece-selector.service.ts

What Happens Each Cycle

flowchart TD
  Select["Sample sampled piece for SP from subgraph"] --> Fetch["GET /piece/{pieceCid}"]
  Fetch --> CommP["Hash bytes → verify CommP"]
  CommP --> HasIpfs{"piece.withIPFSIndexing<br/>and ipfsRootCid?"}
  HasIpfs -- "no" --> Record["Persist ClickHouse row + emit Prometheus metrics"]
  HasIpfs -- "yes" --> ParseCar["Parse bytes as CAR"]
  ParseCar --> SampleBlocks["Pick N random CIDs<br/>(SAMPLED_RETRIEVAL_BLOCK_SAMPLE_COUNT)"]
  SampleBlocks --> Ipni["IPNI: verify SP advertises root + sampled CIDs"]
  SampleBlocks --> BlockFetch["GET /ipfs/{cid}?format=raw for each sampled CID"]
  BlockFetch --> HashCheck["Hash-verify each response against its CID"]
  Ipni --> Record
  HashCheck --> Record

Piece Fetch

• URL: {spBaseUrl}/piece/{pieceCid} (HTTP/2)
• Buffered in memory — piece sizes are capped at 100 MiB by selection (the upper bound of the large bucket).
• Download ceiling enforced — the download is aborted once received bytes exceed SAMPLED_MAX_PIECE_DOWNLOAD_BYTES; the partial buffer is discarded and the result is recorded as failure.too_large. The advertised size is the subgraph's claim, not an enforced limit, so this is the actual on-the-wire ceiling.
• Validates CommP — the CommP of the response bytes must match pieceCid.

Source: piece-retrieval.service.ts

CAR / IPNI / block-fetch validation (only when piece advertises IPFS indexing)

When the selected piece has withIPFSIndexing = true and a non-null ipfsRootCid, three dimensions are exercised by PieceValidationService. Each dimension has an independent outcome; a failure or skip in one never bleeds into another's status.

1. CAR parse: @ipld/car parses the response bytes; a random sample of SAMPLED_RETRIEVAL_BLOCK_SAMPLE_COUNT CIDs is selected for the next two steps.
2. IPNI check: IpniVerificationService.verify(rootCid, sampledCids, sp) polls filecoinpin.contact until each CID resolves to the SP under test, the timeout fires, or IPNI_VERIFICATION_TIMEOUT_MS is reached.
3. Block fetch check: for each sampled CID, fetch {spBaseUrl}/ipfs/{cid}?format=raw and hash-verify the response against the CID. Non-2xx, hash mismatch, unsupported codec, a response exceeding SAMPLED_MAX_BLOCK_DOWNLOAD_BYTES, or transport errors all count as a single failed block.

CAR parse failure (failure.not_parseable) is attributed to the client (bad upload), not the SP. When the CAR is unparseable, IPNI and block fetch are skipped because there are no sampleable CIDs to verify or fetch.

Source: piece-validation.service.ts

What Gets Asserted

Sub-status meanings

Unlike the Data Storage check, sampled retrieval does not have a rolled-up status (e.g., sampledRetrievalStatus). Piece retrieval, CAR parsing, IPNI verification, block-fetch outcomes are recorded independently. Each status metric below is emitted exactly once per check, except when sampledPieceRetrievalStatus=skipped because selection itself failed and no HTTP/CAR/IPNI/block-fetch work ran.

Sources: - sampled-retrieval.service.ts — orchestrates the dimensions and emits the four status metrics - piece-retrieval.service.ts — classifies piece-fetch outcomes - piece-validation.service.ts — produces CAR / IPNI / block-fetch outcomes independently - types.ts (CarParseStatus, IpniCheckStatus) — enum source of truth

Result Recording

Each sampled retrieval attempt writes one row to the sampled_retrieval_checks ClickHouse table — including a skipped row (piece_fetch_status='skipped') when piece selection found no candidate for the SP after all fallbacks, so the empty-pool case is recorded as check data rather than counted as a job failure. The row is emitted even on abort or unexpected error so that the partial evidence (TTFB, bytes, response code) is preserved. On a skipped row there is no piece to test, so the piece-identity columns (piece_cid, data_set_id, piece_id, raw_size) carry sentinel values ('' / 0).

The DDL and column-level comments in clickhouse.schema.ts are authoritative. The summary below is for orientation.

Source: sampled-retrieval.service.ts

Metrics Recorded

Sampled-retrieval Prometheus metric definitions live in Dealbot Events & Metrics. All sampled-retrieval metrics carry checkType=sampledRetrieval.

Configuration

Key environment variables that control sampled retrieval testing:

The in-memory download ceilings (SAMPLED_MAX_PIECE_DOWNLOAD_BYTES, SAMPLED_MAX_BLOCK_DOWNLOAD_BYTES) are compile-time constants in sampled-piece-selector.service.ts, not environment variables. The piece ceiling is derived from SIZE_BUCKETS.large.max so it stays in sync with the selection buckets.

See also: docs/environment-variables.md for the full configuration reference.