Retrieval Check

Overview

The Retrieval check tests that previously stored pieces from data storage checks remain retrievable over time. It runs on a separate schedule from data storage checks.

This is distinct from the inline retrieval verification in the data storage check, which confirms an SP can serve data immediately after indexing. The Retrieval check answers a different question: does the SP continue to serve data correctly after the initial storage operation?

Definition of Successful Retrieval

A successful retrieval requires ALL of:

1. Randomly select a previously stored test piece of a dealbot-managed dataset.
2. Verify the root CID is discoverable via IPNI and the SP is listed as a provider
3. Perform /ipfs retrieval with the SP.
4. Fetch the root CID from the SP and confirm the response is HTTP 2xx.
5. Verify the root block hashes to its CID.
6. Walk the DAG and repeat steps 4–5 for each child block.

Failure occurs if ANY required check fails (IPNI verification, download, or content verification) or the retrieval exceeds its max allowed time.

Operational timeouts exist to prevent jobs from running indefinitely. If a retrieval exceeds the configured limit (RETRIEVAL_JOB_TIMEOUT_SECONDS), it is marked as failed.

Note on location: Retrieval latency varies by dealbot-to-SP distance. Measurements reflect dealbot's probe location, not absolute SP performance. This check tests retrievability, not latency.

What Happens Each Cycle

The scheduler triggers retrieval testing on a configurable interval.

flowchart TD
  SelectPiece["Select random test piece for SP under test"] --> IPNIVerify["IPNI verification"]
  IPNIVerify --> RecordResult["Record result (success/failure)"]
  SelectPiece --> DownloadPiece["Download via SP IPFS gateway<br/>/ipfs/{rootCid}"]
  DownloadPiece --> ValidateData["Validate downloaded data"]
  ValidateData --> RecordResult

Piece Selection

Dealbot randomly selects one piece per SP for each scheduled retrieval job. The per‑SP job frequency is controlled by RETRIEVALS_PER_SP_PER_HOUR. Selection follows these constraints:

• Only pieces from "data storage" check deals with overall status success (saved in the DB as DEAL_CREATED).
• Only pieces with IPNI metadata enabled and a root CID.
• Only pieces of size RANDOM_PIECE_SIZES (matched against metadata.ipfs_pin.originalSize).

Source: retrieval.service.ts (selectRandomDealsForRetrieval)

Retrieval Checks

For each selected piece, dealbot performs the following in parallel:

IPNI Verification

Retrieval checks only query the IPNI indexer to confirm the SP is listed as a provider for the root CID. We do not poll the SP for piece status in retrieval checks. The polling interval and timeout are controlled by IPNI_VERIFICATION_POLLING_MS and IPNI_VERIFICATION_TIMEOUT_MS.

/ipfs Retrieval

Dealbot retrieves the content by traversing the DAG rooted at the IPFS Root CID and fetching each block from the SP IPFS gateway.

• Root URL: {serviceURL}/ipfs/{rootCID}?format=raw
• Block URL: {serviceURL}/ipfs/{cid}?format=raw
• Request: HTTP/2 with Accept: application/vnd.ipld.raw
• Applicable when: Piece has IPNI metadata enabled with a root CID
• What this tests: The SP can serve the root CID and all linked blocks in its DAG via the IPFS gateway

Source: apps/backend/src/retrieval-addons/strategies/ipfs-block.strategy.ts

What Gets Asserted

For each retrieval attempt:

Retrieval Result Recording

Each retrieval step (post IPNI validation) creates a Retrieval entity in the database:

Source: retrieval.entity.ts

Source: retrieval-addons.service.ts, cdn.strategy.ts

Source: apps/backend/src/config/app.config.ts

Metrics Recorded

Metric definitions (including Prometheus metrics) live in Dealbot Events & Metrics.

retrievalStatus counts the /ipfs transport stage only (assertions 2 and 3). The IPNI assertion (1) is counted on discoverabilityStatus. There is no composite "retrieval check" counter; overall success comes from dataStorageStatus, which is success only when all sub-statuses succeed. See Deal Status Progression.

skipped.piece_missing

Emitted when a retrieval pre-flight probe to ${serviceUrl}/pdp/piece/:pieceCid/status returns 404. The deal is marked cleaned_up=true and removed from future retrieval candidate selection. This is not a failure of the SP's transport surface, but a signal that the piece no longer exists on the SP while the dataset is still live (for example, the SP scheduled a piece removal via PDP, or the piece dropped without an on-chain notification). The probe runs before IPNI verification and transport, so a 30s IPNI timeout is avoided on stale candidates. Search logs for retrieval_skipped_piece_missing to correlate.

Configuration

Key environment variables that control retrieval testing:

See also: docs/environment-variables.md for the full configuration reference.