Talks

PL Demo Day Dec 2022

NAT Hole Punching Campaign Update

Dennis Trautwein

A mid-campaign update from PL Demo Day 2022 on ProbeLab's distributed NAT hole punching measurement, in which volunteers run a "punchr" client that attempts libp2p hole punches against random peers across the public IPFS network. By mid-December the campaign had around 80–90 active clients per day across all continents except Africa, contributing roughly 150,000 hole-punch results daily and over three million in total. Dennis shows that the remote peers being punched span a geographically diverse set of countries, which compensates for client-side location skew, and reports a preliminary success rate fluctuating between 60–70% — slightly below earlier controlled-network measurements, as expected for a heterogeneous deployment. The talk closes with a call for additional participants and points to both the menu-bar macOS client and command-line tooling for joining the campaign.

Watch

PL Demo Day Dec 2022

Unplugging the Hydra-Booster DB

Dennis Trautwein

A PL Demo Day 2022 update on the decision to "unplug" the shared database behind Protocol Labs' Hydra Booster fleet, which at the time consisted of around 135 Hydras with roughly 2,000 heads making up about 12% of the IPFS DHT. Dennis explains that the heads were retained for their network-bridging role but stripped of provider-record storage and serving to cut operating costs, with a preceding A/B experiment running two six-node Kubo fleets across regions — one ignoring Hydra responses, one not — to estimate the latency impact at the 50th, 90th, and 95th percentiles. After the cutover on December 1st, observed retrieval latency degraded slightly more than predicted, partly because skipping non-responsive Hydras still costs an extra DHT hop. The change yielded roughly 36% cost savings, with AWS egress (largely TCP handshakes, RTTs, and multistream-select negotiation) identified as the next-largest remaining cost driver and a candidate for further optimization.

Watch

IPFS Camp Oct 2022

Decentralized NAT Hole-Punching

Dennis Trautwein

This talk presents a measurement study of decentralized NAT hole punching in libp2p, focusing on the DCUtR (Direct Connection Upgrade through Relay) protocol that enables peers behind NATs and firewalls to establish direct connections without centralized infrastructure. Dennis explains how DCUtR uses a relay to coordinate synchronized connection attempts via Connect and Sync messages, then describes a measurement setup combining a honeypot DHT server, a coordination server, and a fleet of clients to observe hole punch outcomes in the wild. The results show roughly an 80% success rate for Go clients, with most successful punches completing on the first attempt, while clients running over VPNs and the Rust IPFS implementation exhibit notably lower success rates, particularly over TCP. The talk also examines the no-stream error pattern in rust-ipfs nodes and discusses transport differences between TCP and QUIC, closing with a call for community participation in a hole-punch measurement campaign.

Watch

IPFS Camp Oct 2022

ProbeLab 2023 Roadmap

Yiannis Psaras

Yiannis Psaras walks through ProbeLab's planned 2023 measurement and protocol-improvement milestones for IPFS and libp2p. The roadmap covers quantifying whether Hydra boosters still add value, consolidating tools like the Nebula crawler and CID Hoarder into a continuous monitoring backend with a unified data lake and dashboards, revisiting the one-second Bitswap-before-DHT delay, finalizing new provider-record expiry and republish intervals, and integrating Thunderdome for automated regression testing of Kubo releases. Further items include shipping Optimistic Provide, replacing magic-number DHT timeouts with adaptive parameters, measuring hole-punching success rates via an opt-in user experiment, and rolling out double-hashing for DHT privacy with attention to the non-backward-compatible transition path. The session closes with GossipSub work focused on instrumenting Filecoin block propagation latency and verifying that the score-function-based mitigation strategies actually exclude misbehaving peers in production.

Watch

IPFS Camp Oct 2022

Data-Driven Protocol Design: What it Is and What Are the Benefits

Yiannis Psaras

Yiannis Psaras introduces ProbeLab's data-driven approach to measuring and improving IPFS and libp2p protocols, framing the team's workflow of hypothesis-driven experiments using crawlers, controlled probes, and node logs. He shares early findings from running the Nebula crawler since 2021, including roughly 200,000 peers seen, a median peer churn of about one hour, and the surprising result that only around 3% of IPFS nodes run on centralized cloud infrastructure. The talk walks through the IPFS DHT provide and lookup process and highlights two performance bottlenecks identified through measurement: a slow provide operation taking tens of seconds despite peers being found in under half a second, and a content lookup pipeline where Bitswap adds a roughly one-second overhead before the DHT walk begins. Psaras closes with pointers to open studies on routing table health, provider record liveness, and the upcoming hole-punching measurement campaign for libp2p.

Watch

IPFS Camp Oct 2022

A Deep Dive into Provider Records

Mikel Cortes

In this IPFS Camp 2022 talk, Mikel Cortes presents a study on provider record liveness in the IPFS DHT, conducted in collaboration with ProbeLab. He explains how content publication actually distributes provider records (links between CIDs and provider PeerIDs) rather than the content itself, and walks through a methodology that publishes 10,000 CIDs without republishing, then probes the K=20 holders every 30 minutes for 36 hours. The results show that liveness is healthy, with a median of 18 successful initial connections and roughly 13 holders still serving records before the 24-hour expiry, while Hydra nodes prune more slowly due to their large databases. He also compares K values from 15 to 40, finding that publication latency scales roughly linearly with K while reachability stays similar, and demonstrates a Hydra-blacklisting experiment showing only modest cost in lookup hops and publish time. The takeaways argue for potentially lowering K from 20 to 15 and extending the 12-hour republish interval to reduce network overhead without harming retrievability.

Watch

IPFS Camp Oct 2022

DHT Lookup Latency Performance

Dennis Trautwein

This talk investigates the role of Hydra boosters in IPFS content routing using the DynamoDB backing store of Protocol Labs' production Hydra deployment, which spans roughly 2,000 heads across 135 instances and handles tens of thousands of RPCs per second. Dennis verifies that Hydras achieve about 97% coverage of the 20-closest-peer neighborhoods across the hash space, then joins provider and peer records to map CIDs to geographic locations, finding that around 55% of unique CIDs are provided from the United States, followed by the Netherlands, France, and Germany. The analysis surfaces a high CID churn rate of roughly 50% per day, that 85% of CIDs have only a single provider, and that the top ten providers account for over half of all CIDs in the network, with one peer alone providing 13%. Latency experiments comparing lookups with and without Hydras show only a marginal speedup and in some regions no improvement at all, suggesting the Hydras' contribution to DHT lookup performance is smaller than intended. The talk also reports on prefetching effectiveness and the dominance of the negative cache among prefetch outcomes.

Watch

PL Demo Day Oct Oct 2022

The IPFS Network from the Hydras' Point of View

Dennis Trautwein

This short demo presents measurement results from analyzing the Hydra boosters' view of the IPFS network using their DynamoDB store of provider and peer records. Dennis verifies that Hydra heads are uniformly distributed across the hash space and appear within the 20 closest peers for over 97% of nodes found by the Nebula crawler, then uses the joined provider and peer records to map CIDs to geographic locations, showing that more than 50% of CIDs originate from the United States, followed by the Netherlands and France. The data reveals roughly 1 billion unique CIDs per day with about 50% daily churn, equivalent to around 120 terabytes of content rotating in and out of the network. He also highlights provider concentration, with a single peer providing 13% of all CIDs, and introduces Antares, a tool that publishes random content and queries it through gateways and pinning services to identify which peer IDs correspond to which operators. The talk closes by previewing follow-up experiments that exclude Hydras from content retrieval and publication to quantify their actual performance impact.

Watch

code.talks Sep 2022

Decentralized Storage with IPFS: How Does It Work Under the Hood?

Dennis Trautwein

A code.talks 2022 introduction to IPFS that walks through how the protocol actually operates beneath the user-facing commands. Trautwein covers content addressing and CIDs, how files are chunked and assembled into Merkle DAGs via IPLD, and how the Kademlia DHT distributes provider records across the network using XOR distance and bucketed routing tables. He explains the full content publishing and retrieval flow, including bootstrapping, routing-table construction, and why provider records are replicated across the 20 closest peers. The talk closes with results from his Nebula-based measurement campaign, including network crawls every 30 minutes, peer churn analysis showing ~50% of nodes leave within an hour, and findings that roughly 97% of IPFS nodes are not hosted on centralized cloud providers.

Watch

PL EngRes Demo Day Sep 2022

IPFS Provider Record Liveness: Final Results

Yiannis Psaras

Yiannis Psaras presents the final results of a joint ProbeLab and Barcelona Supercomputing Center study on IPFS provider record liveness, motivated by churn rates as high as 70% within two hours that could leave content unreachable before the 12-hour reprovide cycle. Using the CIDHoarder tool, the team published provider records and tracked how many of the 20 replica holders remained online and continued serving the records over time. Results show that roughly 15 of the 20 nodes keep records reachable for more than 35 hours even when Hydra boosters are excluded (around 12 nodes), and 15 of the originally selected peers remain among the K-closest to the CID over the same window. Based on these findings, the talk recommends reducing the replication factor K from 20 to 15 and at least doubling the reprovide interval from 12 to 24 hours, which together would cut provider-record-related overhead on content publishers by a substantial margin.

Watch

SIGCOMM '22 Aug 2022

Design and Evaluation of IPFS: A Storage Layer for the Decentralized Web

Yiannis Psaras

In this SIGCOMM 2022 technical session, the authors present the design and evaluation of IPFS (the InterPlanetary File System), a community-driven storage layer powering a growing slice of the decentralized web. The talk walks through how IPFS combines content-addressable storage, a Kademlia-based distributed hash table for peer and content discovery, and the Bitswap block-exchange protocol to let any participant publish, locate, and retrieve data without relying on centralized servers. Drawing on large-scale measurements of the live network, the speakers examine real-world performance — including content-routing latency, retrieval times, and the geographic distribution of peers — and discuss the trade-offs between decentralization, scalability, and user-perceived speed. They close by reflecting on lessons learned from operating a production decentralized storage system and outline open challenges for making the decentralized web competitive with traditional content delivery infrastructure.

Watch

IPFS þing Jul 2022

libp2p NAT Hole Punching Success Rate

Dennis Trautwein

A talk from IPFS þing 2022 on measuring the success rate of libp2p's hole punching protocol, DCUtR (Direct Connection Upgrade through Relays), which became broadly viable after Kubo 0.11 turned every node into a limited relay. Trautwein explains how NATs and firewalls block direct peer connections and walks through the DCUtR handshake, where peers exchange Connect and Sync messages over a relay and use round-trip-time measurements to coordinate simultaneous dial-outs. He then describes the punchr measurement setup: a honeypot that walks the DHT to attract NATed peers, a server exposing a gRPC API, and Go and Rust clients that perform hole punches and report outcomes. Initial results from a single home-network client show a ~72% success rate across ~13,300 attempts to ~2,500 unique peers, with 97% of successful punches landing on the first attempt and a median duration of around 0.9 seconds. He closes with next steps around comparing TCP vs. QUIC behavior, deploying more vantage points, and a call for participants to run clients from their own networks.

Watch

IPFS þing Jul 2022

Provider Records: Do They Stay Alive Long Enough?

Mikel Cortes

At IPFS þing 2022 in Reykjavík, Mikel Cortes presents an early version of ProbeLab's measurement study on whether IPFS provider records actually stay alive for their intended 24-hour lifetime. He details the publication mechanics (K=20 closest peers via XOR distance, 12-hour republish interval) and his measurement tool, which publishes random CIDs, pings the original holders every 30 minutes, and re-runs DHT lookups to track the in-degree of those holders over time. The data shows publication is slow (median around 12 seconds per CID, 95th percentile up to 44 seconds), roughly two of the 20 chosen peers are unreachable at publish time, and the original holders remain stable and close to the CID well beyond 24 hours, with Hydra nodes notably continuing to serve records past the expiry due to slower garbage collection. Comparing K=15, 20, and 25, he finds little practical difference in retrievability and only marginal latency gains, leading him to argue that extending the republish interval beyond 12 hours is a more promising optimization than lowering K. The talk closes with an open discussion on dynamic per-node K values driven by content popularity and observed reachability.

Watch

Paris P2P #1 Apr 2022

IPFS Network Measurements and Improvement Opportunities

Yiannis Psaras

Yiannis Psaras introduces the IPFS Observatory, a measurement effort aimed at identifying bottlenecks and quantifying improvements in libp2p and IPFS protocols. He walks through two concrete findings: the content provide process takes tens to over 100 seconds but the relevant peers are typically discovered within the first second (motivating the Optimistic Provide work), and the Bitswap broadcast step adds a roughly one-second delay before the DHT lookup begins despite a low discovery rate. The talk also covers results from the Nebula crawler, including a roughly two-hour median session length, agent version uptake comparisons showing Kubo 0.9 as more stable than newer releases, churn behaviour of Hydra boosters, and the surprising finding that under 2% of IPFS nodes run on major cloud providers while the top 10 ASes hold around 65% of observed addresses. Psaras outlines further studies on hole-punching success rates, GossipSub performance, and DHT routing table health, and points to the dgm.xyz grants platform and an upcoming academic workshop for community involvement.

Watch

Paris P2P #1 Apr 2022

Gossipsub: A Gossip-Based Pubsub Protocol

Yiannis Psaras

Yiannis Psaras explains GossipSub, the libp2p pubsub protocol used by Filecoin and other blockchain networks, and the security extensions developed to harden it against Sybil, eclipse, censorship, cold-boot, and covert flash attacks. He details the protocol mechanics including the global mesh, local mesh degree D with D_low and D_high thresholds, heartbeat-driven graft and prune operations, and the eager-push plus lazy-pull gossip exchange where only message IDs are forwarded outside the mesh. The core hardening is a per-peer score function combining time-in-mesh, first-message deliveries, mesh failure penalties, invalid messages, an application-specific term, and IP colocation, complemented by mitigation strategies such as controlled mesh maintenance, flood publishing of first-seen messages, and prune backoff. Using Testground simulations with 5,000 peers and a 20:1 attacker-to-honest ratio, Psaras shows that GossipSub keeps propagation under the six-second Filecoin deadline across all attack scenarios while the Bitcoin and Ethereum 1 pubsub equivalents degrade significantly, and demonstrates how honest peers progressively prune attacker connections to reconstitute an honest-only mesh.

Watch

Paris P2P #1 Apr 2022

Optimistic Provide: Optimize the IPFS DHT

Dennis Trautwein

Presented at P2P Paris in April 2022, this talk introduces "Optimistic Provide," a proposal to make the IPFS DHT provide operation roughly an order of magnitude faster. Trautwein first reviews how Kademlia, routing tables, and the lookup algorithm drive content advertisement, then shows measurement data where the median provide takes around 30 seconds and the 95th percentile exceeds two minutes, even though the closest peers are typically discovered in under one second. He proposes two approaches: running multiple parallel lookups from different seed peers and terminating once their query queues intersect, or estimating peer proximity using a local network-size estimator (based on a published technique that avoids gossip) to decide when a discovered peer is "close enough" to receive the provider record. Preliminary results show the multi-query approach intersects too early and yields peers that are too far in XOR distance, while the proximity-estimation approach produces closeness in the same order of magnitude as the status quo and a network-size estimate that tracks crawler ground truth. Next steps include sweeping parameters, evaluating the network-size estimator across a distributed fleet, and measuring the actual end-to-end publication-time improvement.

Watch