160 Dials: How Two go-libp2p Defaults Halved Our View of Polkadot

Introduction

ProbeLab crawls the Polkadot network every two hours with Nebula and publishes the results on probelab.io/polkadot: network size, client versions, cloud providers, and more. Since mid-2025, these numbers have been jumpy. On 6 October 2026, one crawl reached 620 Polkadot nodes and the next one, two hours later, reached 949. Every chart built on these crawls inherited the noise.

Our first suspect was litep2p, the Rust networking library behind most of the Polkadot nodes we reach today. It has known interoperability issues with go-libp2p, which Nebula is built on (litep2p#467, polkadot-sdk#10364). That suspicion turned out to be wrong. The cause was two limits in go-libp2p's dialer. We had tightened one of them ourselves in May 2025, without noticing, when we moved the crawler to a smaller machine as funding for operating the data collection ran out.

This post explains how go-libp2p dials a peer, why Polkadot in particular runs into these limits, how we tracked the problem down, and what changed after we raised one limit. Reachable peers per crawl went from 753 on average to 1,443, and the variation between crawls dropped from a standard deviation of 102 peers to 13. Then a third go-libp2p default showed up: a cap on how many addresses go-libp2p stores per peer. With all three out of the way, each crawl since the evening of 13 October 2026 has reached between 1,421 and 1,461 peers.

None of these limits caused an error and other data collectors seemingly run into the same problem. While we fixed our data collection the observered behaviour in the Polkadot network certainly hints at a misconfiguration.

TL;DR

• go-libp2p dials at most 8 addresses of one peer at the same time, and at most 160 TCP dials per host at the same time. A TCP dial to an address that doesn't answer takes up to 5 seconds.
• Polkadot nodes advertise many addresses, and most of them don't answer. In October 2026, a reachable node advertised 26 addresses at the median, and one in ten advertised 90 or more.
• Raising the TCP Dial limit from 160 to 4,096 per host nearly doubled the reachable peers per crawl, from 753 to 1,443 on average. The share of peers that the next crawl reached again rose from 65% to 96%.
• go-libp2p v0.50 keeps at most 64 addresses of a peer it isn't connected to. Removing that cap brought back about 50 reachable peers per crawl.
• Nebula v3, our own version of Nebula, includes all of these fixes and a dial timeout that grows with the number of addresses a peer advertises.
• Nebula v3 records the result of every address it dials. This showed that in each crawl, about 1,000 peer IDs belong to nodes that now answer under a new peer ID.
• Continuous monitoring of crawl quality would have caught all of this early. For a whole year, only 49% to 65% of the peers that one crawl reached were reached again two hours later, and nothing alerted on it. Watching this number, the reach of known peers, and the host's network counters would have shown the problem in 2025, not in October 2026.

How Nebula Crawls a Network

Nebula starts from a list of bootstrap peers. A pool of workers, 1,000 for Polkadot, takes peers from a queue. A worker connects to a peer and then sends it 16 FIND_NODE requests, one for each bucket of the peer's routing table, which returns a large part of the peers the node knows. Every new peer goes into the queue, and the crawl ends when the queue is empty. On Polkadot, a crawl takes about five minutes and finds about 27,000 peer IDs, most of which we can't connect to.

One detail matters for the rest of this post. Nebula spreads its workers over several go-libp2p hosts, one per CPU core of the machine.

How go-libp2p Dials a Peer

When Nebula asks go-libp2p to connect to a peer, go-libp2p takes all known addresses of that peer and works through them in a fixed order. The figure below shows the path a dial takes and the two limits along the way.

Address Ranking

go-libp2p doesn't dial all addresses at once. Its dial ranker sorts them: QUIC before TCP, IPv6 before IPv4, and lower port numbers first. The algorithm is called Happy Eyeballs. It also gives the first one or two addresses a head start of 250 ms, so that a fast first address can win before the others start. For a peer with 30 TCP addresses, whether the working one is tried early depends on its port number.

The Per-Peer Limit: 8

swarm.DefaultPerPeerRateLimit is 8. At most 8 addresses of one peer are being dialed at any time. The 9th address waits until one of the first 8 finishes. The value is a package variable, so you can change it before you create a host.

The Per-Host TCP Limit: 160

swarm.ConcurrentFdDials is 160. Each host runs at most 160 dials at the same time that need their own file descriptor. That covers TCP and everything built on it, such as WebSocket. QUIC, WebTransport, and WebRTC share one UDP socket and skip this limit. The value is a constant, but go-libp2p reads LIBP2P_SWARM_FD_LIMIT when it creates a host. Dials beyond the limit wait in one shared queue, first come, first served, no matter which peer they belong to.

The Connect Timeout: 5 Seconds

go-libp2p's TCP transport gives each connection attempt 5 seconds. An address behind a firewall that drops packets never answers, so a dial to it holds its slot for the full 5 seconds.

WebSocket dials can take longer. go-libp2p's WebSocket transport allows a dial 15 seconds, and since WebSocket runs over TCP, the dial holds a TCP slot the whole time. Many Polkadot nodes advertise WebSocket addresses next to their plain TCP ones.

The Dial Timeout

Nebula sets a deadline for the whole dial, which we set to 10 seconds for Polkadot. The clock starts when the dial starts, so time spent waiting in either queue counts against it. Nebula also passes the same value to go-libp2p as the timeout for each single address, so a WebSocket dial ends after 10 seconds instead of 15.

Putting It Together

Two numbers follow from these rules. First, how many addresses of one peer go-libp2p can try within a dial timeout T, assuming every address that doesn't answer takes the full 5 seconds:

n_{\max} = 8 \cdot \left\lfloor \frac{T}{5\,\text{s}} \right\rfloor

For T = 10\,\text{s}, that's 16 addresses, and fewer if some of them are WebSocket addresses that take longer. Second, dials never wait for a slot as long as every worker can use its full per-peer allowance at the same time:

\text{workers} \times 8 \;\le\; \text{hosts} \times L

where L is the per-host limit, 160 by default. On a 2-vCPU machine with 1,000 workers, the left side is 8,000 and the right side is 320.

These defaults make sense for a node that dials a handful of peers. A crawler dials tens of thousands.

Why Polkadot Hits These Limits

Most networks we crawl are easy on these limits. In recent crawls, an Ethereum consensus peer advertised 1.2 addresses on average and an IPFS peer 5.4. On top of that, IPFS connects mostly over QUIC, which doesn't use the TCP slots.

Polkadot is different. In October 2026, a reachable Polkadot node advertised 35 addresses on average. The figure below shows how this number grew during 2026: the median stayed between 6 and 8 until March, then rose to 26. One in ten nodes now advertises 90 addresses or more. In a single crawl, Nebula sees about 750,000 TCP addresses.

Below you can see a line chart of the number of addresses Nebula dialed per reachable Polkadot peer, by month from November 2025 to October 2026. The median stays between 6 and 8 until March 2026, then rises to 26 in October 2026. The 90th percentile rises from 51 to 90. A reference line at 16 marks how many addresses a 10-second dial timeout covers.

Most of the addresses a peer advertises don't answer. From the outside, we can't tell which of a node's addresses is the working one until we dial it, so filtering addresses up front isn't an option. Nebula v3 records the result of every address it dials, which shows where the dead addresses come from. About 2,000 peers advertise around 25 addresses each on that port, from 23,303 different IP addresses. We know the client of 41 of these peers from the past month, and 40 of them run the Mosaic Chain collator. Next come 30334 and 30333, the ports Polkadot nodes usually listen on, and then ports 1024 to 1027, which look like short-lived port mappings on home routers. We don't know why the number of addresses per node rose in 2026.

Putting the pieces together, you get a long queue. 1,000 workers each dial up to 8 addresses, most of them dead, and each dead address holds a slot for 5 seconds, all competing for only 320 slots. Reachable nodes sit in the same queue.

Root Cause Analysis

It Wasn't litep2p

litep2p#467 describes how litep2p closes a Kademlia stream after one response, while go-libp2p may want to send more requests on the same stream. Nebula opens a new stream for every request, so this doesn't affect it. In the seven days before the fix, 89% of Nebula's connections to litep2p nodes ended with all 16 bucket requests answered. litep2p nodes were among the most reliably reachable peers. Parity's two bootnodes, both running litep2p, were reached in every one of 24 crawls.

It Wasn't Discovery

When a crawl missed a peer that the previous crawl had reached, Nebula had almost always found that peer again. The connection itself failed: the dial ran until the 10-second dial timeout ended it.

It Wasn't our VM

During a crawl, the 2-vCPU machine used at most 0.33 CPU cores, 0.74 GB of its 4 GB of memory, and 612 of its 65,535 file descriptors.

Most Dials Never Left the Machine

The clearest evidence came from counting packets. In the last crawl before the fix, 28,666 peer visits failed. If every failed peer had dialed up to 16 of its TCP addresses, the most that 10 seconds allow, Nebula would have made about 187,000 TCP connection attempts. Linux sends up to three SYN packets per attempt within 5 seconds, so that's roughly 560,000 packets. The crawler's network interface counted about 267,000 packets sent for the entire crawl, including all Kademlia traffic with the 763 peers it reached. Most connection attempts never reached the network. They waited in a queue and were cancelled when the dial timeout ran out.

That pointed at the per-host limit. The crawler ran on 2 vCPUs, so it had 2 hosts × 160 = 320 TCP dial slots.

Our Fix: Increase the Swarm's File Descriptor Limit

On 7 October 2026, we increased the LIBP2P_SWARM_FD_LIMIT from 160 to 2048 for the Polkadot crawler, which gives 4,096 slots on 2 hosts. Nothing else changed: the same 2-vCPU machine, 1,000 workers, and the 10-second dial timeout.

The figure below shows the week around the change. Before it, crawls reached between 534 and 970 peers, and only 47% to 83% of the peers one crawl reached were reached again by the next. After the change, every crawl reached between 1,412 and 1,456 peers, and 94% to 98% were reached again.

The extra reach had a modest cost:

Packets sent per crawl went up about fourfold, which shows that the extra dials now leave the machine.

Two hours later, we raised the limit for all our crawlers to 4,096 per host, which doubled the Polkadot crawler's slots again to 8,192. The result didn't change: 1,438 peers per crawl on average with 4,096 slots, 1,445 with 8,192. The per-host limit is no longer the bottleneck.

The Second Limit: 8 Dials per Peer

With enough slots, the per-peer limit is what remains. A 10-second timeout covers two rounds of 8 addresses, so 16 addresses. A peer whose working address comes later in the dial order is often never tried.

The figure below shows this for the 16 crawls after the fix. For each peer, we took the TCP address that Nebula connected to between 9 and 11 October, and looked up where that address falls in go-libp2p's dial order. We left out peers that changed their peer ID during these two weeks. The section on peer IDs below explains why.

Up to position 8, Nebula reached 98% of these peers, and 88% at positions 9 to 16. Beyond that, the rate drops to 50%, then 31%, then 14%. That cost almost 60 peers per crawl, about 4% of the reachable network. It's a lower bound, because we can only count peers whose working address we've seen at least once.

Improving Nebula

Raising the per-peer limit would bring the queue back. Each worker would then hold up to 32 slots instead of 8, which makes 32,000 possible dials for 1,000 workers. In local test crawls, a per-peer limit of 32 lowered the share of reliably reachable peers we reached from 87% to 64%. So we changed Nebula, in what is now Nebula v3, to treat the configured dial timeout as a minimum. For peers with more addresses than the timeout covers, it adds 5 seconds for every additional round of 8, up to one minute by default:

T_{\text{peer}} = \min\left(T + 5\,\text{s} \cdot \max\left(0,\; \left\lceil \frac{n}{8} \right\rceil - \left\lfloor \frac{T}{5\,\text{s}} \right\rfloor\right),\; 60\,\text{s}\right)

With T = 10\,\text{s}, a peer with up to 16 addresses keeps 10 seconds, and a peer with 50 addresses gets 35 seconds. Each peer still dials at most 8 addresses at a time, so the slot math doesn't change. The cost is a longer crawl, because unreachable peers with many addresses keep their worker busy longer. To compensate, we raised the workers together with the slots. With 2,000 workers, 8,192 slots per host keep 2,000 × 8 = 16,000 at or below 2 × 8,192 = 16,384. Today, the Polkadot crawler runs 3,000 workers.

One More go-libp2p Default: 64 Addresses per Peer

Nebula v3 also bumped from go-libp2p to the latest v0.50.0. Between 13:04 and 18:25 UTC on 13 October, its crawls reached 1,378 to 1,404 peers, about 50 fewer than Nebula v2 after the first fix. The reason was another default. Newer versions of go-libp2p limit how many addresses the in-memory peerstore keeps for a peer that the host isn't connected to, and the limit is 64. Once a peer has 64 addresses, each new one replaces the address that expires first, which for a crawler is the one it learned first. The code comment calls 64 "well above observed real-world maxima (~26 for well-connected multi-transport nodes)". In Polkadot, the median reachable node advertises 26 addresses, and one in ten advertises 90 or more.

So for peers with more than 64 addresses, go-libp2p dropped some of them before Nebula could dial them. When the dropped addresses included the working one, Nebula v3 reached the peer in 70% of crawls, against 92% for Nebula v2 with go-libp2p v0.38. Nebula v3 now tunes the peerstore to avoid thel limit and these peers are back at 90%. That's about 50 reachable peers per crawl.

Where We Are Now

Since 13 October 2026 at 18:38 UTC, the Polkadot crawler has run Nebula v3 with all of these changes, 3,000 workers, and an 8-second minimum dial timeout. Here are its 23 crawls up to 15 October, 14:25 UTC, next to the crawls after our first fix:

The dynamic timeout does what it should. The figure below compares the 16 crawls after the first fix with the first seven Nebula v3 crawls, measured as in the figure for the second limit. Beyond position 16, Nebula used to miss almost 60 peers per crawl. In these seven crawls, it missed about 16. At positions 1 to 8, Nebula v3 comes out 2 points lower (96% against 98%). We don't know why yet; the shorter minimum timeout is one candidate.

Nodes That Change Their Peer ID

In each Nebula v3 crawl, 930 to 1,100 peers fail with peer_id_mismatch. It also records the peer ID of the node that answers. In 77% to 90% of these cases, Nebula reaches that node in the same crawl under its new peer ID. A few operators cause most of them. In the crawl at 14:25 UTC on 15 October, 31 nodes running polkadot-parachain v1.24.3 accounted for 594 old peer IDs, and 13 Peaq collators for 155. In the past two weeks, one address answered under 145 different peer IDs in 168 connections, so that node had a new peer ID almost every time we reached it. The node name changes along with the peer ID. This suggests that these nodes start from scratch on every restart, without the network key they stored before.

If you run Polkadot nodes, keep the node key across restarts, for example with --node-key-file or a persistent base path. Every new peer ID leaves old entries in other nodes' routing tables, and other nodes dial those entries in vain.

Beyond Polkadot

We raised the limit to 4,096 per host for all our go-libp2p crawlers. The effect depends on the network:

• discv5: about 3% more reachable peers per crawl
• IPFS: no change. Most IPFS connections use QUIC, which doesn't use the TCP slots.
• Smaller networks such as Avail, Celestia, and Filecoin: these crawls visit at most about 700 peers and never fill even 320 slots.

Nebula v3 doesn't need this setting anymore. It sizes each host's TCP dial slots from its worker count: the workers per host times the per-peer limit, at least go-libp2p's default of 160, and no more than the open-file limit allows. It logs the result at startup and warns if the slots can't keep up with the workers.

What Monitoring Would Have Caught

Each of the above limits would have shown up in at least one of the following signals:

• Retention between consecutive runs. For Polkadot, it should stay above about 95%. A drop points at the crawler, not at the network.
• The reach of a fixed set of known peers, such as bootnodes and nodes that have been reachable for weeks.
• The spread between runs, and step changes in any count right after a deployment.
• The host's network counters, such as packets sent and dropped. If far fewer packets leave the machine than your dials need, most dials never reach the network.
• How dials end. If most failed dials end exactly at your dial timeout, they probably waited in a queue.
• Library behavior. Nebula v3 now has canary tests that pin the go-libp2p behavior it relies on: the transport timeouts, LIBP2P_SWARM_FD_LIMIT, the per-peer limit, the peerstore cap, and the dial errors at the deadline. A go-libp2p update that changes how Nebula dials fails a test instead of changing the crawls without an error.

What's Next

The Polkadot data on probelab.io has been stable again since 13 October 2026, and the charts built on it no longer swing from one crawl to the next.

We'd also like go-libp2p to make these bottlenecks visible. Metrics for the dial limiter, such as the queue length and the slots in use, would have shown us the cause directly. We plan to propose this upstream. When a dial times out, go-libp2p also returns only the timeout and drops what happened to each address. Our fork of go-libp2p returns the result of every address instead, which is how Nebula v3 tells a dead address from one it never tried.

We don't know why Polkadot nodes started advertising so many more addresses in 2026. If you work on polkadot-sdk or litep2p and know the reason, we'd like to hear from you.

Nebula v3

Nebula is open source, and its public repository stays available. Nebula v3 is our own version of it, and it now runs our Polkadot crawls. The Nebula changes from this post are part of it:

• The dial timeout grows with the number of addresses a peer advertises.
• It sets go-libp2p's TCP dial slots from the worker count and warns at startup if they can't keep up.
• It dials every address of a peer, without go-libp2p's limit of 64.
• It records the result of every address it dials, so you can tell an address that timed out from one that refused the connection or was never tried.
• When a different node answers, it records that node's peer ID.
• Canary tests fail when a go-libp2p update changes how Nebula dials.

Nebula v3 crawls 33 networks from 23 projects, including IPFS, Filecoin, Ethereum, Celestia, Avail, the Bitcoin family, Monero, and HyperDHT, and it reaches Bitcoin-family nodes over Tor and I2P. If you'd like to crawl your network with Nebula v3 or use its data, get in touch.

Come Work With Us

If you need network measurements or custom analyses for your peer-to-peer network, get in touch.

References

• go-libp2p v0.50.0: swarm_dial.go (ConcurrentFdDials, DefaultPerPeerRateLimit), limiter.go (LIBP2P_SWARM_FD_LIMIT), dial_ranker.go, tcp.go (connect timeout), websocket.go (WebSocket handshake timeout), addr_book.go (addresses per peer)
• litep2p#467: Support long-lived inbound substreams reused for multiple requests, and the proposed fix litep2p#647
• polkadot-sdk#10364: go-libp2p polkadot/litep2p interop challenges
• Nebula, the open-source crawler
• ProbeLab Polkadot network page