Introducing ProbeLab Sonar

Introduction

Today we’re launching ProbeLab Sonar in public beta. Sonar takes a PeerID, an ENR or a Multiaddress, probes the peer live from our infrastructure, and shows the results next to what our crawlers have recorded about that peer over the years.

Motivation

When a p2p application misbehaves, operators often have to guess where the problem is. It can be a bug in their own code. It can also be the network: the node sits behind a NAT, other peers can’t reach it, or the overlay topology has drifted into a bad state. From inside your node, you only see your own side. Sonar lets you debug your node from the outside and shows how the rest of the network sees it. This is possible thanks to ProbeLab's tooling that has run 24/7 for the last few years.

What works in the beta

Today, ProbeLab Sonar can tell you:

• whether other peers can reach your node, and on which addresses
• which client and protocols the network sees you run
• how many peers keep your node in their DHT routing tables
• whether a sensitive port is open to the internet
• what changed since our crawlers last saw you

And this is just the beginning. We have many more ideas in the pipeline - see below.

At the time of writing, we probe peers on IPFS, Filecoin, discv5 (Ethereum, Gnosis, and more), Celestia, Polkadot, and Avail - with more to come.

Sonar runs its full set of live probes for two kinds of input:

• PeerIDs from any of our monitored libp2p-based networks.
• ENRs from Ethereum. Sonar finds the node through discv5 and then connects to it over libp2p (consensus-layer nodes) or devp2p (execution-layer nodes), depending on what the record declares.
• Multiaddresses like /ip4/1.2.3.4/tcp/5678. Sonar connects to the address and tries to figure out what protocol is spoken there.

Each lookup returns a collection of cards.

Cards

Identifier Encoding

You can see live the breakdown of the identifier into its components: multibase, multihash, CID, multicodec and more, as shown below.

ENRs encode even more information, such as:

Agent and Protocols

These cards show the user agent that your peer advertises and the protocols it supports. Further, it shows historical information about the points in time when the peer has updated.

GeoIP

The GeoIP card shows where the peer’s IP address is located. It also labels whether the peer runs in a cloud environment, behind a VPN, or over Tor.

Reachability

Sonar dials each advertised multiaddress one at a time and reports the result: how long the connection took and the round-trip time (RTT) once connected. If a dial fails, you see the full error message.

Sonar learns a peer’s addresses from several places: the identifier itself (ENRs carry addresses), the DHT, IPNI, and the libp2p identify exchange. The symbols on the right show where each address came from. The last source, “Recorded”, is off by default. Turn it on to see every address our crawlers have stored for this peer. We have crawled these networks for years, so this list can go back a long way.

If Sonar experiences problems connecting to one or more addresses you can inspect the error right there:

DHT Overview

If Sonar can connect to the peer, it drains the peer’s routing table to check that it is healthy. The card shows how many peers are in each bucket. Hover over a cell to see their PeerIDs. Below that, the card shows how many other peers in the network have this peer in their routing tables.

IPNI Indexer Status

For Filecoin and IPFS peers, it matters whether the InterPlanetary Network Indexers (IPNI) know about them. Sonar checks whether the peer is registered as a provider. It then fetches the latest state, such as the CID of the most recent advertisement or the last error.

Port Exposure

Depending on the network a peer belongs to, Sonar checks a small set of ports. A node that is deployed in a hurry can leave a sensitive port open to the internet, and that makes it vulnerable to attacks.

Linking Straight to a Result

Every result page has its own URL, so you can share it in a chat or put it in a bug report. The URL takes three parameters:

Card names: networks, dissection, agent, protocols, consensus, geoip, reachability, routing-table, ipni, port-exposure.

This means by providing &cards=dissection you can replicate our ENR-Viewer behavior: Example.

# Everything Sonar knows about a peer
https://probelab.io/sonar/search/?q=<peer-id>

# A Qm… string is both a valid CID and a valid PeerID. Read it as a PeerID:
https://probelab.io/sonar/search/?q=Qm...&type=peer-id

# Only the cards about reachability
https://probelab.io/sonar/search/?q=<peer-id>&cards=reachability,geoip,port-exposure

# All cards except the map
https://probelab.io/sonar/search/?q=<enr>&cards=-geoip

Sonar ignores card names it doesn’t know, so your links keep working if we remove or rename a card. If a cards value selects no cards at all, you get the full page.

What's Next

The cards above use our own crawlers and live probes. Next, we want to add outside sources: public chain RPC endpoints, ecosystem services such as the FOC Observer, and free internet APIs. The DNS card is already in progress. Everything else on this list is an idea, and we will build first the ones you ask for.

Identifiers

We plan to support more identifier types:

• CIDs
• Multihashes
• IPNS names
• DNSLink names

Sonar already recognizes these identifiers but can't probe them yet. For a CID, you still get an offline breakdown of how it is encoded.

CIDs and multihashes come first. For these we want to show who provides the content, from the IPNI indexers and from the Amino DHT, and check whether the content can be retrieved. IPNS names and DNSLink names follow after that.

DNS

Many peers advertise /dns4, /dns6 or /dnsaddr addresses, and every peer that uses AutoTLS gets a name under libp2p.direct. The DNS card resolves these names through several public resolvers and follows the complete _dnsaddr tree. It flags the problems that make libp2p dials fail: a _dnsaddr chain deeper than go-libp2p follows, a response too large for UDP, an AutoTLS name that points to a different IP address or PeerID, resolvers that disagree, or a CAA record that blocks Let's Encrypt.

Every network

Some useful facts come straight from the years of data our crawlers have recorded. Others need one call to a public API:

• How often our crawlers found the peer online in the last 30 days, and when it dropped out.
• What share of the network runs the same client and version.
• How many other PeerIDs our crawlers see behind the same IP address. A high number can point to a hosting provider or a Sybil cluster.
• Whether a newer release of the client exists, and whether the running version has known security advisories, from GitHub releases and OSV.dev.
• The ASN, the announced prefix and its RPKI status from RIPEstat, plus the reverse DNS name of the IP address.
• Other open ports and known CVEs on the host, from Shodan's free InternetDB. This would extend the Port Exposure card.

Network-specific cards

IPFS

• Whether delegated-ipfs.dev returns the peer, and with which addresses. Browser nodes usually find peers only this way.
• When the peer's AutoTLS certificate expires, from the certificate transparency logs.
• The IPNS record that the peer publishes. Every PeerID is also an IPNS name, so we can fetch the record from the DHT, verify its signature, and show its value and age.
• The gossipsub topics that the peer subscribes to

Filecoin

Filecoin Onchain Cloud (FOC) storage providers register their PeerID on chain. For a PeerID that belongs to a FOC provider, a card could show:

• the registry entry: name, location, and whether the provider is approved and endorsed
• the Curio version and the /pdp/ping result from the provider's service URL
• deal and retrieval success rates from our dealbot measurements
• the proof fault rate
• how much data the provider stores, and for how many clients, from the FOC Observer

After FOC, we want to cover classic storage providers. Their miner actor stores a PeerID and a list of multiaddresses on chain, and clients use these to find the provider. If the on-chain data does not match what the provider advertises live, deals and retrievals can fail. Sonar could show both side by side, next to the provider's power and faults.

Ethereum and Gnosis

Here is a mock-up of what we could show if we detect that a PeerID or ENR belongs to an Ethereum consensus-layer node:

With a beacon node API to compare against, the card could also show:

• how many slots the node is behind the canonical head
• whether the node's finalized checkpoint matches the canonical one. If it doesn't, the node follows a different fork.
• whether the node is ready for the next hard fork, from the fork version announced in its ENR
• how many PeerDAS custody groups the node declares, and whether it acts as a supernode

Execution-layer nodes send a fork ID (EIP-2124) and their head block hash when they connect over devp2p. With a JSON-RPC endpoint to compare against, the same sync and fork checks work for them too.

Optimism and Base

Sonar already reads the chain ID from the opstack entry of an ENR. It could also check that the node subscribes to the block gossip topics of that chain. An OP Stack node needs these topics to receive new blocks from the sequencer.

Celestia

• The node type (bridge, full or light), from the protocols the node supports.
• How far the node lags behind the chain tip. Sonar would ask the peer for its latest header and compare it with a public Celestia RPC endpoint.

Polkadot and Avail

Both networks run on Substrate, so the same ideas work for both:

• Whether the peer is an active validator. Validators publish signed records in the DHT that link their authority key to a PeerID and addresses. If Sonar collects these records, it can tell whether a peer is a validator and whether the addresses in its record work.
• The node name and version from public telemetry, which lists nodes by PeerID.
• The best block and the genesis hash from the block announce handshake. These show whether the node follows the right chain and how far behind it is.

New networks

We also want to support more libp2p networks, such as Starknet, Mina and Aztec.

Which of these cards would you use first? Is there a check you still run by hand that Sonar should do for you? Tell us. If your network uses libp2p and you want to see it in Sonar, tell us too.

Monitoring Your Node Over Time

Sonar probes your node only when you ask. Each lookup is a snapshot. It shows the state of your node at that moment, next to what our crawlers have recorded about it. Our crawlers visit peers regularly, but they don't run the full set of Sonar probes and they don't alert anyone.

We want to add continuous monitoring. You would register your node once, and Sonar would probe it on a fixed schedule. From these probes, we could send you daily, weekly or monthly reports, and alerts when something changes. For example:

• your node goes offline
• fewer peers in the network can see your node
• a sensitive port opens to the internet

If you want this for your nodes, let us know.

Available today

ProbeLab Sonar is free to use at probelab.io/sonar today! If a result doesn’t match what you know about your node, tell us. Send the result URL and the trace ID from the bottom of the page. That gives us enough to find out what went wrong, or to confirm that Sonar found a real problem with your node.

We have built Sonar so that operators and protocol teams can see what the network sees. We’ll keep adding networks, identifier types and probes during the beta.