Troubleshooting the network agent

Restart loops, never online, tests that never run, mesh peers unreachable, traps not arriving, 'unavailable' or 'sampled' dependencies, resource use, and removing an agent cleanly — with where to look first.

networkagenttroubleshootingconnection-keyofflinemeshsnmppassiveebpflogs

Where to look first

Almost every problem is answered by two things: the agent's log and the Agents tab. Open both before anything else.

Windows Linux
Is it running? Start Menu → VerOps Network Agent → Agent Status (or sc query VerOpsNetworkAgent) systemctl status verops-network-agent
What is it saying? View Agent Log — C:\ProgramData\VerOps\Network Agent\logs\network-agent.log journalctl -u verops-network-agent -f
Is it alive locally? curl http://127.0.0.1:9877/health — answers only from the host itself
Does the platform see it? Network → Agents — status, last heartbeat, version, capabilities

The service starts and stops, over and over

The log says Connection key not configured and the service restarts every 10 seconds. The agent exits on purpose when it has no key — a service that silently idles is worse than one that visibly fails. Put the key in [platform] connection_key (or AGENT_CONNECTION_KEY) and restart. On Windows this state almost always means the config page was skipped by an unattended install that did not pass /ConnectionKey=; re-run the wizard, or the silent command with the key.

It runs, but never shows online

  1. The key was rejected. The log shows an authentication failure from /api/network/agents/authenticate. The key was mistyped, belongs to an agent that has since been deleted, or was rotated in the platform after this host was configured — rotation invalidates the old key immediately. Copy the current key from the agent's card (rotate again if you no longer have it) into the config and restart.
  2. It cannot reach the platform. Connection refused, timeouts or TLS errors to the platform URL. From the same host, curl -sv https://app.verops.io/actuator/health tells you whether it is DNS, a proxy, an egress firewall or an interception certificate. If outbound traffic must go through a proxy, set HTTPS_PROXY in the service's environment — the agent honours the standard proxy variables — and make sure the platform's certificate chain is trusted by the host if the proxy intercepts TLS.
  3. Wrong platform URL. A plain http:// URL for a public host, a trailing path, or a typo. The value must be the origin you sign in to — https://app.verops.io — and nothing more.

Online, but tests never run or results never arrive

  • The test is not assigned to this agent. Open the test and check its source agents. A test targeted by region or tag only reaches agents whose region or tags match exactly — EU and eu are different labels.
  • The test is disabled. The Tests tab marks it; enable it.
  • Ping, traceroute or MTU fail while TCP and HTTP work. Those three need raw sockets. The installers arrange the privileges (LocalSystem on Windows; root or ambient CAP_NET_RAW/CAP_NET_ADMIN on Linux), but a hand-written unit, a hardened container, or a host firewall that blocks outbound ICMP will show exactly this pattern. The log names the probe that failed and why.
  • Results arrive late. The agent polls for assignments every 30 seconds and heartbeats every 30 seconds by default; the first result of a new test lands after its first scheduled run, not instantly.

Mesh shows a peer as unreachable

Agents measure each other over TCP 9876 (the [mesh] listen_port). A peer that is online but red in the matrix usually cannot be reached on that port: a firewall between the two hosts, or the agent advertising an address the peer cannot route to. On a host with several addresses — a public interface and a VPN or private one — set [agent] advertise_ip to the address the other agents can reach, and bind the mesh with [mesh] bind_address to that same address so the port is not exposed on the public side. Agents learn each other's addresses from the platform; nothing needs listing by hand.

SNMP traps do not arrive

  • The listener is off by default. Enable it for this agent (from the platform, or [snmp] enabled = true) and confirm the log says it bound UDP 162.
  • Port 162 is privileged. The installers grant it; a least-privilege Linux setup needs CAP_NET_BIND_SERVICE in the unit, which the shipped unit already carries.
  • The device must be sending traps to this agent's address, and a host firewall must allow inbound UDP 162. tcpdump -ni any udp port 162 on the agent host settles whether anything is arriving.
  • If a community allow-list is set, traps with any other community are dropped silently by design. SNMPv3 traps show as encrypted until a matching user is configured.

The device-side walkthrough is in SNMP traps.

Dependencies shows "unavailable" or "sampled"

This is the agent being honest, not broken. Passive observation is off by default ([passive] enabled, local file, restart required). Once on, the agent uses the best collector the host permits and says which:

  • Unavailable — no collector could start on that host, and the reason is reported with it. On Linux that means eBPF could not load and the socket table could not be read either — typically a container or sandbox that hides /proc/net from the agent.
  • Sampled · counts are lower bounds — eBPF could not load (older kernel, missing BTF, a tracepoint layout the probe refused to attach to rather than report the wrong port), so the agent reads the socket table every sample_interval_secs and cannot see connections shorter than that. Shorten the interval to see more, at the cost of CPU; only eBPF makes it exact.

High CPU or memory

Steady state is a few percent of a core and tens of megabytes. The Linux unit caps the agent at CPUQuota=50% and MemoryMax=300M so a bug cannot take the host with it; if it is anywhere near those, the usual causes are a bandwidth test scheduled too often, discovery port-scanning a large subnet on a short interval, or a very short passive sample interval. Tune from the platform's Configure dialog; none of those needs a restart.

After an upgrade the agent lost its settings

It should not: the Windows wizard pre-fills from the existing config.toml and backs it up to config.toml.bak; install.sh never overwrites an existing file. If settings did change, the previous file is beside the new one on Windows, and on Linux nothing was touched — check that you edited /etc/verops/network-agent.toml and not the example in the extracted tarball.

Removing an agent cleanly

Uninstall on the host (Installation covers both platforms), then Delete Agent under Network → Agents. Deleting in the platform invalidates the key; uninstalling on the host stops the heartbeats. Do both — an agent deleted only in the platform keeps trying to register with a dead key, and one removed only from the host sits in the fleet list as offline forever.