Daemon Troubleshooting

This page covers a scanner daemon that is offline, failing to start, stuck on an old version, or not reporting to the portal, for both a direct systemd install (install.sh, Installation) and a container run (Docker). Work through Fast triage first to identify which failure mode you have, then jump to the matching section.

Fast triage

Answer three questions before digging further: is it running, what version is it, and can it reach the portal.

Direct install (systemd):

systemctl is-active netvuln-daemon
journalctl -u netvuln-daemon -n 20 --no-pager
cat ~/netvuln-tool/VERSION

is-active prints active, failed, or activating (the last one, seen alongside auto-restart, means systemd is repeatedly retrying a crashing service). The journal lines show the actual exit reason. A healthy startup also logs Raw-socket nmap scan capability at startup: true; see Verifying recovery for the full healthy sequence.

Docker:

docker compose ps
docker logs <container>
docker run --rm <image> --help   # confirms the image runs at all

Docker does not run a persistent daemon container (see the note at the top of Docker below), so "is it running" mostly applies to a scan that is in progress or just exited; docker logs is the equivalent of journalctl here.

Direct install (systemd)

Decision flow

systemctl is-active netvuln-daemon
        |
        |-- active -----------------> check VERSION, portal Fleet page,
        |                             and "Raw-socket ... : true" in the log
        |                             (see Verifying recovery)
        |
        `-- failed / activating (auto-restart)
                |
                v
        journalctl -u netvuln-daemon -n 20 --no-pager
                |
    +-----------+------------------------------+-------------------------------+
    |                                           |                               |
 "Permission denied" running the           "mkdir: cannot create           Starts fine, but VERSION
  script itself (exit 126)                  directory '/root':              is old and C2 `update`
    |                                        Permission denied" (exit 1)     never fixes it
    |                                           |                               |
    v                                           v                               v
 systemctl show netvuln-daemon              Unit still carries              See "Stale version /
   -p User -p CapabilityBoundingSet         Environment=HOME=/root          cannot self-update"
    |                                        from a root install             below
    |                                           |
    +-- User=root, and                          v
    |   stat -c '%A' <home dir>            Add Environment=HOME=<home>
    |   shows drwx------ (0700)            drop-in, daemon-reload, restart
    |     -> root/CAP_NET_RAW/0700-home    (the fix block below also covers
    |        chain, see the diagram        this)
    |        under "exit 126: permission
    |        denied" below
    |
    +-- ls -l scripts/netvuln_daemon.sh
    |   shows no `x` bit
    |     -> lost exec bit: chmod +x
    |        scripts/netvuln_daemon.sh
    |
    `-- findmnt -no OPTIONS -T scripts/netvuln_daemon.sh
        contains `noexec`
          -> noexec mount: ExecStart= drop-in
             that runs the script via `bash <path>`

Symptom -> cause -> fix

Symptom (journal)	Cause	Fix
exit 126, /bin/bash: <path>/netvuln_daemon.sh: Permission denied	Unit installed with User=root; CapabilityBoundingSet=CAP_NET_RAW strips CAP_DAC_READ_SEARCH/CAP_DAC_OVERRIDE, so root cannot traverse a 0700 daemon-user home directory	Drop-in pinning User=/Group=/HOME to the daemon user (below); durable fix is re-running install-service --user <name> as that user. As of 4.9.0 this cannot happen on a fresh install: install-service refuses to render an implicit User=root unit
exit 1, mkdir: cannot create directory '/root': Permission denied	Unit still carries Environment="HOME=/root" left over from a root install; the de-rooted process runs mkdir "$HOME" against /root	Add Environment=HOME=<daemon-user-home> to the unit (the fix block below sets this too)
exit 126, journal shows no permission-denied text but the process never starts	Script lost its executable bit (a fresh checkout, an archive extraction, core.filemode off)	chmod +x scripts/netvuln_daemon.sh (only the executed script; do not broad-chmod, see the note below)
exit 126 on a freshly mounted volume or a hardened host	The filesystem holding the script is mounted noexec	findmnt -no OPTIONS -T scripts/netvuln_daemon.sh to confirm, then a drop-in resetting ExecStart= to run the script through bash explicitly
Daemon runs, but VERSION never advances and C2 update reports completed with no effect	Pre-4.8.6 code lacks the self-exec re-exec (#205), or hits the bad tag-ref bug (#266)	One-time host escape: git fetch/checkout/pull by hand, then restart
Portal C2 output shows Session: /home/nvt/netvuln-tool/sessions/... on a migrated host	Scanner library ignored NVD_BASE_DIR, unlike the daemon itself (#307)	Update to 4.9.2; see Sessions still land under /home/nvt after the /opt migration

exit 126: permission denied (the root / CAP_NET_RAW / 0700-home chain)

This is the most common real-world cause. It looks at first like a simple lost-permission problem, but chmod does not fix it. Rule out the usual exit-126 suspects first:

• Lost exec bit? ls -l scripts/netvuln_daemon.sh still shows the x bits (for example -rwxrwxr-x nvt nvt).
• noexec mount? findmnt -no OPTIONS -T scripts/netvuln_daemon.sh shows rw,noatime, no noexec.
• SELinux / AppArmor / fapolicyd? No MAC policy active, no kernel denial in journalctl -k or dmesg.

None of those apply. The actual chain:

sudo netvuln_daemon_ctl.sh install-service
        |
        v
  sudo escalates the WHOLE script, not just the
  tee/systemctl steps inside it
        |
        v
  unit rendered from whoami/id -gn/$HOME of that
  root shell: User=root, Group=root,
  Environment="HOME=/root"
        |
        v
  unit also carries (by design, the rootless
  raw-socket scanning in templates/netvuln_daemon.service):
    AmbientCapabilities=CAP_NET_RAW
    CapabilityBoundingSet=CAP_NET_RAW
        |
        v
  CapabilityBoundingSet=CAP_NET_RAW strips every
  other capability from the root process, including
  CAP_DAC_READ_SEARCH / CAP_DAC_OVERRIDE
        |
        v
  a fleet hardening pass tightened home directories
  to 0700 (drwx------ nvt nvt)
        |
        v
  root can no longer traverse /home/nvt to reach
  scripts/netvuln_daemon.sh
        |
        v
  exec fails with EACCES; systemd reports exit 126:
  "/bin/bash: /home/nvt/netvuln-tool/scripts/netvuln_daemon.sh:
   Permission denied"
        |
        v
  chmod +x changes nothing here: the script file
  already has its exec bit. The failure is on
  DIRECTORY traversal into /home/nvt, not the file's
  own mode.

Diagnose:

systemctl show netvuln-daemon -p User -p CapabilityBoundingSet
stat -c '%A' /home/nvt

The first command shows User=root; the second shows drwx------. Together they confirm the chain above rather than a lost exec bit or a mount issue.

Reproduce (optional, confirms the diagnosis before you touch the unit):

# Fails with exit 126, mirroring the real unit: root, bounded to CAP_NET_RAW only
sudo systemd-run -p CapabilityBoundingSet=CAP_NET_RAW --pipe --wait \
  /bin/bash -n scripts/netvuln_daemon.sh

# Succeeds (rc 0): same script, run as the daemon user instead of root
sudo systemd-run --uid=nvt --gid=nvt --pipe --wait \
  /bin/bash -n scripts/netvuln_daemon.sh

Fix (run as the daemon user, not root):

sudo chown -R "$(whoami):$(id -gn)" ~/netvuln-tool
sudo mkdir -p /etc/systemd/system/netvuln-daemon.service.d
printf '[Service]\nUser=%s\nGroup=%s\nEnvironment=HOME=%s\n' "$(whoami)" "$(id -gn)" "$HOME" \
  | sudo tee /etc/systemd/system/netvuln-daemon.service.d/run-as-nvt.conf
sudo systemctl daemon-reload && sudo systemctl restart netvuln-daemon

Run this logged in (or su'd) as the daemon user itself, so $(whoami), $(id -gn), and $HOME resolve to the account the daemon should run as (nvt / /home/nvt in the incident), not root. The drop-in sets User, Group, and HOME together, which also covers the mkdir '/root' failure below if you have not hit it separately.

This is a workaround; see Root cause and prevention to fix it durably and remove the drop-in.

exit 1: mkdir: cannot create directory '/root': Permission denied

Once User/Group are pinned to the daemon user, the unit can still carry a stale Environment="HOME=/root" from the original root install. The de-rooted process starts up and calls mkdir "$HOME" to ensure its home directory exists, now mkdir /root, which the non-root process cannot do.

Fix: add Environment=HOME=<daemon-user-home> to the unit (a drop-in, or edit the installed unit directly), then daemon-reload and restart. The combined fix block in the previous section already sets this.

exit 126: lost exec bit

A plain, unrelated cause worth ruling out first, and the one chmod actually fixes:

ls -l scripts/netvuln_daemon.sh   # look for the x bits
chmod +x scripts/netvuln_daemon.sh

Note: as of 4.9.1, every tracked shell script is committed executable (100755) and CI guards it (scripts/check_exec_bits.sh), so a lost exec bit on any script is now a local-only condition, not something a checkout or update can reintroduce. The recovery above still stands: chmod +x scripts/netvuln_daemon.sh. If an old checkout still carries scripts marked modified from a broad chmod +x under core.filemode true, clear it with git config core.filemode false (or git checkout -- scripts/ lib/), then re-dispatch the update.

exit 126: noexec mount

If the checkout lives on a filesystem mounted noexec (common on some hardened images or certain network mounts), the kernel refuses to execute the script directly regardless of its permission bits.

findmnt -no OPTIONS -T scripts/netvuln_daemon.sh

If the output contains noexec, the durable options are moving the checkout to an executable filesystem, or making systemd invoke the script through the interpreter instead of executing it directly:

sudo mkdir -p /etc/systemd/system/netvuln-daemon.service.d
printf '[Service]\nExecStart=\nExecStart=/bin/bash %s/scripts/netvuln_daemon.sh\n' \
  "$(pwd)" | sudo tee /etc/systemd/system/netvuln-daemon.service.d/noexec-workaround.conf
sudo systemctl daemon-reload && sudo systemctl restart netvuln-daemon

The empty ExecStart= line clears the templated ExecStart= before the override takes effect; systemd requires that to replace rather than append.

Stale version / cannot self-update

A daemon on code older than 4.8.6 cannot self-update through C2: builds before 4.8.5 lack the self-exec re-exec that makes an update command take effect without a manual restart (#205), and versions before 4.8.6 hit a separate bug where the C2 update command built a bad vv<version> tag reference (#266). A daemon stuck on one of these versions needs a one-time host escape, done once, by hand, as the daemon user:

cd ~/netvuln-tool
git fetch origin --tags --prune
git checkout main
git pull --ff-only origin main || git reset --hard origin/main
chmod +x scripts/netvuln_daemon.sh
sudo systemctl restart netvuln-daemon

Once the daemon is on 4.8.6 or later, C2 self-update works normally again; dispatch it with a bare version string, for example 4.8.7, never v4.8.7 (see the update command in Command and Control).

Sessions still land under /home/nvt after the /opt migration

A host migrated to the /opt layout with scripts/migrate_to_opt.sh (see Root cause and prevention below), but portal C2 output for a scan still shows a session under the old home directory:

[INFO] Session: /home/nvt/netvuln-tool/sessions/20260902_102805

Cause: the 4.9.x unit renders Environment="NVD_BASE_DIR=/opt/netvuln-tool", and the daemon itself honored that variable for its own paths, but lib/netvuln_common.sh hardcoded NV_BASE_DIR="$HOME/netvuln-tool" and scripts/log_syslog.sh separately hardcoded its own LOG_DIR. Every scan the daemon spawned, the C2 session-directory fallbacks in lib/command_handler.sh, and every scan's own log lines inherited one of those hardcoded paths instead of NVD_BASE_DIR, so scans kept writing sessions and logs under /home/nvt/netvuln-tool/ even after the migration. Fixed in 4.9.2 (#307).

Fix: update to 4.9.2, then move the sessions and logs written in the interim, as the nvt user:

sudo -u nvt bash -c '
mkdir -p /opt/netvuln-tool/sessions /opt/netvuln-tool/logs
mv /home/nvt/netvuln-tool/sessions/* /opt/netvuln-tool/sessions/ 2>/dev/null
mv /home/nvt/netvuln-tool/logs/* /opt/netvuln-tool/logs/ 2>/dev/null
rmdir /home/nvt/netvuln-tool/sessions /home/nvt/netvuln-tool/logs /home/nvt/netvuln-tool 2>/dev/null
true
'

Root cause and prevention

All of the above traces back to how the service was installed. Before 4.9.0, cmd_install_service in scripts/netvuln_daemon_ctl.sh rendered the unit from the shell that runs it: whoami, id -gn, and $HOME of that process became User=, Group=, and Environment=HOME= in the unit. Running install-service under sudo did not just escalate the tee/systemctl steps that actually need root, it ran the entire script as root, baking User=root into the unit from the start.

As of 4.9.0, install-service takes explicit --user/--group/--home/ --install-dir/--base-dir flags, install.sh always passes all five, and install-service refuses to render an implicit User=root unit when none of those flags are given and the resolved user is root. This closes #297's tracking of issue #281: the root-install bug can no longer recur from a supported install path.

If a host is already broken by a pre-4.9.0 root install, the drop-in in the fix block above gets it running again immediately. The durable fix is re-running install-service with explicit flags for the intended user and removing the drop-in. A host that was hand-patched onto a dedicated nvt account with the incident-era run-as-nvt.conf drop-in (the exact shape this bug produced) can instead run scripts/migrate_to_opt.sh (or install.sh --migrate), which moves it onto the supported /opt/netvuln-tool + nvt layout in one step and removes the drop-in for you. See Raspberry Pi for the layout this produces.

Docker

There is no supported long-running daemon container in this repo. netvuln daemon <action> exists as a CLI passthrough (it forwards to netvuln_daemon_ctl.sh), but install-service requires systemd, which the Alpine image does not have, and there is no compose service, restart policy, or volume wired up for running the daemon unattended in a container. The supported container path is a one-shot scan:

docker compose run --rm scan recon -t 192.168.1.0/24

The failure modes below apply to that one-shot path, and they cover the same ground as the systemd section above (LAN reach, raw-socket capability, portal reporting) because those are the parts an operator moving a scheduled scan into a container is most likely to trip over.

Symptom -> cause -> fix

Symptom	Cause	Fix
Scan finds no hosts on the LAN, or only the container's own subnet	Default bridge network NATs the container away from the LAN; on Docker Desktop (macOS/Windows) "host" networking reaches the Linux VM, not your laptop's LAN	Use --network host (network_mode: host in the compose file, already the default there); run from a Linux host for real LAN reach
OS detection / SNMP scans silently skipped, session shows an "OS Detection Skipped" finding, heartbeat reports raw_scan_capable: false	NET_RAW was dropped from the container (--cap-drop=NET_RAW), the image was rebuilt without the baked setcap cap_net_raw+eip on nmap, or NMAP_PRIVILEGED is unset	Do not drop NET_RAW (Docker grants it by default); rebuild from the shipped Dockerfile rather than a stripped-down variant; --user root is the blunt fallback if capabilities are unavailable
Container exits immediately after --cap-drop=NET_RAW with Operation not permitted	The nmap file capability's effective bit needs NET_RAW present in the bounding set; removing it makes the kernel refuse to exec nmap at all	Stop dropping NET_RAW; if you must run with all capabilities dropped, use a host install instead
No reports or session data appear on the host after a scan	No volume mounted, or the mounted ./out directory is not writable by the container's netvuln user	Mount -v $PWD/out:/opt/netvuln-tool/sessions (the compose scan service already does this); check host directory permissions
Scan runs an older engine than expected, or is missing a recent fix	IMAGE_TAG pinned to an old tag, or a stale local build cached from an earlier docker build	docker compose pull (or drop IMAGE_TAG to use :latest), or rebuild locally with docker build -t netvuln-tool:dev .
upload/push fails, or the license soft gate reports no key	NV_API_KEY (or the mounted config) was not passed into the container, or the portal host is unreachable from the container's network	Pass -e NV_API_KEY=... or mount report_config.conf; confirm the container can reach the portal host (network mode, DNS)

Contrast with the systemd path

The User=root/CapabilityBoundingSet=CAP_NET_RAW/0700-home chain that causes exit 126 under systemd does not occur in Docker. There is no systemd unit to install, so there is no root-vs-daemon-user install mistake to make: the image always runs as the fixed non-root netvuln user under /opt/netvuln-tool, and that user's home directory permissions are set by the image build, not by a fleet hardening pass against a real host account. If a container scan fails with a permission error, it is a volume-mount ownership issue (see the reports-not-landing row above), not the systemd capability chain.

Verifying recovery

After applying a fix, confirm all of the following, not just that the process is running:

systemctl is-active netvuln-daemon    # expect: active
journalctl -u netvuln-daemon -n 20 --no-pager

The journal should show the full healthy startup sequence, in this order:

[INFO] Loading configs from: /home/nvt/netvuln-tool/configs
[INFO] Loaded: archer-energy.conf ... next_run=...
[INFO] Loaded 1 schedule(s)
[INFO] Daemon started (PID ..., 1 schedule(s), poll=60s, heartbeat=300s)
[INFO] Raw-socket nmap scan capability at startup: true
[INFO] Status dumped to /home/nvt/netvuln-tool/logs/daemon_status.json

Then check beyond the local host:

• Raw-socket ... : true, not false. false means OS detection (-O) and SNMP/UDP (-sU) scans will be silently skipped even though the daemon itself is running fine; see Privilege Model.
• Portal Fleet page shows the agent online, at the expected version, under the same agent_id as before the incident. The agent id is derived from the host's machine-id, not from the OS user the daemon runs as, so a user/group change should never produce a new fleet entry; a new entry appearing instead points at a different underlying problem (for example a machine-id change) worth investigating separately.
• Schedule "Last run" advances on a scheduled host. Either netvuln_daemon_ctl.sh status or the portal's schedule detail should show a last_run timestamp after the next scheduled time passes, not never or a stale date from before the incident.

See also

• Daemon Mode: full daemon architecture, signals, heartbeat payload, and the privilege model this page's exit-126 chain builds on.
• Command and Control: the update command, agent lifecycle, and remote fleet management referenced in "Stale version / cannot self-update" above.
• Installation: install.sh, the systemd daemon install step, and the setcap/apt-hook mechanics behind CAP_NET_RAW.
• Docker: building and running the container image, and the raw-socket capability model it shares with the systemd unit.
• Issue #281: the install-service root-install bug this page's systemd section documents, closed in 4.9.0 by explicit --user/--group/--home/ --install-dir/--base-dir flags and a refusal on an implicit root render. This page's workaround and scripts/migrate_to_opt.sh remain the path for hosts installed before the fix.