The Ghost Device: Why Your USB Dongle Dies Inside a Rootless Podman Container

Huub Daems

You pass --device /dev/ttyACM0 to your rootless Podman container. You check the device inside: crw-rw-rw-. It works. You walk away.

Ten seconds later, your Z-Wave stick is dead. The device node shows c---------. Zero permissions. Your home automation can’t reach it.

You didn’t change anything. The container is still running. The device permissions just vanished.

This is the story of how I spent three days debugging a problem that doesn’t exist in any Podman documentation — because it’s caused by a component most people don’t know is there.

The Setup

I was deploying a smart home server on Rocky Linux using rootless Podman containers managed by systemd quadlets. Everything was containerized: Home Assistant, Mosquitto, Zigbee2MQTT, and Z-Wave JS UI. The Z-Wave dongle (an Aeotec Z-Stick Gen5) appeared as /dev/ttyACM0 on the host.

The quadlet configuration looked straightforward:

[Container]
Image=docker.io/zwavejs/zwave-js-ui:latest
ContainerName=zwavejsui
AddDevice=/dev/ttyACM0
SecurityLabelDisable=true
GroupAdd=keep-groups
Network=host

The UDEV rule set MODE="0666" on the device. Everything should have worked.

It didn’t.

Symptoms

Inside the container, /dev/ttyACM0 existed but had permissions c---------. The Z-Wave JS UI application couldn’t open it. The logs said “permission denied.”

But when I ran the exact same command manually with podman run -d ... --device /dev/ttyACM0 ..., it worked! The device showed crw-rw-rw-.

Or so I thought.

What I Tried (And What Failed)

I went through the full checklist of reasonable approaches:

ApproachResult
AddDevice=/dev/ttyACM0 in quadletc---------
--device /dev/ttyACM0:/dev/ttyACM0:rwmc---------
GroupAdd=keep-groupsc---------
--privilegedc--------- (rootless has no real privileges)
--cap-add SYS_ADMINc---------
Systemd DeviceAllow= overridec---------
Systemd Delegate=no overridec---------
--cgroups=enabled instead of splitc---------
chcon -t container_file_t /dev/ttyACM0 (UDEV)Broke things further
Plain systemd service instead of quadletStill c---------

Every single approach failed. Some were obviously wrong in retrospect (chcon applies the wrong SELinux type for device nodes). Others seemed like they should work but didn’t.

The Key Observation

The breakthrough came when I checked device permissions at different time intervals after starting the container manually:

$ podman run -d --name test --device /dev/ttyACM0 ...
$ podman exec test ls -la /dev/ttyACM0
crw-rw-rw-    1 nobody   nobody    166,   0 Apr 29 16:33 /dev/ttyACM0

# ... 8 seconds later ...

$ podman exec test ls -la /dev/ttyACM0
c---------    1 nobody   nobody    166,   0 Apr 29 16:33 /dev/ttyACM0

The device started accessible and then became inaccessible. Something was changing the permissions after container creation.

The Root Cause: eBPF Device Filtering in cgroup v2

The culprit is crun, the default OCI runtime for Podman on RHEL and Rocky Linux.

When a container starts on a cgroup v2 system, crun applies an eBPF device program to the container’s cgroup. This program implements a default-deny policy: only devices explicitly listed via --device are allowed. All other devices get their permissions reset to 0000.

The program is applied asynchronously — it takes 5 to 10 seconds to take effect. This is why manual testing appeared to work: I was checking permissions before the eBPF program was applied. Under systemd, the service initialization takes long enough that the eBPF program is always in place by the time you check.

The --device /dev/ttyACM0 flag does add the device to the allow list in the eBPF program. But the cgroup device namespace is separate from the filesystem namespace. The device node exists in /dev (filesystem), but the BPF program controls access at the cgroup level — and it resets all permissions not in the explicit allow list. The interaction between these two namespaces is the source of the bug.

This behavior is not documented in Podman’s documentation. It’s a crun implementation detail that only manifests in rootless mode under systemd.

The Fix: Bind-Mount /dev Read-Only

The solution is to bypass the cgroup device namespace entirely by bind-mounting the host’s /dev directory into the container:

# In quadlet .container file:
Volume=/dev:/dev:ro
AddDevice=/dev/ttyACM0
# Or in podman run:
-v /dev:/dev:ro --device /dev/ttyACM0

When you bind-mount /dev from the host, the device nodes are accessed through the host’s filesystem namespace, not the container’s cgroup-restricted device namespace. The :ro flag prevents the container from modifying /dev.

The AddDevice=/dev/ttyACM0 is still useful — it tells Podman the device should exist, which helps with container initialization. But the real access comes from the bind mount.

After applying this fix, /dev/ttyACM0 shows crw-rw-rw- and stays that way indefinitely.

Why Doesn’t Group-Based Access Work?

You might wonder why UDEV rules with MODE="0660", GROUP="dialout" don’t work. The answer is user namespace mapping.

In rootless Podman, the container’s user namespace maps:

  • Host user (UID 1000) → Container root (UID 0)
  • Host subuid range (100000+) → Container UIDs 1+
  • Unmapped UIDs → nobody (65534)

The container’s dialout group is not the same as the host’s dialout group. Device nodes owned by host groups appear as nobody inside the container. Setting MODE="0666" (world-readable/writable) is the only reliable approach because it doesn’t depend on group membership.

UDEV Rules for Rootless Podman

For USB dongles, use these rules:

# Z-Wave dongle (Aeotec Z-Stick Gen5) — appears as /dev/ttyACM*
SUBSYSTEM=="tty", ATTRS{idVendor}=="0658", ATTRS{idProduct}=="0200", MODE="0666", SYMLINK+="zwave-dongle"

# Zigbee dongle (Sonoff) — appears as /dev/ttyUSB*
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea60", MODE="0666", SYMLINK+="zigbee-dongle"

Key points:

  • MODE="0666" — not group-based permissions. Groups don’t map across user namespaces.
  • Don’t use chcon on device nodes. container_file_t is for files, not character devices. Use SecurityLabelDisable=true in the quadlet to handle SELinux.
  • Know your driver: Z-Wave dongles use the cdc_acm driver and appear as /dev/ttyACM*, not /dev/ttyUSB*.

Quadlet vs Plain Service: Did It Matter?

During debugging, I switched from a quadlet to a plain systemd service, hoping that removing the quadlet generator’s hidden flags (--cgroups=split, conmon, Delegate=yes) would fix the device access.

It didn’t.

The eBPF device filter is applied by crun regardless of how the container is launched. Quadlet, plain service, or manual podman run — same result after 10 seconds. The “improvement” I saw with the plain service was actually because I was testing the fix (Volume=/dev:/dev:ro) at the same time as switching service types.

Don’t conflate “I changed X and it worked” with “X was the problem.” When you make two changes at once, you can’t tell which one mattered.

The right debugging pattern is:

  1. Fix the actual problem first
  2. Then decide whether to simplify the service type
  3. Never do both at once

For the record: quadlets are the better default. They give you declarative configuration, automatic restarts, and podman-auto-update support. Plain services should only be used when you need full control over ExecStart.

Checklist: Rootless Podman + USB Device

Before deploying a container that needs USB/serial device access:

  • [ ] UDEV rule sets MODE="0666" (not group-based permissions)
  • [ ] UDEV rule does not use chcon -t container_file_t
  • [ ] Quadlet/container includes SecurityLabelDisable=true
  • [ ] Quadlet/container includes GroupAdd=keep-groups
  • [ ] Quadlet/container includes Volume=/dev:/dev:ro
  • [ ] Quadlet/container includes AddDevice=/dev/ttyACM0 (or your device)
  • [ ] Verify device path: Z-Wave = /dev/ttyACM*, Zigbee = /dev/ttyUSB*
  • [ ] Test device access 10+ seconds after container start, not immediately
  • [ ] loginctl enable-linger is set for the container user

References