Hardening Linux Containers: Rootless Docker Inside Unprivileged Proxmox LXC

Running containerized application stacks (such as Docker Compose workloads) on Proxmox VE presents systems architects with an interesting dilemma: Should you run Docker inside a full Kernel-based Virtual Machine (KVM/QEMU), or inside an unprivileged Linux Container (LXC)?

Full VMs provide strict hardware-level hypervisor boundary isolation, but they consume substantial baseline RAM, require dedicated virtual disk allocations, and carry OS virtualization overhead. In contrast, LXC containers share the host Linux kernel directly, offering near bare-metal I/O throughput, instantaneous boot times, and microscopic RAM footprints.

However, nesting Docker inside an unprivileged LXC container requires navigating Linux user namespaces (userns), UID/GID remapping offsets, AppArmor profiles, and ZFS storage driver interactions. Misconfigure them, and you either end up with permission errors that render bind mounts unusable or inadvertently run privileged containers that grant container root escapes to the underlying hypervisor.

In this systems engineering guide, we dissect the architecture of running Docker inside unprivileged Proxmox LXC containers safely, efficiently, and with maximum storage performance.

---

1. Privileged vs. Unprivileged Containers: The Security Boundary

The critical distinction in Linux container virtualization lies in whether UID 0 inside the container maps to UID 0 on the hypervisor host.

[ PRIVILEGED LXC (HIGH RISK) ]
Container UID 0 (root)  ───────────────────► Host UID 0 (root)
* Any kernel vulnerability or breakout gives immediate full root access to Proxmox!

[ UNPRIVILEGED LXC (RECOMMENDED / HARDENED) ]
Container UID 0 (root)  ──[ User Namespace ]──► Host UID 100000 (unprivileged daemon)
Container UID 1001 (app) ──[ Shift Offset ]───► Host UID 101001 (unprivileged daemon)
* If an attacker escapes the container, they land on the host as an unprivileged UID 100000 with zero system privileges!

Why Privileged Containers Are a Dangerous Shortcut

Many homelab tutorials suggest: "Just tick the Privileged checkbox in the Proxmox UI to make Docker work easily." Do not do this in production.

In a privileged container, device nodes (/dev), raw sockets, and kernel capabilities are exposed directly. If a Docker container running inside that LXC is compromised via a container escape, the attacker gains direct root access to the entire Proxmox hypervisor, including all other VMs, storage pools, and host networking.

---

2. Configuring the Proxmox LXC Host for Nested Docker

To allow an unprivileged LXC container to spawn nested container namespaces and manage overlay filesystems without compromising host integrity, configure the container options on your Proxmox host (/etc/pve/lxc/<CTID>.conf).

Essential Container Configuration Flags

# /etc/pve/lxc/107.conf
arch: amd64
cores: 2
features: keyctl=1,nesting=1
memory: 2048
swap: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=10.100.0.1,hwaddr=BC:24:11:AA:BB:CC,ip=10.100.0.4/24,type=veth
ostype: debian
rootfs: local-zfs:subvol-107-disk-0,size=16G
unprivileged: 1

Let us understand what these two features do:

1. nesting=1: Allows the container to use mount, create user/pid/network namespaces, and manage cgroups inside its own sub-hierarchy. Without this, the Docker daemon (dockerd) cannot initialize.
2. keyctl=1: Grants the container access to the Linux kernel keyring subsystem. Modern Docker and systemd versions require this to store cryptographic session keys and credentials.

---

3. The Dreaded Storage Driver Battle: ZFS vs. overlay2

If your Proxmox hypervisor runs on ZFS (the gold standard for data integrity and copy-on-write snapshots), nesting Docker introduces an interesting storage challenge.

By default, Docker attempts to use the overlay2 storage driver. However, overlay2 over a nested ZFS POSIX filesystem can encounter issues with whiteout files and extended attributes depending on kernel versions.

The Two Viable Approaches

APPROACH A: Docker Native ZFS Driver
[ Host ZFS Pool ]
       │
       ▼ (ZFS Dataset delegated to LXC)
[ LXC Container: /var/lib/docker using native zfs driver ]
* Pro: Direct copy-on-write snapshots.
* Con: Docker creates dozens of subvolumes directly in your ZFS hierarchy.

APPROACH B: Standard overlay2 inside LXC Rootfs (Recommended)
[ Host ZFS Pool ]
       │
       ▼
[ LXC ext4-compatible subvol mount ]
       │
       ▼
[ Docker daemon with native overlay2 driver ]
* Pro: Clean isolation; container layers remain inside the LXC disk image.
* Con: Requires Proxmox PVE kernel 6.x+ (fully supported out of the box in PVE 8+).

For Proxmox VE 8.x with kernel 6.5+, the standard overlay2 driver inside Debian 12 / Ubuntu 22.04 unprivileged LXC containers works flawlessly without manual ZFS delegation.

Verify inside your container:

docker info | grep "Storage Driver"
# Output: Storage Driver: overlay2

---

4. Solving the UID/GID Remapping & Bind-Mount Trap

This is the single most common frustration engineers face when self-hosting with unprivileged LXC: file permissions on bind mounts.

How User Namespaces Map UIDs

In an unprivileged LXC, Proxmox maps 65,536 container IDs starting at host UID 100000:

| Inside LXC Container | On Proxmox Host Kernel | | :--- | :--- | | UID 0 (root) | UID 100000 | | UID 1000 (admin) | UID 101000 | | UID 1001 (nodejs / nextjs) | UID 101001 | | UID 33 (www-data) | UID 100033 |

The Problem

If you create a directory on the Proxmox host (/mnt/storage/blog-content) owned by host root:root (0:0) and bind-mount it into your LXC:

• Inside the LXC, that directory appears owned by nobody:nogroup (65534:65534).
• Your container user (UID 1001) receives Permission denied on any write attempt.

Solution 1: Explicit Chown for Container UIDs

If the directory lives entirely inside the LXC filesystem (e.g. /opt/blog/content), set ownership to the container user's internal UID:

# Inside the LXC container:
chown -R 1001:1001 /opt/blog/content /opt/blog/public/uploads
chmod -R 775 /opt/blog/content /opt/blog/public/uploads

Solution 2: Passthrough Host Bind Mounts

If you are passing through a host ZFS dataset or external NFS share to the LXC, set the host permissions to match the remapped ID:

# On the Proxmox HOST:
# Remap target: 100000 (LXC root) + 1001 (container app user) = 101001
chown -R 101001:101001 /mnt/pve-storage/blog-content
chmod -R 775 /mnt/pve-storage/blog-content

Inside the LXC container, user 1001 will now have native read and write access without compromising security boundaries!

---

5. Security Hardening Checklist

To ensure your unprivileged Docker LXC setup remains impenetrable:

1. Verify Unprivileged Status:

   # On Proxmox host:
   pct status 107 -verbose | grep unprivileged
   # Must return: unprivileged: 1
   

2. Never Run Containers as Root Internally: Ensure your application Dockerfiles explicitly create and switch to a non-root user:

   RUN addgroup --system --gid 1001 nodejs && \
       adduser --system --uid 1001 nextjs
   USER nextjs
   

3. Limit Resource Quotas: Enforce CPU core limits and memory ceilings in the LXC configuration to prevent runaway workloads from starving other hypervisor services:

   cores: 2
   cpulimit: 2
   memory: 2048
   swap: 512
   

4. AppArmor Profile Confinement: Proxmox assigns the lxc-default-cgns AppArmor profile by default. Keep it enabled. Do not set lxc.apparmor.profile: unconfined unless you are testing kernel-level eBPF or low-level sysctl probes.

---

6. Summary: Benchmark & Efficiency Gains

| Metric | Virtual Machine (QEMU/KVM) | Unprivileged LXC + Docker | | :--- | :--- | :--- | | Idle RAM Footprint | ~650 MB – 1.2 GB (Guest OS overhead) | ~45 MB – 85 MB | | Boot & Startup Time | 25 – 45 seconds | 1.2 seconds | | Disk Overhead | Virtual disk image pre-allocation | Shared ZFS CoW subvolume | | I/O Throughput | VirtIO translation layer | Native Linux kernel direct VFS | | Security Posture | Hardware isolation | User namespaces (userns) + cgroups |

By configuring unprivileged LXC containers with user namespace remapping and proper storage driver alignment, you obtain the agility and convenience of Docker Compose with the performance and efficiency of bare metal.