2022-03-12 / Bartłomiej Kurek
LXC unprivileged containers

Overview

For development purposes I'm creating a docker swarm/kubernetes infrastructure. It will consist of two nodes running as full operating system LXC containers (running init). Once set up I'll install docker and kubernetes in these nodes. The LXC containers will run Debian images (bookworm release).

There are plenty of tutorials on how to get LXC configured and running. However, I'm following the official Debian guide on LXC as it covers details specific to Debian and its kernel version/configuration.

The containers will run in unprivileged mode. This means that the processes in containers that are run as root user are mapped to a regular user on the main host). This is achieved by subordinate user id (man 5 subuid).
The containers' network intefaces will be "Virtual Ethernet Devices" (man 5 veth). LXC will use a individual lxcbr0 interface and all container interfaces will be added to this bridge. I opt for static IP addressing, so this will be configured in dnsmasq (using regular hardware addresses (MAC)).

LXC and configuration check

Installation:

# apt install lxc libvirt0 libpam-cgfs bridge-utils uidmap

Confiugration check.

# lxc-checkconfig

Depending on the system configuration, the above command may report missing cgroups.
In my case I had to update grub configuration and reboot.

File /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0"

Then update grub and reboot:

# update-grub
# reboot

Once the system boots, check lxc-checkconfig again.

LXC networking

I'm configuring lxc to use the separate bridge interface. This is controlled via USE_LXC_BRIDGE in /etc/default/lxc-net file.

USE_LXC_BRIDGE="true"

The bridge comes up after restarting lxc-net.

# /etc/init.d/lxc-net restart
Restarting lxc-net (via systemctl): lxc-net.service.

The bridge interface should now be present.

# ifconfig lxcbr0
lxcbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.0.3.1  netmask 255.255.255.0  broadcast 10.0.3.255
        ether 00:16:3e:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Configuring defaults

The system wide LXC configuration resides in /etc/lxc/default.conf;
In my case it is as follows:

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up

lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

However, unprivileged containers are started as a regular user which which will have a custom (but similar) lxc configuration. In case of unprivileged containers the apparmor profile will have to be changed.

Unprivileged containers

Adding a dedicated user

I'm adding a new user (lxcuser). At the moment I'm not interested in logging in as this user, so I'm setting the shell to /usr/bin/nologin. This users account will only be available to root via su with a shell argument.

# useradd -s /usr/sbin/nologin -d /home/lxcuser --create-home lxcuser

Upon the Creation of a new user account, new uid and gid range are added in the system. Check:

# grep lxcuser /etc/subuid
lxcuser:231072:65536
# grep lxcuser /etc/subgid
lxcuser:231072:65536

The specific numbers depend on other user accounts. When there are no other user accounts, the first number will usually be 100000. I have other accounts in the system, so a different id map was allocated.

Allow the regular user to create virtual interfaces

File /etc/lxc/lxc-usernet:

# cat /etc/lxc/lxc-usernet
lxcuser veth lxcbr0 10

This allows lxcuser to create veth interfaces that will be added to lxcbr0 (at most 10 of them).

Unprivileged userns clone (sysctl)

If unprivileged user namespaces are not enabled - update sysctl.conf.

File /etc/sysctl.conf:

kernel.unprivileged_userns_clone=1

LXC configuration (regular user)

Once the dedicated account has been created, switch to it using su.

# su - lxcuser -s /bin/bash

Custom user configuration for LXC resized in ~/.config/lxc/default.conf. It is very similar to the system-wide defaults, but aooarmor profile is changed and idmaps are added.

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up

lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1

lxc.idmap = u 0 231072 65535
lxc.idmap = g 0 231072 65535

For uid/gid maps use the ids from /etc/subuid and /etc/subgid, respectively.

Creating containers

Switch to the dedicated user account:

# su - lxcuser -s /bin/bash

Create a container. If lxc-create command fails with "Unable to fetch gpg key from keyserver" message, it means the GPG keyserver has to be configured.
This can be set as environment variable (DOWNLOAD_KEYSERVER) or the server address may be passed as an option.

$ lxc-create --template download \
    --name node-a \
    -- \
    --dist debian --release bookworm -a amd64 \
    --keyserver hkps://keyserver.ubuntu.com:443

Start the container:

$ lxc-start node-a

Attach:

$ lxc-attach node-a
root@node-a:/# 
root@node-a:/# grep NAME /etc/os-release 
PRETTY_NAME="Debian GNU/Linux bookworm/sid"
NAME="Debian GNU/Linux"

Checking a container

These should be a full operating system container, so attach to it and confirm that the init is running.

root@node-a:/# ps 1
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:00 /sbin/init

Processes run by root in a container should be mapped to lxcuser on the main host. Execute a command in a container and check ps output on the main host.

Container:

root@node-a:/# sleep infinity

Main host:

# ps waux | grep infinity
231072     19926  0.0  0.0   5416   676 pts/4    S+   15:03   0:00 sleep infinity

Indeed, although the process in a container runs as root, the process on the main host belongs to a regular user.

Container networking: DHCP, Firewall, NAT

If IPv6 support is not needed in a container, it can be turned off:

(root@container) # echo 'net.ipv6.conf.all.disable_ipv6=1' >> /etc/sysctl.conf 
(root@container) # echo 'net.ipv6.conf.default.disable_ipv6=1' >> /etc/sysctl.conf 

DHCP

By default, container IP addresses are assigned by DHCP.
DHCP is served by dnsmasq on the main host. Static container IPs can be configured as well.
I choose to set MAC addresses for individual containers and add corresponding entries in dnsmasq.

Example - setting MAC address for node-a container:

File ~/.local/share/lxc/node-a/config:

lxc.net.0.hwaddr = 00:00:00:00:00:0a

Entries in dnsmasq (file /etc/dnsmasq.conf, main host):

domain-needed
bogus-priv
except-interface=wlan0
expand-hosts
dhcp-range=lxc,10.0.3.100,10.0.3.200,12h
dhcp-option=lxc,option:router,10.0.3.1
dhcp-host=00:00:00:00:00:0a,node-a,10.0.3.100
dhcp-host=00:00:00:00:00:0b,node-b,10.0.3.101
log-queries
log-dhcp
conf-dir=/etc/dnsmasq.d/,*.conf

Restart the container:

$ lxc-stop node-a
$ lxc-start node-a

Check:

$ lxc-ls -f
NAME   STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED 
node-a RUNNING 0         -      10.0.3.100 -    true

NAT

A permissive iptables setup (main host):

iptables -t filter -A INPUT -i lxcbr0 -j ACCEPT
iptables -t filter -A FORWARD -i lxcbr0 -j ACCEPT
iptables -t filter -A FORWARD -o lxcbr0 -j ACCEPT
iptables -t filter -A OUTPUT -o lxcbr0 -j ACCEPT

iptables -t nat -A POSTROUTING -s 10.0.3.0/16 ! -d 10.0.3.0/16 -j MASQUERADE

Create another container

$ lxc-create --template download --name node-b \
    -- \
    --dist debian --release bookworm -a amd64 \
    --keyserver hkps://keyserver.ubuntu.com:443

Set hardware address for the container's ethernet device (in ~/.local/share/lxc/node-b/config):

[...]
# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.hwaddr = 00:00:00:00:00:0b
lxc.net.0.flags = up
[...]

Start and inspect the container.

$ lxc-start node-b
$ lxc-ls -f
NAME   STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED 
node-a RUNNING 0         -      10.0.3.100 -    true
node-b RUNNING 0         -      10.0.3.101 -    true 

If the MAC address was different than the one configured in dnsmasq, the container would get assigned a different IP.

DNS

DNS for the containers is handled by dnsmasq, so the nodes should be able to resolve the hostnames.

Ping test: node-a -> node-b.

root@node-a:/# ping node-b -c 1 
PING node-b (10.0.3.101) 56(84) bytes of data.
64 bytes from node-b (10.0.3.101): icmp_seq=1 ttl=64 time=0.128 ms

--- node-b ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.128/0.128/0.128/0.000 ms