LXC & LXD

tags : Containers

LXC §

• LXC Images : Linux Containers - Image server
• OS level virtualization for running multiple isolated containers on single Linux kernel.
• Introduced 2008
• Lowlevel
• Usually managed via manager like LXD
• LXC is because it doesn’t support OCI.
• Since it uses the same kernel, dmesg will be the same as the host. If you have unprivileged containers, you won’t be able to run dmesg

LXD §

Intro §

• LXD is built on top of LXC and aims to provide a better user experience. Basically a management system on top of LXC.
• It implements a single REST API for both local and remote access.
• Scales from one instance on a single machine to a cluster in a full data center rack

LXD consists of two parts:

• Daemon (the lxd binary)
• Client (the lxc binary)

Confusion w LXC §

lxc is not LXC; the naming is a bit confusing.

• lxc is the lxd client
• All lxc-* commands are low level LXC commands

Setting up LXD §

• Daemon
  • Anyone belonging to the lxd group will have access to the daemon.
  • LXD daemon can be accessed locally over a Unix socket or, if configured, remotely over a TLS socket.
• User
  • LXD loads the subuid/subgid values from /etc on startup.
• Config & Profile
  • Everytime you lxd init, config gets overwritten, you can use --preseed to provide config via file.
  • For the bare minimum, you need to set a storage pool and network bridge.
  • Once the network bridge is setup, it’ll show up in ip addr
  • See Instance options - LXD documentation
• Net NS
  • Once containers are started namespaces can be listed using

      ip netns list-id

Instances §

Containers §

Implemented through the use of liblxc (LXC).

• System container
  • Usually one which runs systemd in it as the init process (where using systemd)
  • Simulates a virtual version of a full operating system.
  • Uses kernel running on the host system.
  • Suitable for full solution of libraries, applications, databases etc.
  • Can create different user spaces and isolate all processes belonging to each user space
• Application container
  • These are more in the likes of what Docker provides
  • Usually packages a single process or application.
  • Suitable to provide separate components

VMs §

• Uses qemu to provide the VM functionality.
• Uses the hardware of the host system
• Uses kernel provided by the virtual machine.
• Can be used to run a different operating system.

Configuration §

• Instance properties
• Instance options
• Instance devices

Privileged and Unprivileged containers §

How does LXD do unprivileged §

• LXD by default is unprivileged
• Default map: 65536 UIDs and GIDs, w host base id of 100000.
• container root (uid0) : mapped to host uid 100000
• container (uid65535) : mapped to host uid 165535

subuid and subgid §

See gids) for details. LXD allows updating of these via security.idmap.isolated, security.idmap.size , security.idmap.base and raw.idmap. Read Idmaps for user namespace for more details.

• Change the size of the default map
• Per container maps

  lxc config set <instance> security.idmap.isolated true # no sharing btwn containers
  lxc config set <instance> security.idmap.size 200000 # change alloc size
  # if allocation > allocated to lxd itself, will throw error

• Access host but not other containers. In unprivileged containers, usually user will be mapped to a user that doesn’t exist on host = no storage sharing w host

Personal config and setup §

Install LXC §

This file is automatically created and populated once LXC is installed.

λ cat /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx

Install LXD §

• The only entries that matter to LXD are the ones for the root user.
• LXD itself is restricted by the root allocation. One may sometimes see lxd user in these files, they are there just to ease tracking by the sysadmin.
• A user can have multiple allocations.

λ sudo usermod -v 1000000-1000999999 -w 1000000-1000999999 root
λ cat /etc/sub{u,g}id
root:1000000:1000000000
root:1000000:1000000000

Make LXD accessible §

• See About security - LXD documentation
• lxc (the LXC client) needs to communicate to the lxd daemon via unix socket
• Access control for LXD daemon is possible only locally via group membership. If anything else, you need to expose LXD over the network.

usermod -a -G lxd geekodour

Sharing folder §

There are couple of cases here. They use the shift method. For readonly none of this is needed.

$ lxc info | grep -E 'idmapped|shiftfs'
idmapped_mounts: "true" # kernel feature
shiftfs: "false"
idmapped_mounts_v2: "true" # lxc feature

If this is enabled, you’re good to go.

• Mounting host directory to instance

  IMPORTANT NOTE

  • Adding a idmap to an actual user allows the container to run actual processes as that user
  • Which means it can use ulimit to apply system-wide restrictions to the uid
  • Potentially impacting processes in other containers or even on the host.

  1. Add disk

     $ lxc config device add <instance_name> <new_device_name> disk source=/home/geekodour/some_dir path=/home/<someuser>/dir_inside_container

  2. Now this will add the disk to the container, but the owner:group will be nobody:nobody and you won’t have edit permissions. These will be readonly, you’ll be able to edit them from your host as usual. So if that’s what you want, that’s all.
  3. If you want container user to have proper access over the files, here’s what you do.

     ########
     # Before changes
     ########
     $ lxc exec <instance> -- cat /proc/self/uid_map
     0    1000000 1000000000
     ########
     # Make changes
     ########
     $ id
     uid=1000(geekodour) gid=1000(geekodour) # truncated
     # - Allow LXD to map uid of host user (eg. 1000) inside the containers.
     # - Add root:1000:1 to subuid and subgid files.
     $ sudo usermod --add-subuids 1000-1000 --add-subgid 1000-1000 root
     λ cat /etc/sub{g,u}id
     root:1000000:1000000000
     root:1000:1
     root:1000000:1000000000
     root:1000:1
     # Update raw.idmap to use host uid 1000
     # lxc config set test raw.idmap='both <host uid/uid-range> <container uid/uid-range>
     # See https://linuxcontainers.org/lxd/docs/latest/userns-idmap/#custom-idmaps
     $ lxc config set <instance> raw.idmap='both 1000 1000' # inside container we want to map to 1000 aswell
     $ lxc restart  <instance> # important
     ########
     # After change
     ########
     $ lxc exec <instance> -- cat /proc/self/uid_map
        0    1000000       1000
     1000       1000          1
     1001    1001001  999998999

  4. To check the mount point: lxc exec <instance> -- cat /proc/self/mountinfo

• Access files

  There are other methods to access files that I’ve to explore.

Python tooling (remove later) §

• Install asdf
• Installing python needs these

    sudo apt-get install build-essential zlib1g-dev libffi-dev libssl-dev libbz2-dev libreadline-dev libsqlite3-dev liblzma-dev

• Install python
• Install poetry
• use poetry to create the env
• You can either use pip or use poetry add now
• direnv not needed yet

LXD Profiles §

• The configuration hierarchy is as follows: on the instance > certain ordered profile > default profile
  • Instance options: array under config.
  • Instance devices: under devices.
• Devices from profiles are applied to the instance in the order in which the profiles are assigned to the instance.
• Devices defined directly in the instance configuration are applied last.
• At each stage, if a device with the same name already exists from an earlier stage, the whole device entry is overridden by the latest definition.

Storage §

Jargon §

• Storage pool = disk to store data (LXD stores its data in storage pools)
• Storage volume = Different partitions used for specific purposes
• Storage buckets = Uses the Amazon S3 protocol.

Storage pool §

• When you first setup LXD, you assign some space for all the containers. This space is called a storage pool.
• Storage pool = Multiple storage volumes of different content type(images/instances) + Storage buckets
• Storage pool needs a storage driver. One of dir, btrfs, lvm, zfs, ceph-X
• In a profile, the storage pool to use is defined by the pool for the root disk device

Storage location §

When using dir, the reported usage by the kernel would be that of the entire partition.

These 3 will be same:

• df -h / on the host
• df -h / in any of the containers
• lxc storage info default

• We have 4 options here: Shared with the host, Dedicated disk or partition, Loop disk, Remote storage
• For my usecase it’s Shared with the host with dir storage driver

Storage volume §

• There is no concept of a default storage pool in LXD. When you create a storage volume, you must specify the storage pool to use.
• 3 types
  • container/vm
    • Auto created at instance creation at storage pool specified by instance/profile/default profile
    • Used as the root disk for the instance
    • Destroyed when the instance is deleted.
  • image
    • Only for storage drivers that support optimized image storage.
    • If deleted manually, auto deleted after ten days after last used to launch an instance.
  • custom
    • Volumes to hold data that you want to store separately from your instances.
    • Can be shared between instances
    • Has to be manually deleted
• 2 content types: filesystem (container and vms) and block (vm only)

Networking §

• Proxy Device vs Network Forwarding
  • Proxy device: Instance specific, bi-directional, does not require a network connection(if non-nat mode)
  • Network forward: Similar to proxy device+nat mode. Defined at network level instead of instance level.
  • More info: Difference between network forward and proxy device

Container nesting §

You can run docker inside LXD by setting security.nesting