tags : OCI Ecosystem, Linux
Basics
There is no such thing as a container, but only as much as containing is a thing you do.
Linux containers
Modern containers are a mix of cgroups, namespaces, pivot_root (like chroot but more appropriate for containers), and seccomp. But this is just one possible implementation. There are also, certainly, other ways of implementing containers.
- Docker : Docker is a reference implementation for the concept of containers. It provided a simple, practical way to bundle applications and their dependencies in a relatively standardized way. Docker’s main innovation was the layering system
- LXC & LXD
- Other linux container solutions: Some implementations (e.g. firecracker-containerd) also use SELinux and CPU virtualization support.
Containers without namespaces
- One could ship application components as containers, but have them run as VMs, eg. firecracker-containerd
- Dockers for Windows and Macs do not even use “cgroups” and “namespaces” because these technologies are not available on these stacks, it resorts to plain old VMs.
Comparison w Solaris Zones and BSD Jails
- Ramblings from Jessie
- Solaris Zones, BSD Jails are first class concepts.
- Containers on the other hand are not real things.
- “Containers are less isolated than Jails/Zones but allow for a flexibility and control that is not possible with Jails, Zones, or VMs. And THAT IS A FEATURE.”
- Containers are not sandboxing but we can attempt to sandbox it and good attempts have been made. One example is gvisor
Images
What is it?
- a
.tar
- We basically bundle the application together w deps and just enough of a Linux root filesystem to run it.
- Usually updated in version compared to updated in place
Implementations
Dockerfile
Nix/Guix
Filesystem and storage
Overlayfs
- Containers usually don’t have a persistent root filesystem
- overlayfs is used to create a temporary layer on top of the container image.
- This is thrown away when the container is stopped.
Bind mounts
- persistent data outside of the container image is grafted on to the container’s filesystem via a bind mount to another location on the host.
Privileged and Unprivileged
Whether the root user in the container is the “real” root user (uid 0 at the kernel level).
Unprivileged containers != Rootless containers
If containers are running as non-root users, when the runtime is still running as root, we don’t call them Rootless Containers.
Unprivileged created by:
- Taking a set of normal UIDs and GIDs from the host
- Usually at least
65536
of each (to be POSIX compliant) - Then mapping those into the container
- Implementations mostly expect
/etc/subuid
to contain at least65536 subuids
. - This allows LXD to do the “shift” in containers because it has a reserved pool of UIDs and GIDs.
subuid and subgid subordinate(uids/gids)
/etc/subuid
and/etc/subgid
let you assign extra user ids and group ids to a particular user. Thesubuid
file contains a list of users and the user ids that the user is allowed to impersonate.- Any resource owned by user(inside the container) which is not mapped(outside container) will get id
-1
(nobody
) - Range of ids you assign are no longer available to be assigned to other users (both primary and via
subuid
) user
to which they are assigned now ‘owns’ these ids.
- Any resource owned by user(inside the container) which is not mapped(outside container) will get id
- Configured via the
subid
field in/etc/nsswitch.conf
(nsswitch) file. (Has default set tofiles
) - Different from setuid and setgid bits
- See shadow-utils
Manual configuration
$ cat /etc/subuid
#<user>:base_id:total_nos_of_ids_allowed
user1:100000:65536
$ cat /etc/subgid
user1:100000:65536
Using usemod
usermod --add-subuids 1000000-1000999999 root
usermod --add-sugids 1000000-1000999999 pappu
usermod --add-subuids 1000000-1000999999 --add-sugids 1000000-1000999999 xyzuser
What about uid_map
and gid_map
?
These are specific to user NS. The newuidmap
tool can help. newuidmap
sets /proc/[pid]/uid_map
based on its command line arguments and the uids allowed.
Container networking
- Also see Docker networking
- net ns can be connected using a Linux Virtual Ethernet Device or
veth pair
- From a network architecture point of view, all containers on a given Docker host are sitting on bridge interfaces.
- Different container managers(docker, podman, lxd etc) provide a number of ways networking can be done with containers.
- An interesting one is the
bridged networking
approach, which essentially boils down to3 things
.- Creating
veth
pair fromhost
tonet namespace-X
. Every new container will add newveth
interface and remove it once container is stopped. Eg.lxc info <instance_name>
will show theveth
created for theinstance
- Adding a
bridge
for theveth
pair to talk through. When you installdocker
, it automatically creates adocker0
bridge created for containers to communicate.bridge
is a L2 device, uses ARP. - Adding iptables rules to access outside network
- Creating
- See Introduction to Linux interfaces for virtual networking and Deep Dive into Linux Networking and Docker - Bridge, vETH and IPTables
- Also see 8.2.5 About Veth and Macvlan
TODO Containers in Practice
TODO Golang Questions
- What user to use gid etc.
- Which image to use, why not alpine
- I recomend agains alpine images. They use muslc, that can be a real troublemaker at times. If you don’t need any tooling, then distroless image is great. Otherwise debian-slim ticks everything for me.
- You can also use SCRATCH but be aware of things like outgoing HTTPS requests, where you need a local CA certificate list for validating certificates.
- We can also do COPY —from=build etc/ssl/certs/ca-certificates.crt /etc/ssl/certs
- Distroless
- a base image with just ca-certificates, a passwd entry and tzdata, which are a few dependencies the Go runtime might look for at runtime.
- So it’s pretty nice, rather than using SCRATCH
- Q: Why not OS base image in golang applications? what about cgo.
- Which image tool to use, docker or something else?
- so if you are using containerization in development and not building inside the container, you are missing out on one of the major advantages of the paradigm.
- So we should be building inside the container?
- https://docs.docker.com/build/building/multi-stage/
- Building a docker image for a Go programm : golang
- Usually people use multi-stage builds. Use the golang image to build the binary, then copy that binary to the more minimal alpine image.