Blog: Forensic container checkpointing in Kubernetes

Authors: Adrian Reber (Red Hat)

Forensic container checkpointing is based on Checkpoint/Restore In
(CRIU) and allows the creation of stateful copies
of a running container without the container knowing that it is being
checkpointed. The copy of the container can be analyzed and restored in a
sandbox environment multiple times without the original container being aware
of it. Forensic container checkpointing was introduced as an alpha feature in
Kubernetes v1.25.

How does it work?

With the help of CRIU it is possible to checkpoint and restore containers.
CRIU is integrated in runc, crun, CRI-O and containerd and forensic container
checkpointing as implemented in Kubernetes uses these existing CRIU

Why is it important?

With the help of CRIU and the corresponding integrations it is possible to get
all information and state about a running container on disk for later forensic
analysis. Forensic analysis might be important to inspect a suspicious
container without stopping or influencing it. If the container is really under
attack, the attacker might detect attempts to inspect the container. Taking a
checkpoint and analysing the container in a sandboxed environment offers the
possibility to inspect the container without the original container and maybe
attacker being aware of the inspection.

In addition to the forensic container checkpointing use case, it is also
possible to migrate a container from one node to another node without loosing
the internal state. Especially for stateful containers with long initialization
times restoring from a checkpoint might save time after a reboot or enable much
faster startup times.

How do I use container checkpointing?

The feature is behind a feature gate, so
make sure to enable the ContainerCheckpoint gate before you can use the new

The runtime must also support container checkpointing:

  • containerd: support is currently under discussion. See containerd
    pull request #6965 for more details.

  • CRI-O: v1.25 has support for forensic container checkpointing.

Usage example with CRI-O

To use forensic container checkpointing in combination with CRI-O, the runtime
needs to be started with the command-line option --enable-criu-support=true.
For Kubernetes, you need to run your cluster with the ContainerCheckpoint
feature gate enabled. As the checkpointing functionality is provided by CRIU it
is also necessary to install CRIU. Usually runc or crun depend on CRIU and
therefore it is installed automatically.

It is also important to mention that at the time of writing the checkpointing functionality is
to be considered as an alpha level feature in CRI-O and Kubernetes and the
security implications are still under consideration.

Once containers and pods are running it is possible to create a checkpoint.
is currently only exposed on the kubelet level. To checkpoint a container,
you can run curl on the node where that container is running, and trigger a

curl -X POST "https://localhost:10250/checkpoint/namespace/podId/container"

For a container named counter in a pod named counters in a namespace named
default the kubelet API endpoint is reachable at:

curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"

For completeness the following curl command-line options are necessary to
have curl accept the kubelet‘s self signed certificate and authorize the
use of the kubelet checkpoint API:

--insecure --cert /var/run/kubernetes/client-admin.crt --key /var/run/kubernetes/client-admin.key

Triggering this kubelet API will request the creation of a checkpoint from
CRI-O. CRI-O requests a checkpoint from your low-level runtime (for example,
runc). Seeing that request, runc invokes the criu tool
to do the actual checkpointing.

Once the checkpointing has finished the checkpoint should be available at

You could then use that tar archive to restore the container somewhere else.

Restore a checkpointed container outside of Kubernetes (with CRI-O)

With the checkpoint tar archive it is possible to restore the container outside
of Kubernetes in a sandboxed instance of CRI-O. For better user experience
during restore, I recommend that you use the latest version of CRI-O from the
main CRI-O GitHub branch. If you’re using CRI-O v1.25, you’ll need to
manually create certain directories Kubernetes would create before starting the

The first step to restore a container outside of Kubernetes is to create a pod sandbox
using crictl:

crictl runp pod-config.json

Then you can restore the previously checkpointed container into the newly created pod sandbox:

crictl create <POD_ID> container-config.json pod-config.json

Instead of specifying a container image in a registry in container-config.json
you need to specify the path to the checkpoint archive that you created earlier:

 "metadata": {
 "name": "counter"
 "image": "/var/lib/kubelet/checkpoints/<checkpoint-archive>.tar"

Next, run crictl start <CONTAINER_ID> to start that container, and then a
copy of the previously checkpointed container should be running.

Restore a checkpointed container within of Kubernetes

To restore the previously checkpointed container directly in Kubernetes it is
necessary to convert the checkpoint archive into an image that can be pushed to
a registry.

One possible way to convert the local checkpoint archive consists of the
following steps with the help of buildah:

newcontainer=$(buildah from scratch)
buildah add $newcontainer /var/lib/kubelet/checkpoints/checkpoint-<pod-name>_<namespace-name>-<container-name>-<timestamp>.tar /
buildah config<container-name> $newcontainer
buildah commit $newcontainer checkpoint-image:latest
buildah rm $newcontainer

The resulting image is not standardized and only works in combination with
CRI-O. Please consider this image format as pre-alpha. There are ongoing
discussions to standardize the format of checkpoint
images like this. Important to remember is that this not yet standardized image
format only works if CRI-O has been started with --enable-criu-support=true.
The security implications of starting CRI-O with CRIU support are not yet clear
and therefore the functionality as well as the image format should be used with

Now, you’ll need to push that image to a container image registry. For example:

buildah push localhost/checkpoint-image:latest container-image-registry.example/user/checkpoint-image:latest

To restore this checkpoint image (container-image-registry.example/user/checkpoint-image:latest), the
image needs to be listed in the specification for a Pod. Here’s an example

apiVersion: v1
kind: Pod
 namePrefix: example-
 - name: <container-name>
 image: container-image-registry.example/user/checkpoint-image:latest
 nodeName: <destination-node>

Kubernetes schedules the new Pod onto a node. The kubelet on that node
instructs the container runtime (CRI-O in this example) to create and start a
container based on an image specified as registry/user/checkpoint-image:latest.
CRI-O detects that registry/user/checkpoint-image:latest
is a reference to checkpoint data rather than a container image. Then,
instead of the usual steps to create and start a container,
CRI-O fetches the checkpoint data and restores the container from that
specified checkpoint.

The application in that Pod would continue running as if the checkpoint had not been taken;
within the container, the application looks and behaves like any other container that had been
started normally and not restored from a checkpoint.

With these steps, it is possible to replace a Pod running on one node
with a new equivalent Pod that is running on a different node,
and without losing the state of the containers in that Pod.

How do I get involved?

You can reach SIG Node by several means:

Originally posted on Kubernetes – Production-Grade Container Orchestration

Deja una respuesta

Tu dirección de correo electrónico no será publicada. Los campos obligatorios están marcados con *