docker-bench-security: Audit Well-Known Docker Vulnerabilities

Posted by DennyZhang

docker-bench-security is a script that checks for dozens of common best-practices around deploying Docker containers in production.
We use Docker to save effort, not the other way around. Try docker-bench-security to avoid common pitfalls using Docker. It’s provided by Docker, Inc. Totally free and constantly polished.

Enclosed is all you need to know, in order to understand and use the tool.

docker-bench-security: audit all known container vulnerabilities

Before we dive deep, please remember there is nothing that is absolutely secure. Nothing. It’s up to you to figure out what your security boundary is, and how you mitigate failures.

Besides this tool, I will introduce more tips or skills in future posts. Hope all the effort can better secure your Docker envs. (Product Review: NeuVector, Build Run-Time Container Security)

Security is one of the biggest challenges for docker. Too many pitfalls and potential vulnerabilities. As normal users, not everyone can be or should be a Docker security expert.

Here comes docker-bench-security in github. By now, the script enforces 63 known security checks for Docker. To audit OS security, people usually use tools like lynis. To audit Docker security, try this one.

1.1 Enforce docker-bench-security is super easy.

Just start a container, run the check, and dispose the container afterwards.

# Pull docker image
docker pull docker/docker-bench-security

# Enforce the check
docker run -it --net host --pid host \
 --cap-add audit_control \
 -v /var/lib:/var/lib \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /usr/lib/systemd:/usr/lib/systemd \
 -v /etc:/etc --label docker_bench_security \
 docker/docker-bench-security

docker-bench-security: audit all known container vulnerabilities

1.2 Understand Common Failures

No doubt we may get many warnings/errors for the first time. No worry, we will conquer the major ones together.

Getting familiar with docker-bench-security is good. More importantly, we learn how to avoid common pitfalls for Docker security.

Let’s go through the common major issues with below format:

  1. Error Msg. Console output of docker-bench-security.
  2. What For. What does it mean and what’s the impact.

1.3 [WARN] – Create a separate partition for containers

  • Error Msg:
[WARN] 1.1  - Create a separate partition for containers

For docker daemon, create a separate partition for the data folder. It is usually /var/lib/docker.

Personally I think it’s also a good practice to do the same for docker containers.

  1. Be safe with our key data. Store key data in a separate partition. This will dramatically simplify our backup/restore process. When something goes terribly wrong, we can skip the trouble shooting. Simply start a new container, and mount the volume. 2. For devicemapper storage driver, containers will only have 10GB disk. Without volume mount, we run into low capacity easily.

1.4 [WARN] – Keep Docker up to date

  • Error Msg:
[WARN] 1.5  - Keep Docker up to date
[WARN]       * Using 1.12.2, when 1.12.3 is current as of 2016-10-26

Docker is in rapid development. New version is usually better. The upgrade might fix issues and improve the performance. Warnings: 1. Run rehearsal before upgrading our critical envs. Always install latest versions could be troublesome. 2. Be prepared for potential upgrade failures.

# Upgrade Docker Daemon

# Check version before upgrade
docker version
service docker stop

curl -sSL https://get.docker.com/ | sudo sh

# Check version after upgrade
docker version
docker ps

1.5 [WARN] – Remove all non-essential services from the host – Network

  • Error Msg:
[WARN] 1.4  - Remove all non-essential services from the host - Network
[WARN]      * Host listening on: 9 ports

Ideally each container should only run one service. If our container listen on multiple ports, usually it would be a violation of this principle.

1.6 [WARN] – Restrict network traffic between containers

  • Error Msg:
[WARN] 2.1  - Restrict network traffic between containers

By default, unrestricted network traffic is enabled between all containers on the same host. If our containers doesn’t need to talk with each other, we should disable it. Add –icc=false to $DOCKEROPTS, when we start docker daemon.

root@denny:/var/log# dockerd --help | grep -C 5 icc
 --fixed-cidr-v6             IPv6 subnet for fixed IPs
 -G, --group=docker          Group for the unix socket
 -g, --graph=/var/lib/docker Root of the Docker runtime
 -H, --host=[]               Daemon socket(s) to connect to
 -h, --help                  Print usage
 --icc=true                  Enable inter-container communication
 --insecure-registry=[]      Enable insecure registry communication
 --ip=0.0.0.0                Default IP when binding container ports
 --ip-forward=true           Enable net.ipv4.ip_forward
 --ip-masq=true              Enable IP masquerading
 --iptables=true             Enable addition of iptables rules

Another useful post about this[1].

1.7 [WARN] – Set the logging level

  • Error Msg:
[WARN] 2.2  - Set the logging level

Specify log level, when we start docker daemon. Usually, we can add “-l log” to $DOCKEROPTS. For Ubuntu/Debian, check /etc/default/docker.

1.8 [WARN] – Do not use privileged containers

  • Error Msg:
[WARN] 5.4  - Do not use privileged containers
[WARN]      * Container running in Privileged mode: my-ss
[WARN]      * Container running in Privileged mode: docker-jenkins

Don’t use “–privileged” to start containers, unless we 100% need that.

1.9 [WARN] – Do not run ssh within containers

  • Error Msg:
[WARN] 5.6  - Do not run ssh within containers
[WARN]      * Container running sshd: docker-jenkins

Avoid running sshd, thus we can reduce the attack surface. Read more: her.[2]

1.10 [WARN] – Limit memory usage for container

  • Error Msg:
[WARN] 5.10 - Limit memory usage for container
[WARN]      * Container running without memory restrictions: my-ss
[WARN]      * Container running without memory restrictions: docker-jenkins

Limit the maximum memory your container can use. Thanks to cgroup, a very good self-protection mechanism.

To enable this, we can start containers with -m option. Let’s do a small experiment. With stress we can simulate a process using any given memory.[3]

# Start a container, only allow to use 600MB RAM at most
docker run -t -d --privileged -h mytest --name my-test \
  -m 600M ubuntu:14.04 /bin/bash

docker exec -it my-test bash

# Install stress tool
apt-get -y update
apt-get -y install stress

# Simulate using 500MB memory: should be fine
stress --vm-bytes 500m --vm-keep -m 1

# Simulate using 1000MB memory: should crash
stress --vm-bytes 1000m --vm-keep -m 1

# Sample output:
# stress: info: [83] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
# stress: FAIL: [83] (416) <-- worker 84 got signal 9
# stress: WARN: [83] (418) now reaping child worker processes
# stress: FAIL: [83] (452) failed run completed in 2s

1.11 [WARN] – Set container CPU priority appropriately

  • Error Msg:
[WARN] 5.11 - Set container CPU priority appropriately
[WARN]      * Container running without CPU restrictions: my-ss
[WARN]      * Container running without CPU restrictions: docker-jenkins

Similar like above.

1.12 [WARN] – Bind incoming container traffic to a specific host interface

  • Error Msg:
[WARN] 5.13 - Bind incoming container traffic to a specific host interface
[WARN]      * Port being bound to wildcard IP: 0.0.0.0 in my-ss
[WARN]      * Port being bound to wildcard IP: 0.0.0.0 in docker-jenkins
[WARN]      * Port being bound to wildcard IP: 0.0.0.0 in docker-jenkins
[WARN]      * Port being bound to wildcard IP: 0.0.0.0 in docker-jenkins
[WARN]      * Port being bound to wildcard IP: 0.0.0.0 in docker-jenkins

This would be the most dangerous issue, if you’re running docker in public cloud.

Let’s say, we start a container like below. And we export the db port 3306.

docker run -t -d -p 3306:3306 denny/app:v1 /bin/bash

This makes the docker wrapper transparent for end users. Whenever I visit $dockerhostip:3306, it points me to my db service directly. This is super nice, especially for development cycle.
However if we expose tcp ports like that, docker will configure our iptables to wildly open. That would be a serious problem, if running in public cloud. Anything can visit our db port directly. Sounds exciting, isn’t it?

The most absurd thing is Docker doesn’t support us to customize the behavior. Surely we can manually hack iptables rules, but it would be very tricky. Say when containers restart, container ip will change. Thus iptables rules need to updated manually. Again!

See more discussion in reddit.[4]

1.13 [WARN] – Avoid image sprawl

  • Error Msg:
[INFO] 6.4 - Avoid image sprawl
[INFO]      * There are currently: 52 images
[WARN]      * Only 9 out of 52 are in use

Remove unnecessary or unused docker images. This will help us to save disk capacity.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.