Docker Low Disk Space. Clean Up Disk Leak!

Posted by DennyZhang

We have changed daily CI tests from VM to Docker, ever since early last year. It is just awesome! Way too fast and cost-effective.

But one annoying thing keeps dragging us slow. The Docker daemon server runs into low disk capacity quite often.

Low Disk

Update Per Audience Feedback:

  • docker-gc[1]: helps to remove all containers that has been exited over an hour ago, together with their respective images.

Just for deployment CI, we run below routine testcases:

  1. All-In-One deployment
  2. Standard 3-nodes cluster, 6-nodes cluster deployment
  3. Customized cluster deployment
  4. DB and application HA deployment test
  5. Continuous upgrade test
  6. Sandbox Test by docker-in-docker
  7. etc

All testcases apply both Docker and DigitalOcean. Docker is the default driver for sure. Consequently, two potential unpleasant issues happen:

  1. Docker machine runs into OOM (Out-Of-Memory), when heavy tests run simultaneously. Then we have to reboot machine by force. This leaves garbage files.
  2. Container removal fails in docker-in-docker scenario. Again the failure of resource cleanup costs.

Eventually, we get alerts of low disk over and over. What to do?

1.1 Remove All Useless Resources.

A very common suggestion. Yes, remove all unused containers, images and volumes.

1.1.1 Remove exited containers

docker ps --filter status=exited -aq \
 | xargs -r docker rm -v

Even better, you can try a handy tool, docker-gc[1]. It removes all containers that has been exited over an hour ago, together with their respective images.

Literally speaking, we’re facing resource leak. This tip helps, unfortunately it doesn’t solve the problem.

1.1.2 Remove unused docker images

docker rmi $(docker images | grep "<none>"\
 | awk -F' ' '{print $3}')

# Remove orphaned docker volumes
docker volume rm \
 $(docker volume ls -qf dangling=true)

1.1.3 Remove dead containers

docker ps --filter status=dead -aq \
 | xargs -r docker rm -v

1.1.4 Remove intermediate containers generated during docker build

docker ps -a | grep "/bin/sh -c" | \
  awk -F' ' '{print $1}' | xargs docker rm

1.1.5 Remove Image with <none> string

echo "Remove docker images with <none> string"
if docker images | grep none | tee; then
   docker rmi $(docker images | grep "<none>"  | awk -F' ' '{print $3}') | tee
fi

1.2 Keep Docker Up To Date

Everyday we see more and more exciting news or improvements of Docker. Definitely newer version is more capable to handle this resource reclaim issue.

Install latest docker:

curl -sSL https://get.docker.com/ | sudo sh

Upgrade docker to given version. Use this on your own risk! Docker might fail to start.

# Install resources
wget -qO- https://get.docker.com/ | sh
# Show version which are available
apt-cache showpkg docker-engine
# Install given version of docker
apt-get install docker-engine=1.12.1-0~trusty
# Prevent upgrade on sys upgrade
apt-mark hold docker-engine
docker version

1.3 Rebuild Docker Env From Scratch

We’re using aufs for docker storage, which quite common. The folders growing in size are:

  • /var/lib/docker/aufs/diff
  • /var/lib/docker/aufs/mnt

Running du command, we know our mnt directory takes 18GB and diff directory 85GB. With reasonable estimation, it should be less than 10 GB. No doubt some folders could be deleted to reclaim capacity. But how to selectively remove folders? Before docker 1.10, we can mapping container id to subdirectories under diff[2]. Now it’s more complicated! Even if we can, it’s better not. No idea how docker would evolve in the future.

Examining our CI case carefully, just notice there is actually only one important container. It runs Jenkins + facilities (Kitchen, Chef, ssh scripts, etc). Any other containers are disposable. Another approach: what if we re-install docker to have a fresh restart?

  • Firstly export unrecoverable data from the container.
# Caculate disk usage of containers.
# Note this do take time!
docker ps -s

# Or run du command inside containers.
du -h -d 1 /

To export valuable data, we don’t want to handle all the tricky application backup or redo manual steps. Thus we simply export container to an archived file. To keep the file as small as possible, remember to do clean up inside the container before export.

docker export $container_id > $container.tar

  • Soft delete /var/lib/docker and reinstall docker daemon.
  • Recreate container and restore to original state.
docker import $container.tar

The whole process might take a while. For your reference, our jenkins container takes 14GB disk, docker export takes 5 min, docker import takes 10 min.

1.4 Recap for our discussion of docker disk reclaim

  1. Keep docker up-to-date.
  2. When ordinary trick doesn’t help for the disk resource leak, try to rebuild docker daemon with proper export/import.

More Reading: Explore Docker Image For Security Concern

[1] https://github.com/spotify/docker-gc
[2] https://docs.docker.com/engine/userguide/storagedriver

linkedin
github
slack

PRs Welcome

Blog URL: https://www.dennyzhang.com/docker_capacity

4 Responses to Docker Low Disk Space. Clean Up Disk Leak!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.