We have changed daily CI tests from VM to Docker, ever since early last year. It is just awesome! Way too fast and cost-effective.
But one annoying thing keeps dragging us slow. The Docker daemon server runs into low disk capacity quite often.
Update Per Audience Feedback:
- docker-gc[1]: helps to remove all containers that has been exited over an hour ago, together with their respective images.
Just for deployment CI, we run below routine testcases:
- All-In-One deployment
- Standard 3-nodes cluster, 6-nodes cluster deployment
- Customized cluster deployment
- DB and application HA deployment test
- Continuous upgrade test
- Sandbox Test by docker-in-docker
- etc
All testcases apply both Docker and DigitalOcean. Docker is the default driver for sure. Consequently, two potential unpleasant issues happen:
- Docker machine runs into OOM (Out-Of-Memory), when heavy tests run simultaneously. Then we have to reboot machine by force. This leaves garbage files.
- Container removal fails in docker-in-docker scenario. Again the failure of resource cleanup costs.
Eventually, we get alerts of low disk over and over. What to do?
Remove All Useless Resources.
A very common suggestion. Yes, remove all unused containers, images and volumes.
Remove exited containers
docker ps --filter status=exited -aq \ | xargs -r docker rm -v
Even better, you can try a handy tool, docker-gc[1]. It removes all containers that has been exited over an hour ago, together with their respective images.
Literally speaking, we’re facing resource leak. This tip helps, unfortunately it doesn’t solve the problem.
Remove unused docker images
docker rmi $(docker images | grep "<none>"\ | awk -F' ' '{print $3}') # Remove orphaned docker volumes docker volume rm \ $(docker volume ls -qf dangling=true)
Remove dead containers
docker ps --filter status=dead -aq \ | xargs -r docker rm -v
Remove intermediate containers generated during docker build
docker ps -a | grep "/bin/sh -c" | \ awk -F' ' '{print $1}' | xargs docker rm
Remove Image with <none> string
echo "Remove docker images with <none> string"
if docker images | grep none | tee; then
docker rmi $(docker images | grep "<none>" | awk -F' ' '{print $3}') | tee
fi
Keep Docker Up To Date
Everyday we see more and more exciting news or improvements of Docker. Definitely newer version is more capable to handle this resource reclaim issue.
Install latest docker:
curl -sSL https://get.docker.com/ | sudo sh
Upgrade docker to given version. Use this on your own risk! Docker might fail to start.
# Install resources wget -qO- https://get.docker.com/ | sh # Show version which are available apt-cache showpkg docker-engine # Install given version of docker apt-get install docker-engine=1.12.1-0~trusty # Prevent upgrade on sys upgrade apt-mark hold docker-engine docker version
Rebuild Docker Env From Scratch
We’re using aufs for docker storage, which quite common. The folders growing in size are:
- /var/lib/docker/aufs/diff
- /var/lib/docker/aufs/mnt
Running du command, we know our mnt directory takes 18GB and diff directory 85GB. With reasonable estimation, it should be less than 10 GB. No doubt some folders could be deleted to reclaim capacity. But how to selectively remove folders? Before docker 1.10, we can mapping container id to subdirectories under diff[2]. Now it’s more complicated! Even if we can, it’s better not. No idea how docker would evolve in the future.
Examining our CI case carefully, just notice there is actually only one important container. It runs Jenkins + facilities (Kitchen, Chef, ssh scripts, etc). Any other containers are disposable. Another approach: what if we re-install docker to have a fresh restart?
- Firstly export unrecoverable data from the container.
# Caculate disk usage of containers. # Note this do take time! docker ps -s # Or run du command inside containers. du -h -d 1 /
To export valuable data, we don’t want to handle all the tricky application backup or redo manual steps. Thus we simply export container to an archived file. To keep the file as small as possible, remember to do clean up inside the container before export.
docker export $container_id > $container.tar
- Soft delete /var/lib/docker and reinstall docker daemon.
- Recreate container and restore to original state.
docker import $container.tar
The whole process might take a while. For your reference, our jenkins container takes 14GB disk, docker export takes 5 min, docker import takes 10 min.
Recap for our discussion of docker disk reclaim
- Keep docker up-to-date.
- When ordinary trick doesn’t help for the disk resource leak, try to rebuild docker daemon with proper export/import.
More Reading: Explore Docker Image For Security Concern
Footnotes:
Original URL: https://dennyzhang.com/docker_capacity
Just Google “docker-gc” and save yourself all that trouble…
Juan, like we discussed in Linkedin, docker-gc is a very handy cleanup tool.
Thanks for sharing that. Updated the post per your comment.
Nice post, thanks for your sharing
[…] Docker Low Disk Space. Clean Up Disk Leak! […]