We have changed daily CI tests from VM to Docker, ever since early last year. It is just awesome! Way too fast and cost-effective.
But one annoying thing keeps dragging us slow. The Docker daemon server runs into low disk capacity quite often.
Update Per Audience Feedback:
- docker-gc: helps to remove all containers that has been exited over an hour ago, together with their respective images.
Just for deployment CI, we run below routine testcases:
- All-In-One deployment
- Standard 3-nodes cluster, 6-nodes cluster deployment
- Customized cluster deployment
- DB and application HA deployment test
- Continuous upgrade test
- Sandbox Test by docker-in-docker
All testcases apply both Docker and DigitalOcean. Docker is the default driver for sure. Consequently, two potential unpleasant issues happen:
- Docker machine runs into OOM (Out-Of-Memory), when heavy tests run simultaneously. Then we have to reboot machine by force. This leaves garbage files.
- Container removal fails in docker-in-docker scenario. Again the failure of resource cleanup costs.
Eventually, we get alerts of low disk over and over. What to do?