Get Alerts, When Containers Run Into Issues

I’m running docker containers for all side projects. Usually one single container. Sometimes multiple containers started by docker-compose[1].

If containers get issues, I want to get alerts.

What a typical monitoring requirement! But if you do some research, you will find the information is just overwhelming. cAdvisor, Prometheus, InfluxDB, etc. Excuse me? Can’t we have a simple solution for this simple requirement? Here is my answer. Try it and discuss with me.

Get Alerts, When Containers Run Into Issues


This solution works for docker-compose deployment. Please check Prometheus[2], for more complex docker environments.

BASIC IDEA

Firstly I admit: Prometheus can do the job. Yes, it’s capable, but …

Hint: Want to use prometheus in a relatively easy way, try dockprom [3]

Frankly speaking, learning Prometheus took much longer than what I had expected. And it’s a bit over-killing for my scenario. Prometheus solution itself will start multiple containers. But my envs only have very few containers to monitor.

Ironically “docker ps -a” can easily answer most of my questions.

root@denny-blog:~# docker ps -a
CONTAINER ID    IMAGE             COMMAND                  CREATED         STATUS                     PORTS                   NAMES
2936e1d7ec34    denny_wordpress   "/bin/sh -c /docke..."   3 weeks ago     Up 13 days (healthy)       0.0.0.0:443->443/tcp    blog-wordpress
87665b7a76df    mysql:5.7         "docker-entrypoint..."   3 weeks ago     Up 3 weeks                 3306/tcp                blog-mysql
4aa821fa9d37    denny/ss:v2       "ssserver -c /etc/..."   3 weeks ago     Up 3 weeks (healthy)       0.0.0.0:6187->6187/tcp  shadowsock
ea96f4dbefaa    ubuntu:14.04      "/bin/bash"              14 seconds ago  Exited (0) 6 seconds ago                           nginx-test
  1. To get stopped container: just check “STATUS” of each container.
  2. To get unhealthy containers: check status for “unhealthy” output.

    Hint: To enable docker healthcheck[4], you need update your Dockerfiles properly
  3. Send alerts, whenever any of above occurrences have happened. Should be easy, Right?

So how about we start a container, which keeps polling status of all containers. Then send out alerts, if necessary?

You may think: we won’t be able to run “docker ps” in containers. It only runs in docker host.

Yes, you’re right. But there is one way out. You can query status via docker unix socket file. Try below locally, you will understand.

docker run -v /var/run/docker.sock:/var/run/docker.sock \
    -t -d --privileged --name test-socket \
    --entrypoint=/bin/sh byrnedo/alpine-curl

docker exec -it test-socket \
    curl -XGET --unix-socket /var/run/docker.sock \
    http://localhost/containers/json

HOW TO USE

Check code in Github. Follow @dennyzhang

github-docker-monitor-container.png

github-docker-monitor-compose.png

Try it, if you find it’s useful. And discuss with me.

2 Responses to Get Alerts, When Containers Run Into Issues

Leave a Reply

Your email address will not be published. Required fields are marked *