8 Lessons Learned Using docker-compose

Host all necessary things in git repo: docker-compose.yml, .env and docker volume files.

• Everything you need can be and should be found in git repo.
• People can easily review and comment your changes via PRs.
• You can audit the change history, and understand issues.

You may get surprised how insecure docker port mapping feature is! Let me show you.

Image you have a mysql instance running with below docker-compose.yml.

version: '2'
services:
  db:
    container_name: db
    image: mysql
    ports:
      - "3306:3306"
    volumes:
     - db_home:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: rootpassword
      MYSQL_USER: user1
      MYSQL_PASSWORD: password1
      MYSQL_DATABASE: db1

volumes:
  db_home:

With default setting, anyone from the internet can access your db, port 3306.[1]

root@denny:# docker-compose up -d
root@denny:# iptables -L -n | grep 3306
ACCEPT  tcp  --  anywhere  172.17.0.2  tcp dpt:3306
root@denny:# telnet $YOUR_IP 3306

Why? docker add iptables rules widely open FOR YOU. (Sweet, isn’t it?) Anywhere can access.

So let’s limit the source ip for the access. Good idea! Unfortunately, it’s impossible. That rule is added by docker. To make it worse, it provides no hook points for us to change this behavior.

With some more thinking, you may think: how about I delete the rules created by docker, and add a new rule? I’m an iptable expert, it’s easy. Well, it’s not easy. The tricky part is your customized iptable rules won’t be recognized and managed by docker. It would easily mess up, when service/machine reboot. Especially when we restart container, the ip will change.

So instead of default port mapping, I only bound the port mapping to specific ip address like below.

ports:
  - "127.0.0.1:3306:3306"

Make sure containers doesn’t hold any unrecoverable application data.

Consequently we can safely recreate docker envs, or even migrate to another environment completely and easily.

It’s always a good idea to run migration rehearsal.

Ideally there should be no more than 3 steps:

• scp data volume from old env to new env
• Install docker compose and run “docker-compose up -d”
• Very few manual steps, mostly it’s about credentials.

All critical data would be in volumes only. To backup the system, we just need to backup the folders of data volumes.

When the data volumes are not very big, we can:

• Enforce periodical folder backup for data volumes, like weekly backup.
• To reduce out-of-disk issues, remember to rotate very old backupset. (Read: Reduce Support Effort Of Low Free Disk Issues).
• To avoid local VM failures, backup to a remote VMs or AWS S3.

During deployment, any request to external services is a failure point.

To make sure the deployment is more smoothly, I always build my own docker images pre-download packages/files, if necessary.

Monitoring and alerting are good.

• Define docker healthcheck, thus trouble shooting would be as simple as below

docker-compose ps
docker ps | grep unhealthy