[Container] Run Process Debug Tools, But Install Nothing

Ever need to debug your process in containers? Use strace, lsof, pstree, or anything you name it. But after login, you get a surprise: The tools are not installed!

So what will you do? Typically we have 3 different methods. Check it out, and discuss with me.

(Hint: You don’t have to install the tools at all)

Container Run Process Debug Tools, But Install Nothing



Check More Discussion In LinkedIn

3 Ways To Run Debug Tools Against Your Container Process


Start a nginx container for explanation. We will run tools against the nginx process afterwards.

# pull docker image
docker pull nginx:alpine

# start container
docker run -t -d --privileged \
    -h nginxtest --name nginx-test \
    -p 8080:80 nginx:alpine

Check nginx process:

# verify nginx httpd service
curl http://localhost:8080

# check process pid
docker exec nginx-test ps -ef | grep nginx

1.1 METHOD1: Debug From The Container Inside

Let’s say you want to “strace -p $nginxpid“. But strace is not available in nginx:alpine image.

Login and install. Yes, it will work. But just old school.

docker exec -u root -it nginx-test sh

# Install strace
apk --update add strace

# strace nginx process
strace -p 1
##   strace: Process 1 attached
##   rt_sigsuspend([], 8

Why it’s old school? It will populate the env. Especially when the containers are in production mode.

The more packages you have installed, the more issues you will get.

(Similar reading: What Packages Were Installed In My Server, 5 Common Failures Of Package Installation)

1.2 METHOD2: Debug From Docker Host

Linux containers share the same linux kernel.

We can find the process id from docker host, then debug the process.

# get nginx process pid
root@denny:/tmp# ps -ef | grep nginx
root     27871 27834  0 22:00 pts/2    00:00:00 nginx: master process nginx -g daemon off;
systemd+ 27931 27871  0 22:00 pts/2    00:00:00 nginx: worker process
root     31324 27756  0 23:26 pts/1    00:00:00 grep --color=auto nginx

# Install tools in docker host
apt-get install -y strace

# Run tools from docker host
root@denny:/tmp# strace -p 27871
strace: Process 27871 attached
rt_sigsuspend([], 8

1.3 METHOD3: Debug From An Ephemeral Container

  • Build a temporary image with tools installed
# Dockerfile
cat > Dockerfile <<EOF
FROM alpine
RUN apk update && apk add strace
CMD ["strace", "-p", "1"]
EOF

# Build image
docker build -t strace .
  • Start a temporary container(nginx-test). Then debug nginx process by strace.
export test_conainter="nginx-test"
docker run -t --name strace-test \
  --pid=container:$test_conainter \
  --net=container:$test_conainter \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  strace

Don’t forget to destroy the container, when you have finished your debugging.

Apparently I like method3 the best. In this article, we use strace tool. Surely we can support more tools like this.

Not so difficult as you thought. Right, my friend?

So why don’t you give it a try now? Or share this post with your colleagues?


More Reading:

linkedin
github
slack

PRs Welcome

Blog URL: https://www.dennyzhang.com/docker_debug


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.