Detect Suspicious Linux Processes

Ever bothered by suspicious processes running in your servers? No doubt how dangerous they might be: valuable data leaked, CPU/memory wasted, or DDoS attack other victims, etc.

How to easily capture those annoying troublemakers? Even better, get alerted without extra human effort.

Suspicious Linux Processes


List all non-kernel processes.

Usually kernel processes are safe and clean. For kernel processes, either PID(process id) is 2 or PPID(parent process id) is 2. Here is how to get all non-kernel processes.

# rss(resident set size): real RAM usage
# -deselect: rule out matched processes
root@denny:~# ps --ppid 2 -p 2 -p 1 \
   --deselect -o uid,pid,rss,%cpu,command
UID   PID   RSS %CPU COMMAND
   0   411  1848  0.0 /lib/systemd/systemd-
   0   572  2904  0.0 dhclient -1 -v -pf /r
 102   902  1244  0.0 dbus-daemon --system
   0   912  1948  0.0 /lib/systemd/systemd-
   0  5869   388  0.0 upstart-socket-bridge
 200  1953   904  0.0 /usr/sbin/apache2 -k
 200  3463  3700  0.0 /usr/sbin/apache2 -k
  ...  ...
  ...  ...
   0  5098  4224  0.0 sshd: ubuntu [priv]
   0  5139  1748  0.0 /usr/bin/python /usr/
 200  5140  3484  0.0 /usr/bin/python /usr/
 200  5176  1904  0.0 sshd: ubuntu@pts/3
 200  5177  3860  0.0 -bash
 200  5193  1200  0.0 tmux attach -t denny
   0  5297  4224  0.0 sshd: ubuntu [priv]
  ...  ...
  ...  ...

Rule out trusted procsses.

We may have many processes running, which are expected and trusted. e.g apache2, tomcat7, mysqld, etc. To avoid distraction, build a white list especial for your project.

Sort processes by memory and cpu

We’re more concerned about suspicious processes using noticeable resource.

# Sort by memory first, then cpu
ps --ppid 2 -p 2 -p 1 --deselect \
  -o uid,pid,rss,%cpu,command, \
  --sort -rss,-cpu

Automate Detection Process and Get Alerts

Here comes a python script in Github (detect_suspicious_process.py).

You can use it to capture all unknown process.

  1. Create a crontab or scheduled jenkins job to check the count of suspicous processes.
  2. Raise alert, if the count is not 0 or it changes.

It might take a while to build a suitable white list. Once it’s done, your servers are always more secured and managed!

github-detect-suspicious-process.jpg

More Reading:


Original URL: https://dennyzhang.com/suspicious_process

Connect with Denny In LinkedIn Or MailList

Leave a Reply

Your email address will not be published. Required fields are marked *