Delegation to tools

Procpath itself is only concerned with Procfs [1], but there is a wide range of Linux tools, language-specific or not, from profilers to system call tracers, which can provide the key to the problem at hand. These tools typically accept a PID or list of PIDs and, hence, benefit from the process tree query capability that Procpath provides.

In a simple case procpath query with shell stdout piping to the target command may suffice:

$ procpath query -d" " \
  "$..children[?('run:watch' in @.cmdline and @.stat.comm == 'procpath')].stat.pid"\
  | xargs -rn1 -- pstree -Tp
procpath(877)───sh(886)───xargs(888)─┬─pytest(889)───python(990)───python(995)
                                     ├─pytest(890)
                                     ├─pytest(891)───gunicorn(991)
                                     ├─pytest(892)───python(992)───python(993)
                                     ├─pytest(893)
                                     ├─pytest(894)───python(994)
                                     └─pytest(895)

But for most of the realistic cases, as a convenience to avoid scripting and/or terminal multiplexers in case of many process trees of interest (e.g. Celery nodes) Procpath has watch command which is analogous to procps watch. In this example watch delegates two process trees to smemstat [2] and py-spy [3].

procpath watch --interval 601 \
  -e TS='date +%s' \
  -e S1='systemctl show --property MainPID redis-server | cut -d "=" -f 2' \
  -e C1='docker inspect -f "{{.State.Pid}}" app_gunicorn_1' \
  -q L1='$..children[?(@.stat.pid == $S1)]..pid' \
  -c 'smemstat -q -o redis-memdiff-$TS.json -p $L1 30 20' \
  -c 'timeout --foreground --signal SIGINT 600 \
      py-spy record --subprocesses --output app-flamegraph-$TS.svg --pid $C1'

Notes:

  1. Typical watch pattern is:

    1. take the root PID from you process supervisor (systemd, Docker, etc)

    2. query all PIDs of its descendant processes

    3. pass the PID list to the analysis tool of choice

  2. The command environment is re-evaluated each --interval seconds

  3. A process is restarted each --interval seconds only if it has stopped and unless --no-restart is provided

  4. A process’s stdout output is forwarded as INFO, and stderr as WARNING logging records

  5. If the analysis tool of choice needs to work continuously and doesn’t have a means to terminate itself, it’s suggested to wrap it into timeout --foreground --signal SIGINT INTERVAL ... to get fixed interval measurements

  6. watch expects to be interrupted by SIGINT (Ctrl+C), where it sends SIGINT (by default) to all its descendant processes

  7. watch can run fixed number of repetitions specified by --repeat

  8. Commands are started serially where each command has PIDs of previously started commands in its environment variables WPS1, WPS2 and so on