Sharing and ops¶

Procpath commands are typically multi-line shell commands, and when it comes to sharing them as such, it can become unwieldy. These commands can range from a couple of queries to diagnose your workstation, which you’d like to share with your colleagues, to distributing a part of a commercial product’s, say delivered on premises of the customers as systemd services, troubleshooting operations procedure.

Playbook¶

To make writing and sharing command bundles easy, Procpath comes with another convenience layer – playbooks. Procpath playbooks are a Python configparser representation of its command-line interface, with a few bits of custom semantics. It looks like this:

[stack]
environment:
  L=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
    docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query: $..children[?(@.stat.pid in [$L])]
procfile_list: stat

# this section inherits some options, and overrides one of them
[stack:status:query]
extends: stack
sql_query: SELECT SUM(status_vmrss) total FROM record
procfile_list: stat,status

[stack:stat:query]
extends: stack
sql_query: SELECT SUM(stat_rss) * 4 total FROM record

Here’s how playbooks are read and interpreted:

A CLI minus-separated argument is written as an underscore-separated option.
The option value delimiter is :. A comment is prefixed with #.
A multi-value option is written one per line. A long line can be broken up by placing a backslash before the newline.
A section name can be compound. Its segments are delimited by :. If a section represents a command, its last segment must be the command’s name.
A section inherits from other sections via extends option.
Single-value option search stops at the first match, going from the command section up.
A multi-value option is joined across the section’s and its parent sections’ values.

A playbook can be saved as a .procpath file and run like:

procpath play -f example.procpath '*:query'

For the playbook CLI, see the listing of play command.

Advanced usage¶

CLI option override¶

Setting and/or overriding options via CLI:

[python:record]
environment:
  PIDS=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
       docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query: $..children[?(@.stat.pid in [$PIDS] and 'python' in @.stat.comm)]
interval: 10
recnum: 30

[python:plot]
query_name:
  cpu
  rss

database_file is required for both record and plot. It can be set via CLI like the following. Hence this records the database and creates CPU vs RSS plot out of it:

procpath play -f demo.procpath -o 'database_file=db.sqlite' '*'

Escalated privileges¶

Running playbook with escalated privileges:

[python:watch]
environment:
  DT=date +"%Y%m%dT%H%M%S"
  STACK=docker ps -f status=running -f name='^stack_name' -q | xargs -I{} -- \
        docker inspect -f '{{.State.Pid}}' {} | tr '\n' ,
query:
  PIDS=$..children[?(@.stat.pid in [$STACK] and 'python' in @.stat.comm)]..pid
interval: 10
repeat: 30
command:
  procpath record -i 1 -d db_$DT.sqlite \
    '$..children[?(@.stat.pid in [$PIDS])]'
  echo $PIDS | tr ',' '\n' | xargs -P0 -I{} -- \
    py-spy record --idle --pid {} -o py_{}_$DT.svg

py-spy typically requires escalated privileges to access the target Python process’ memory. xargs -P0 can be used to spawn py-spy per PID, because py-spy doesn’t support multiple targets natively. A playbook running py-spy with sudo can be run like the following:

sudo env "PATH=$PATH" procpath play -f demo.procpath python:watch

Alternatively, sudo ln -s ~/.local/bin/procpath /usr/local/bin/procpath so root under sudo can run it by default.

Target life-cycle¶

A playbook can cover full target process measurement life-cycle i.e. start the process of interest, run procpath record against it, and automatically stop with the target process:

[watch]
environment:
  DT=date +"%Y%m%dT%H%M%S"
interval: 1
no_restart: 1
command:
  xz -9 /some/big/database.sqlite
  procpath record -i 0.1 -f stat -d xz_$DT.sqlite --stop-without-result \
    -p $WPS1 "$..children[?(@.stat.pid == $WPS1)]"

A similar approach can be used when the target process is run from a detached Docker container.

[redoc:watch]
interval: 1
no_restart: 1
command:
  rm -f redoc_build.sqlite
  docker run --rm -d -v $PWD:/tmp/build --name=redoc \
    ghcr.io/redocly/redoc/cli:v2.0.0-rc.76 \
    build -o /tmp/redoc.html /tmp/build/openapi.json

[redoc:record]
environment:
  ROOT=docker inspect -f '{{.State.Pid}}' redoc
database: redoc_build.sqlite
query: $..children[?(@.stat.pid in [$ROOT])]
stop_without_result: true
interval: 0.025

[redoc:plot]
database_file: redoc_build.sqlite
plot_file: redoc_rss.svg
query_name:
  rss

Known limitations¶

There is a known limitation for item 3 regarding the use of multiline strings with escaped newlines. Use an empty string to end such string, like this:

[test:watch]
interval: 1
no_restart: 1
command:
  docker run --rm -e TEST=42 debian:bullseye \
    bash -c "\
      echo 'Debian container'; \
      cat /etc/os-release; \
      echo \$TEST; \
    "\
    ""

Procpath

Navigation

Related Topics