Easily run Python at the shell! Magical, but never mysterious.
Easily run Python at the shell! Magical, but never mysterious.
Run pip install pypyp
(note the extra "yp"!)
pyp requires Python 3.6 or above.
pyp will statically analyse the input code to detect undefined variables. Based on what it finds,
it will proceed to transform the AST of the input code as needed. We then compile and execute the
result, or if using --explain
, unparse the AST back to source code.
This section will walk you through the details of using pyp, and hopefully replace your needs
for many common shell utilities. For a cheatsheet / tldr, run pyp --help
.
Just use one of the magic variables x
, l
or line
to refer to the current line.
# pyp like cut
ls | pyp 'x[:3]'
ps x | pyp 'line.split()[4]'
Use the magic variable lines
for a list of rstripped lines or stdin
for sys.stdin
.
# pyp like wc -c
cat /usr/share/dict/words | pyp 'len(stdin.read())'
# pyp like awk
seq 1 5 | pyp 'sum(map(int, lines))'
# pyp like sh
echo echo echo | pyp 'subprocess.run(lines[0], shell=True); pass'
# pyp like jq
curl -s 'https://api.github.com/repos/hauntsaninja/pyp/commits?per_page=1' | pyp 'json.load(stdin)[0]["commit"]["author"]'
# pyp like egrep
cat /usr/share/dict/words | pyp 'x if re.search("(p|m)yth", x) else None'
For collections
, math
, itertools
, pathlib.Path
, pprint.pp
, pyp will figure it out even
if you don't use the qualified name.
# pyp like bc
pyp 'sqrt(5)'
# pyp like ${x##*.}
ls | pyp 'Path(x).suffix'
i
, idx
or index
.# pyp like line numbers
cat setup.py | pyp 'f"{idx+1: >3} {x}"'
print
!By default, pyp will print the last expression in your code — except if it evaluates to None
(or the last statement is pass
).
And you can always explicitly call print
yourself, in which case pyp will stay out of your way.
# pyp like grep
cat /usr/share/dict/words | pyp 'x if "python" in x else None'
cat /usr/share/dict/words | pyp 'if "python" in x: print(x); "this will not be printed"'
This makes the output of pyp easier to compose with shell tools.
Again, explicit printing will stop this magic, but pyp makes the function pypprint
available if
you do want to explicitly opt back in.
# pyp like tail
ls | pyp 'lines[-10:]'
# pyp like sort
ls | pyp 'sorted(lines)'
ls | pyp 'print(f"Sorting {len(lines)} lines"); pypprint(sorted(lines))'
# pyp like sort | uniq
ls | pyp 'sorted(set(lines))'
Note if you run into trouble with semicolons and want a new line (without using a multiline string in your shell), you can just pass another string to pyp. You can also always pipe pyp to pyp!
# pyp like anything!
ps aux | pyp -b 'd = defaultdict(list)' 'user, pid, *_ = x.split()' 'd[user].append(pid)' -a 'del d["root"]' -a 'd'
Use --explain
or --script
and pyp will output a script equivalent to what it would run. This can also serve as a
useful starting point for more complex scripts.
pyp --explain -b 'd = defaultdict(list)' 'user, pid, *_ = x.split()' 'd[user].append(pid)' -a 'del d["root"]' -a 'd'
#!/usr/bin/env python3
from collections import defaultdict
from pyp import pypprint
import sys
d = defaultdict(list)
for x in sys.stdin:
x = x.rstrip('\n')
(user, pid, *_) = x.split()
d[user].append(pid)
del d['root']
if d is not None:
pypprint(d)
And if your command hits an exception, pyp will reconstruct a traceback into the generated code.
Point the environment variable PYP_CONFIG_PATH
to a file containing, for example:
import numpy as np
import tensorflow as tf
from pipetools import *
def p95(data):
return np.percentile(data, 95)
class PotentiallyUsefulClass: ...
When attempting to define undefined names, pyp will statically* analyse this file as a source of
possible definitions. This means that if you don't use tf
, we won't import tensorflow
! And of
course, --explain
will show you exactly what gets run (and hence what doesn't!):
pyp --explain 'print(p95(list(map(float, stdin))))'
#!/usr/bin/env python3
import sys
import numpy as np
def p95(data):
return np.percentile(data, 95)
stdin = sys.stdin
print(p95(list(map(float, stdin))))
Note, importing things from libraries like pipetools in your configuration can allow you to achieve high levels of syntax sugar:
seq 1 110 | pyp 'lines > foreach(int) | where(X > 100) | group_by(X % 3) | sort_by(X[0])'
*If you use wildcard imports, we will need to import those modules if there remain undefined
names, though we skip this in the happy path. If this matters to you, definitely don't
from tensorflow import *
in your config!
If definitions in your config file depend on magic variables, pyp will substitute them in the way that makes sense. For example, put the following in your config...
n = int(x)
f = x.split()
j = json.load(stdin)
import pandas as pd
csv = pd.read_csv(stdin)
...to make pyp easier than ever for your custom use cases:
ps | pyp 'f[3]'
cat commits.json | pyp 'j[0]["commit"]["author"]'
< cities.csv pyp 'csv.to_string()'
There's additional documentation and examples at FAQ. If that doesn't answer your question, please open an issue!
pyp takes inspiration (and the command name!) from here. However, Pyed Piper appears to be unmaintained, Python 2 only, and further away from Python syntax than pyp aims to be. Github mirror here.
I discovered Pyped while making this project! It's actually very similar, probably similar enough that I wouldn't have written this had I known. However, Pyped doesn't do the AST introspection and manipulation that we do. This means:
--explain
/--script
.However,
Since writing pyp, it turns out there are more alternatives out there than I thought :-) Some quick notes:
--explain
.For whatever it's worth, I've listed these projects in approximate order of my personal preference.
mario
is a featureful take on shell processing with Python. It doesn't use undefined name
detection, instead relying on a pluggable subcommand system. While the subcommands can be more
verbose than pyp, mario
makes up some ground by automatic application of functions and a custom
command chaining syntax. The result can feel a little DSL-like, while pyp tries to feel very close
to writing Python.
Consider using mario
if:
Consider pyp if:
--script
to deal
with complexity.xonsh
is a shell whose language is a superset of Python; this is more ambitious and pretty
different from pyp. pyp is easier to use for the one-liner piping use case, but if you need
more Python in your shell, check out xonsh
.
If awk
works for you, how did you end up here?