Pipe is a Python module enabling infix syntax in Python.
For those asking “Why ?” let’s take an example :
Compare the readability of the classical prefix syntax :
sum(select(where(take_while(fib(), lambda x: x < 1000000) lambda x: x % 2), lambda x: x * x))
And the infix syntax :
fib() | take_while(lambda x: x < 1000000) \ | where(lambda x: x % 2) \ | select(lambda x: x * x) \ | sum()
Isn’t the infix syntax more readable ?
The base class of Pipe is kept simple (7 lines of python) and is usable as a decorator permitting you to create new ‘pipeable’ functions easily. The module provides ~30 prepared pipes functions like ‘where’, ‘group_by’, ‘sort’, ‘take_while’ … A pipeable function takes an iterable (tuple, list, generator) and yields to be itself an iterator, so pipeable function can be piped together.
Let me introduce the basic usage of the Pipe module, then I’ll write some bits on how to build new ones :
To start, get it from PyPI http://pypi.python.org/pypi/pipe/1.3 and install it, open a REPL, import pipe, and play :
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from pipe import * >>> [1, 2, 3, 4, 5] | add 15 >>> [5, 4, 3, 2, 1] | sort [1, 2, 3, 4, 5]
Until here it’s easy, to know more about available pipes, just read the help(pipe) in the REPL, all are explained with an example as a doctest
Now as we know that pipeable functions use iterables, we can try to pipe together two or more pipeables :
>>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | concat '1, 3, 5' >>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | tail(2) | concat '3, 5' >>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | tail(2) | select(lambda x: x * x) | concat '9, 25' >>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | tail(2) | select(lambda x: x * x) | add 34
Now, a bit about lazyness, as Pipe use iterables, the evaluation of a whole Pipe is lazy, so we can play with infinite generators like this one :
>>> def fib(): ... a, b = 0, 1 ... while True: ... yield a ... a, b = b, a + b
Now we can do every kind of stuff into the fibonacci sequence, like solving the 2nd problem of http://projecteuler.net in a readable one liner :
Find the sum of all the even-valued terms in Fibonacci which do not exceed four million.
>>> euler2 = fib() | where(lambda x: x % 2 == 0) | take_while(lambda x: x < 4000000) | add >>> assert euler2 == 4613732
Isn it pretty ?
Let now see how to create new pipeable functions using the @pipe decorator :
You want to create a function that yields the first x elements from its input
You want its usage to be (1, 2, 3, 4, 5) | take(2) to take the fist 2 elements.
I know that you are thinking about a basic implementation like :
def take(iterable, qte): for item in iterable: if qte > 0: qte -= 1 yield item else: return
Right ? You take an iterable, a qantity, and while the quantity is not reached, you just have to yield ?
OK, just add @pipe to you take function and it’s pipeable :-)
This is a great idea, and is one of the niceties of Haskell (where they use ‘.’ for ‘|’, and I think you even give an implicit not to it with the ‘take_from’ function). I’ve just been reading up on some other languages, and just came across this in Haskell, and was wondering if this concept would be used in another language where it wasn’t primary, and now I have this serendipitous post!
Ooops, I should add that Haskell ‘.’ does function composition, vs. pipe, which is, err, piping. The difference being that in Haskell you’d read inner to outer as right-to-left, where here, piping makes it a more natural left-to-right reading. I got thrown by the ‘take_from’ function and immediately went ‘Aha! Haskell!’
For a fairer comparison, though, the “where” and “select” clauses should be rolled into a generator expression in the non-infix version:
from itertools import takewhile
x = sum(takewhile(lambda x: x < 10000, (x*x for x in fib() if x % 2))
Which would become:
x = (x*x for x in fib() if x % 2) | take_while(lambda x: x < 1000) | sum()
I'd also compare it with a version that names the intermediate steps:
all_elements = (x*x for x in fib() if x % 2)
selected = takewhile(lambda x: x < 10000, all_elements)
x = sum(selected)
In general, quite an interesting approach – certainly makes iterator pipelines more decomposable within a single expression.
If you wanted to explore a more "real world" example, I suggest looking through http://www.dabeaz.com/generators/ and seeing how the examples in that presentation would change if using the pipeline model of iterator composition.
(Note: your PyPI metadata has a typo in the URLs – it should be "pipe", not "Pype").
(In looking for the presentation linked above, I came across another project similar to yours: http://code.google.com/p/python-pipeline/)
Thanks :)
My inspirations was sh, LINQ, and my needs solving project euler problems, I have a friend writing in Haskell and took a look for this language but never tried to write a single line :p
Bests
Amazing and simple!
The function call returns another pipe object, just like that.
I was looking forward to play with pipe (I’m a sh fan and miss the expressiveness of pipes in python) but I failed to install with easy_insall.
panzani pipe$ easy_install pipe
Searching for pipe
Reading http://pypi.python.org/simple/pipe/
Reading https://github.com/JulienPalard/Pipe
Reading https://github.com/JulienPalard/Pipe/tarball/master
No local packages or download links found for pipe
Best match: None
Traceback (most recent call last):
… then a looooong traceback. Any ideas ?
Oops i was missing the upload in setup.py, for developers who found this page googling for ‘No local packages or download links found’ try :
./setup.py register sdist upload -r pypi
Interesting…
I just miss tee…(from bash). Could you add it ?
slight typo in article: readeable ->readable
(no need to publish this comment!)
If you’re looking for something similar for Javascript, take a look at Kaffeine: http://weepy.github.com/kaffeine/
Lionel : I just added tee, pushed it to github && PyPI :-)
In Haskell, $ does piping and it’s my favourite combinator.
it is a great tool!
clear and pythonic!
maybe one day it will be a default module in python?
I have built something similar, coming at the problem from a different direction. I wanted a Unix-style shell, but piping objects instead of strings. For example, to sum x^2 from the command line, x = 0-9:
^ is the pipe symbol, red means “reduce”, $ means output, and f means apply the given function.
The same capabilities are available as a Python library:
or
Osh can also run results on multiple hosts at once, in parallel, combining the results in various ways; and integrates database access, piping tuples to and from sql commands.
http://geophile.com/osh
Yeah osh seems like pipe, but I prefer limiting the use of useless parenthesis :
gives :
And you can more easily indent it :
I like the idea but I’m not sure that all those lambdas are very readable – also they are impacting the execution speed – each lambda is a function call.
Take a look at http://ipython.scipy.org/moin/UsingIPipe
Julien, what happens in your example with xrange(0)? Does aggregate have an optional initial value? Otherwise I don’t see how aggregate can emit the right answer.
Jack, as expected, it throw: reduce() of empty sequence with no initial value.
So I just added the optional ‘initializer’ argument of ‘reduce’ to my ‘aggregate’ giving :
You have to love LINQ and monads! This is Just Great!
Great trick !
I didnt know this…
Tks
For the people offering Haskell as a solution: It’s not quite the (.) operator, which is function composition, nor the ($) operator, which is application. It’s much closer to ($) though; in fact, it’s the reverse of it. Given that, writing a `pipe’ operator isn’t terribly difficult. But, the pipe character is reserved so we’ll use .|. instead.
This can then be used as described in the blog post
Keep working on this great Python syntax, but steal ideas from many years of CMS Pipelines research and development.
Wikipedia
from John Hartmann
tons more info
Search for cms pipelines windows for various ports of it.
Fred Jones: I only heard about http://en.wikipedia.org/wiki/Douglas_McIlroy , however documents about CMS Pipelines seems interesting, I’ll try to read them (if I found some free time) :-)
I’ve been playing with a similar idea for a while: https://github.com/perimosocordiae/plumbum
It’s certainly a compelling style.
Why not call it “infix”? Maybe a whole class of Haskell type functional ideas can be put together and eventually added to core as “functools” or something to make functional programming in Python better?
Ooops…already is a functools. lol
Seems close to one of my experiments, see http://honeyman.livejournal.com/122675.html (on Russian and Python, but I believe there is more Python there than Russian). Except in my case, every predicate could’ve been used, as long as the chain/pipe started with a special call.
Your ‘tee’ function is wrongly named. You should call it ‘trace’.
In the same vein, ‘select’ should be named ‘apply’
Cool library! Is there something similar for Ruby?
@Iowa Hansen
Yes, there is something similar for Ruby — just define methods on Enumerable and use . instead of | :)
Since the expected use is `from pipe import *`, would you please consider using `__all__` so your imports don’t pollute users’ namespaces?
Cool project, makes for very expressive code!
I added support for __all__:
https://github.com/mrjbq7/Pipe/commit/e52df7d0764db6651f5b9879ff86d081ff8575b5
I like that! I have only two issues:
- What about debugging? Is there any support imaginable that uncovers the piped values?
- How efficient is piping compared to not piping? Any numbers on that?
John: I merged your commit, thanks :)
Andreas: For debugging purpose, you can insert a ‘tee’ in your pipe to dump every item while passed to standard output.
I don’t ran any benchmark to compare a code using pipe and a code don’t using it cause I wrote it for readability, (for performance, I write in C http://www.youtube.com/watch?v=1S1fISh-pag :-p )
Dave Riggs: Thanks to the commit of John, __all__ is now present :)
Alain: I called my tee like the unix command, i’ll meditate on the best name to choose :)
Alain: For select, i’ll keep using select as the standard use is not to apply a lambda to each items but to select some items from the input, typically :
Pingback: Pipes y sintaxis infija en Python | gr3p
In the second code example fib(), don’t you need to add the Python line continuation character \ in order to spread the expression on multiple lines?
Wai Yip Tung: You’re right, I fixed it. I missed the \ cause I was writing more a sample than pure python ^^ but it have to be right to not confuse people so it fixed now, thanks :)
Pingback: Python: Introducing ppipe : Parallel Pipe | {Dev Tricks}
Pingback: Pipes in Python – Infix Syntax for Function Calls « Blog 4
Remarkable issues here. I am very glad to see your post. Thank you so much and I’m taking a look ahead to contact you. Will you kindly drop me a e-mail?
My email is very easy, it’s myfirstname@mylastname.fr ;-)