Pipe: Infix syntax for Python

Pipe is a Python module enabling infix syntax in Python.
For those asking “Why ?” let’s take an example :

Compare the readability of the classical prefix syntax :

sum(select(where(take_while(fib(), lambda x: x < 1000000) lambda x: x % 2), lambda x: x * x))

And the infix syntax :

fib() | take_while(lambda x: x < 1000000) \
      | where(lambda x: x % 2) \
      | select(lambda x: x * x) \
      | sum()

Isn’t the infix syntax more readable ?

The base class of Pipe is kept simple (7 lines of python) and is usable as a decorator permitting you to create new ‘pipeable’ functions easily. The module provides ~30 prepared pipes functions like ‘where’, ‘group_by’, ‘sort’, ‘take_while’ … A pipeable function takes an iterable (tuple, list, generator) and yields to be itself an iterator, so pipeable function can be piped together.

Let me introduce the basic usage of the Pipe module, then I’ll write some bits on how to build new ones :

To start, get it from PyPI http://pypi.python.org/pypi/pipe/1.3 and install it, open a REPL, import pipe, and play :

Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pipe import *
>>> [1, 2, 3, 4, 5] | add
>>> [5, 4, 3, 2, 1] | sort
[1, 2, 3, 4, 5]

Until here it’s easy, to know more about available pipes, just read the help(pipe) in the REPL, all are explained with an example as a doctest

Now as we know that pipeable functions use iterables, we can try to pipe together two or more pipeables :

>>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | concat
'1, 3, 5'
>>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | tail(2) | concat
'3, 5'
>>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | tail(2) | select(lambda x: x * x) | concat
'9, 25'
>>> [1, 2, 3, 4, 5] | where(lambda x: x % 2) | tail(2) | select(lambda x: x * x) | add

Now, a bit about lazyness, as Pipe use iterables, the evaluation of a whole Pipe is lazy, so we can play with infinite generators like this one :

>>> def fib():
...    a, b = 0, 1
...    while True:
...        yield a
...        a, b = b, a + b

Now we can do every kind of stuff into the fibonacci sequence, like solving the 2nd problem of http://projecteuler.net in a readable one liner :

Find the sum of all the even-valued terms in Fibonacci which do not exceed four million.

>>> euler2 = fib() | where(lambda x: x % 2 == 0) | take_while(lambda x: x < 4000000) | add
>>> assert euler2 == 4613732

Isn it pretty ?

Let now see how to create new pipeable functions using the @pipe decorator :
You want to create a function that yields the first x elements from its input
You want its usage to be (1, 2, 3, 4, 5) | take(2) to take the fist 2 elements.
I know that you are thinking about a basic implementation like :

def take(iterable, qte):
    for item in iterable:
        if qte > 0:
            qte -= 1
            yield item

Right ? You take an iterable, a qantity, and while the quantity is not reached, you just have to yield ?
OK, just add @pipe to you take function and it’s pipeable :-)

This entry was posted in Code, Python. Bookmark the permalink.

42 Responses to Pipe: Infix syntax for Python

  1. Matt Pettis says:

    This is a great idea, and is one of the niceties of Haskell (where they use ‘.’ for ‘|’, and I think you even give an implicit not to it with the ‘take_from’ function). I’ve just been reading up on some other languages, and just came across this in Haskell, and was wondering if this concept would be used in another language where it wasn’t primary, and now I have this serendipitous post!

  2. Matt Pettis says:

    Ooops, I should add that Haskell ‘.’ does function composition, vs. pipe, which is, err, piping. The difference being that in Haskell you’d read inner to outer as right-to-left, where here, piping makes it a more natural left-to-right reading. I got thrown by the ‘take_from’ function and immediately went ‘Aha! Haskell!’

  3. Nick Coghlan says:

    For a fairer comparison, though, the “where” and “select” clauses should be rolled into a generator expression in the non-infix version:

    from itertools import takewhile
    x = sum(takewhile(lambda x: x < 10000, (x*x for x in fib() if x % 2))

    Which would become:

    x = (x*x for x in fib() if x % 2) | take_while(lambda x: x < 1000) | sum()

    I'd also compare it with a version that names the intermediate steps:

    all_elements = (x*x for x in fib() if x % 2)
    selected = takewhile(lambda x: x < 10000, all_elements)
    x = sum(selected)

    In general, quite an interesting approach – certainly makes iterator pipelines more decomposable within a single expression.

    If you wanted to explore a more "real world" example, I suggest looking through http://www.dabeaz.com/generators/ and seeing how the examples in that presentation would change if using the pipeline model of iterator composition.

    (Note: your PyPI metadata has a typo in the URLs – it should be "pipe", not "Pype").
    (In looking for the presentation linked above, I came across another project similar to yours: http://code.google.com/p/python-pipeline/)

  4. Julien Palard says:

    Thanks :)
    My inspirations was sh, LINQ, and my needs solving project euler problems, I have a friend writing in Haskell and took a look for this language but never tried to write a single line :p


  5. Bernardo says:

    Amazing and simple!

    The function call returns another pipe object, just like that.

  6. I was looking forward to play with pipe (I’m a sh fan and miss the expressiveness of pipes in python) but I failed to install with easy_insall.

    panzani pipe$ easy_install pipe
    Searching for pipe
    Reading http://pypi.python.org/simple/pipe/
    Reading https://github.com/JulienPalard/Pipe
    Reading https://github.com/JulienPalard/Pipe/tarball/master
    No local packages or download links found for pipe
    Best match: None
    Traceback (most recent call last):

    … then a looooong traceback. Any ideas ?

  7. Oops i was missing the upload in setup.py, for developers who found this page googling for ‘No local packages or download links found’ try :
    ./setup.py register sdist upload -r pypi

  8. Interesting…

    I just miss tee…(from bash). Could you add it ?

  9. BOb says:

    slight typo in article: readeable ->readable
    (no need to publish this comment!)

  10. Frank Denis says:

    If you’re looking for something similar for Javascript, take a look at Kaffeine: http://weepy.github.com/kaffeine/

  11. Lionel : I just added tee, pushed it to github && PyPI :-)

  12. Ilya Kasnacheev says:

    In Haskell, $ does piping and it’s my favourite combinator.

  13. halida says:

    it is a great tool!
    clear and pythonic!
    maybe one day it will be a default module in python?

  14. I have built something similar, coming at the problem from a different direction. I wanted a Unix-style shell, but piping objects instead of strings. For example, to sum x^2 from the command line, x = 0-9:

    bash$ osh gen 10 ^ f ‘x: x**2’ ^ red + $

    ^ is the pipe symbol, red means “reduce”, $ means output, and f means apply the given function.

    The same capabilities are available as a Python library:

    from osh.api import *
    osh(gen(10), f(lambda x: x**2), red(lambda x, y: x + y), out())


    from osh.api import *
    osh(gen(10), f(lambda x: x**2), red(‘+’), out())

    Osh can also run results on multiple hosts at once, in parallel, combining the results in various ways; and integrates database access, piping tuples to and from sql commands.


  15. Yeah osh seems like pipe, but I prefer limiting the use of useless parenthesis :

        from osh.api import *
        osh(gen(10), f(lambda x: x**2), red(lambda x, y: x + y), out())

    gives :

        from pipe import *
        xrange(10) | select(lambda x: x ** x) | aggregate(lambda x, y: x + y)

    And you can more easily indent it :

        from pipe import *
        xrange(10) | select(lambda x: x ** x)
                   | aggregate(lambda x, y: x + y)
  16. Fred says:

    I like the idea but I’m not sure that all those lambdas are very readable – also they are impacting the execution speed – each lambda is a function call.

    Take a look at http://ipython.scipy.org/moin/UsingIPipe

  17. Julien, what happens in your example with xrange(0)? Does aggregate have an optional initial value? Otherwise I don’t see how aggregate can emit the right answer.

  18. Jack, as expected, it throw: reduce() of empty sequence with no initial value.
    So I just added the optional ‘initializer’ argument of ‘reduce’ to my ‘aggregate’ giving :

    xrange(0) | select(lambda x: x ** x)
              | aggregate(lambda x, y: x + y, initializer=0)
  19. Anthony says:

    You have to love LINQ and monads! This is Just Great!

  20. Great trick !
    I didnt know this…

  21. mf says:

    For the people offering Haskell as a solution: It’s not quite the (.) operator, which is function composition, nor the ($) operator, which is application. It’s much closer to ($) though; in fact, it’s the reverse of it. Given that, writing a `pipe’ operator isn’t terribly difficult. But, the pipe character is reserved so we’ll use .|. instead.

    (.|.) = flip ($)

    This can then be used as described in the blog post

    &gt; [5, 4, 4, 3, 4, 2] .|. sort .|. take 5
  22. Fred Jones says:

    Keep working on this great Python syntax, but steal ideas from many years of CMS Pipelines research and development.


    from John Hartmann

    tons more info

    Search for cms pipelines windows for various ports of it.

  23. Fred Jones: I only heard about http://en.wikipedia.org/wiki/Douglas_McIlroy , however documents about CMS Pipelines seems interesting, I’ll try to read them (if I found some free time) :-)

  24. CJ says:

    I’ve been playing with a similar idea for a while: https://github.com/perimosocordiae/plumbum

    It’s certainly a compelling style.

  25. Robert says:

    Why not call it “infix”? Maybe a whole class of Haskell type functional ideas can be put together and eventually added to core as “functools” or something to make functional programming in Python better?

  26. Robert says:

    Ooops…already is a functools. lol

  27. Alex says:

    Seems close to one of my experiments, see http://honeyman.livejournal.com/122675.html (on Russian and Python, but I believe there is more Python there than Russian). Except in my case, every predicate could’ve been used, as long as the chain/pipe started with a special call.

  28. Alain says:

    Your ‘tee’ function is wrongly named. You should call it ‘trace’.
    In the same vein, ‘select’ should be named ‘apply’

  29. Iowa Hansen says:

    Cool library! Is there something similar for Ruby?

  30. Andrey Tarantsov says:

    @Iowa Hansen

    Yes, there is something similar for Ruby — just define methods on Enumerable and use . instead of | :)

  31. Dave Riggs says:

    Since the expected use is `from pipe import *`, would you please consider using `__all__` so your imports don’t pollute users’ namespaces?

    Cool project, makes for very expressive code!

  32. Andreas says:

    I like that! I have only two issues:

    – What about debugging? Is there any support imaginable that uncovers the piped values?

    – How efficient is piping compared to not piping? Any numbers on that?

  33. John: I merged your commit, thanks :)
    Andreas: For debugging purpose, you can insert a ‘tee’ in your pipe to dump every item while passed to standard output.

    I don’t ran any benchmark to compare a code using pipe and a code don’t using it cause I wrote it for readability, (for performance, I write in C http://www.youtube.com/watch?v=1S1fISh-pag :-p )

    Dave Riggs: Thanks to the commit of John, __all__ is now present :)

    Alain: I called my tee like the unix command, i’ll meditate on the best name to choose :)
    Alain: For select, i’ll keep using select as the standard use is not to apply a lambda to each items but to select some items from the input, typically :

    >>> employees | select(lambda employee: employee.salary) | average
  34. Pingback: Pipes y sintaxis infija en Python | gr3p

  35. Wai Yip Tung says:

    In the second code example fib(), don’t you need to add the Python line continuation character \ in order to spread the expression on multiple lines?

  36. Wai Yip Tung: You’re right, I fixed it. I missed the \ cause I was writing more a sample than pure python ^^ but it have to be right to not confuse people so it fixed now, thanks :)

  37. Pingback: Python: Introducing ppipe : Parallel Pipe | {Dev Tricks}

  38. Pingback: Pipes in Python – Infix Syntax for Function Calls « Blog 4

  39. Remarkable issues here. I am very glad to see your post. Thank you so much and I’m taking a look ahead to contact you. Will you kindly drop me a e-mail?

  40. どのようにそれだけで誰もがブログを書いて、このように人気を得ることができるのですか?そのyouveはもっとホードような非常に印象的なものは、あなたがについて何も知らない問題に対してかなり絵を描いたと述べた好きではありません!私はここで、平均音にしたいいけません。しかし、あなたは本当にあなたには、いくつかのかなりの写真を追加して逃げると真剣に何かを言うことができないことだと思いますか?ちょっと人

Leave a Reply

Your email address will not be published. Required fields are marked *