DISQUS

Asktav: Ruby-style Blocks in Python

  • Guido van Rossum · 9 months ago
    Do not reuse the 'with' keyword for this. It already has a well-defined meaning that is completely different from what you are proposing. In particular turning the block into an anonymous function is a dramatic difference with the existing with-statement; it introduces a new scope which means that assignments to locals of the immediately surrounding scope don't work unless you add a 'nonlocal' statement for each one you plan to assign to. If you can't solve that, your proposal is dead.
  • tav · 9 months ago
    I'll try to find a better replacement for ``with``...

    Suggestions from everyone very welcome!
  • Muhammad Alkarouri · 9 months ago
    Thanks tav. This is a great idea, and we need to have more of these:)
    I suggest the word "on" instead of "with" for the moment.

    on employees.select do (emp):
    if emp.salary > developer.salary:
    return fireEmployee(emp)
    else:
    return extendContract(emp)
  • Todd Lucas · 9 months ago
    You could borrow C#'s using keyword.
  • Bill · 9 months ago
    But only for a week, then we want it back.


    --Bill
  • tav · 9 months ago
    Thanks Todd! That fits the bill quite nicely -- despite being slightly longer to type...

    Will update this proposal now.
  • Reid K · 9 months ago
    Why not hijack the lambda keyword? You're basically trying to unbreak lambdas anyway, and the name does suggest that yes, you are making a real function with real scope.
  • Verte · 9 months ago
    How about 'let' or 'let .. in' ? we could even have letrec.
  • Demur Rumed · 9 months ago
    It would be nice, though I'm unsure of the mixed use of tuples. What if the argument is suppose to be a tuple? Perhaps it'd be better if the system was purely syntactic sugar that was indistinguishable from writing a throwaway function and passing it to the expression
  • Demur Rumed · 9 months ago
    Though it just occured to me, how much of this can be done with decorators? The function definition would just become the return value of the function call, which your solution doesn't allow for
  • tav · 9 months ago
    You have a point with the mixed use of tuples -- but unfortunately the syntax of yield expressions works against us on that =(

    As for a decorator, yes, we could do a do() decorator, I could code one up if it would help make the case better...
  • Tom Ritchford · 9 months ago
    What's the advantage over simply creating a new, local function?
  • tav · 9 months ago
    Tom, it simply makes life easier.

    I believe it could be similarly popular in the same way that @decorators have been tremendously popular despite just saving a few bytes.

    Ruby has already proven the merits of a block syntax -- check out Rake and Rails for examples of how blocks are used to increase productivity!
  • thebitguru · 9 months ago
    Partly that, you don't have to create a new local function. I am not sure if this approach is any faster than defining a function.
  • tav · 9 months ago
    @thebitguru this wouldn't be faster in terms of performance for the default case but will save the explicit creation of the local function as you point out
  • Steven D'Aprano · 9 months ago
    I question the basic assumption of anonymous blocks. I'm not convinced that they're a good idea, even if they are possible, even if the syntax is "sexy" (your term, not mine). More on syntax later.

    Lambda functions are useful simply because they are *short*: short enough to take in at a glance, and short enough not to require documentation, or testing. By allowing multi-line lambdas, you encourage people to write complex, long functions that can't have doc strings and can't (easily? or at all?) be tested. Use-once-and-throw-away code blocks discourage code reuse and DRY. These criticisms also apply to lambda, of course, but with lambda the harm is minimised because the functions are small and easily understood. That anonymous blocks are very popular among Ruby programmers doesn't mean much to me. GOTOs were hugely popular among Basic programmers too, and look at how many people use XML for supposedly human-editable config files.
  • tav · 9 months ago
    Steven, you have a point with GOTOs and XML-as-config... *shudder*

    But Ruby has shown that anonymous blocks can be used to improve the elegance of the code as well as increase the productivity of the developer.

    As for testing, I don't see why such an approach precludes either testing or documentation.

    In fact, it could make testing more pleasant than the java-inspired unittest framework we currently have.

    See this http://blog.excastle.com/2005/05/18/ruby-coolne... for

    CheckThrows(EInvalidOperation) do
    DoSomething
    end

    I can even see interesting ways that __do__ could be used to create nice doctests.
  • Steven D'Aprano · 9 months ago
    Tav, you've missed my point. I didn't say that anonymous code blocks can't be used *for* testing. But how do you test the code block? The page you link to doesn't say anything about that. A one line lambda should be simple enough to just see what it does and that it does it correctly, but even a three line lambda should be treated as a function needing testing.

    If your lambda is complicated enough to need testing, and for some reason you don't want to (or can't) move it out to the top level into a def, it's still a first-class object so you can do this:

    func = lambda x,y,z: something(x)+something(y)+another(z)
    if testing: test(func)
    use(func)

    Unless I've misunderstood something, your multi-line lambdas are second-class functions, so the only way to do the same would be:

    if testing: with test do (x,y,z): something(x)+something(y)+another(z)
    with use do (x,y,z): something(x)+something(y)+another(z)

    You have to repeat yourself twice, which leads to copy-and-paste programming, code duplication, and other Bad Things.
  • hugo · 9 months ago
    Why do you request specific testing of the anonymous block - how would you test a single if? or a single while? The anonymous block is just a syntactic element, like while, for etc. There is no need to request specific documentation or testing needs for it that goes beyond what you would ask of other syntactic elements.

    One thing I _do_ see as problematic is something else: functions are first-class objects in Python, but this "syntactic sugar" would remove the first-class-ness of that anonymous block in a way. If you do define the local function, you can pass it along just as any other value, while with the suggestion it wouldn't. And that just for the sake of getting rid of a function definition. Not too sure about the value of that.

    Think about refactoring your code and discovering that it would be nice to pass the "anonymous block" function as argument - you would have to change the code quite drastically if you used the suggested "do" block, while with the classic approach, not much changes - you still have your local function and it's cl osure and just pass that on elswhere.
  • Steven D'APrano · 9 months ago
    Hugo, a function (anonymous or named) isn't "just a syntactic element" like while, if etc. It's a compound element taking arguments, with a defined entry point and a defined return value. A function is, or at least should be, coupled loosely to other functions while syntactic elements are coupled very strongly to other elements.

    An analogy might be that functions are molecules while single elements are atoms. The semantics of the atom "if x > y: x += y" depends on the other atoms around it, while ideally the function sin() is always the trigonometric sine no matter what other functions are around it. So you should be able to test sin() in isolation, which you can't do with the if block.

    (Besides, you can perform limited, ad hoc testing of individual elements simply by sprinkling asserts after each element. Not a good solution, but it is something.)

    The whole purpose of functions (procedures or sub-routines) is to lessen the coupling between parts of code, to partially isolate chunks of code for ease of re-use and testing. If a code block is tightly coupled like a while or if, then it is a step backwards towards spaghetti code; and if it is loosely coupled, like a function, then there's no benefit to that looseness if you can't test it or re-use it.
  • Adam Olsen · 9 months ago
    I disagree on your last point. The biggest reason to use functions is to ease understanding. Most programs could be inlined into a single massive function, but the mental effort to understand it would increase by many times, effectively distracting you from the real work that's going on.

    Unit testing, although nice, falls out of this "unitizing". It's not the original motivation.

    (Although that may also harm performance, compilers could be written to recognize all the duplication and merge common bits. Functions are quite entrenched in common styles though, so there's little motivation to do this today.)
  • Alan · 9 months ago
    I'd like to hear more of your ideas with doctest, or on doctest
  • Steven D'Aprano · 9 months ago
    As for the syntax... I'm afraid that I find your syntax, and the Ruby syntax you base it on, unspeakably awful. You're suggesting a second syntax for calling a function, one that apparently only works if the argument is an anonymous code block. Calling a function is consistent in Python, it is always FUNCTION(ARG) no matter what ARG happens to be, but you're proposing a second syntax:

    with FUNCTION do (MAGIC):
    ARG

    where ARG is a code block, and MAGIC is actually part of the code block but appears outside of it. And what happens when you have multiple arguments to FUNCTION? How do you combine multi-line anonymous blocks with other arguments? Or do you make multi-line lambdas a second-class citizen by prohibiting them from being passed to functions of two or more arguments?

    In your post, you say "This is also possible in Python but at the needless cost of naming and defining a function first". I dispute that it is "needless cost". The programmer's effort in creating the code block, versus the effort in creating a function, is precisely the same except for a single line: "def throwaway_function(emp)" in your example. Your proposed implementation also creates a function object. So there's unlikely to be any performance boost at compile time, and probably a performance cost at runtime due to the extra redirection with your __do__ function.

    As far as I can see, the only advantages for anonymous blocks are that the programmer doesn't have to think of a name, and that you define the function where you need it instead of immediately prior to where you need it. Both are tiny advantages, as far as I can see. I'm not convinced that multi-line lambdas are needed at all.
  • Anonymous · 9 months ago
    Here's another way to call a function:

    @func
    def foo(): ...

    (which is:)

    def foo(): ...
    foo = func(foo)

    So no it is not unique.

    Also, code flows better this way, especially with e.g. callbacks.
  • sjs · 9 months ago
    shenpen on reddit posted this link, it might be of interest to you: http://code.activestate.com/recipes/534150/
  • tav · 9 months ago
    Thanks sjs, perhaps an accompanying decorator would help too?
  • Joeri · 9 months ago
    Hey, could you please remove ':' from 'else: ' i one of the following Ruby snippet?
  • tav · 9 months ago
    Thanks Joeri -- fixed.
  • Tony Arcieri · 9 months ago
    The main way Ruby is different compared to this is function calls with blocks are expressions, so you can do things like:

    sorted_stuff = stuff.sort_by do |s|
    s.name
    end

    do_something_with(sorted_stuff)

    or

    useful_infos = stuff.map do |x|
    x.useful_info.to_s
    end.join(', ')

    ...which isn't possible with blocks-as-statements.
  • JimJJewett · 9 months ago
    Naming the multi-line lambda shouldn't be that hard; just call it f1 if you insist on an empty name.

    I thought (wrongly?) that the real advantage of blocks was that they shared scope with the funciton where they're used, instead of with the location where they're defined.

    defop add(a, b):
    # Today, a and b would be undefined
    sum=a+b

    def caller():
    a=5
    b=6
    defop()
    # today, sum would still be undefined,
    # -- even if defop "worked"
    print(sum)
  • David Lynch · 9 months ago
    I think that, given Python's "There should be one-- and preferably only one --obvious way to do it." principle, you should add some examples for cases that couldn't be fairly easily done as for loops.

    (I actually really like anonymous functions -- I've done a lot of work in javascript and Lua, and they're very useful.)
  • anonymous coward · 9 months ago
    When reading your code the first 5 times, I was confused by "return" meaning "return from this block", not "return from this def", which it means everywhere else in Python (and also Ruby).

    I think for this to work well, you really need to first have Python return the last value of a block, like Ruby and Lisp do. (Or you could disambiguate by adding a RETURN-FROM form like Lisp has, but that I *really* can't imagine ever being adopted in Python!) Then you need to retrofit values to statements (like "if"), and that's a can of worms Guido doesn't seem likely to ever want to reopen.

    It also seems to go counter to Python's "one way to do things" mantra.

    I'm not saying it's a bad feature idea. (I'm a Lisp programmer and we're not even happy with Perl's 27 ways to do it. We need access to the machine shop so we can build a new language from scratch ourselves.) I just don't see it being very Pythonic on its own, or being very Pythonic with respect to other features, or resulting in more Pythonic code to be written. It's really more of a cultural change than a syntactic one.
  • Andrew Montalenti · 9 months ago
    You might want to look at PEP 359, The "make" statement. It was rejected, and though different than your proposal, I think it shared some goals. Worth studying:

    http://www.python.org/dev/peps/pep-0359/
  • Carl · 9 months ago
    As Guido notes, this goes completely against the existing semantics of with. Currently, with does not create a new namespace for the block that follows it. I find that sort of confusing, but there it is.

    As another commenter noted, this can be done with decorators. For example,

    @employees.select
    def emp (emp):
    if emp.salary > developer.salary:
    return fireEmployee(emp)
    else:
    return extendContract(emp)

    The downside of this is that "emp" ends up being assigned the result rather than a function, which is a little confusing.
  • Adam Olsen · 9 months ago
    If we fix anything it should be having a way to not assign the result of @foo to the given name. Perhaps by returning a singleton, or by an empty name.

    Besides that, injecting globals and locals is horrible. __do__ should take a single argument, a callback, that's a rather mundane function.
  • Carl · 9 months ago
    I once proposed that the @ sign be used to mean "pass in a function here, to be defined on the next line." It could be used like so:

    records = employee.select(@):
    def @(employee):
            return do_something(employee) if cond(employee) else None

    It didn't get much traction on Python-ideas, unfortunately.
  • Demur Rumed · 9 months ago
    @employee.select
    def records():
  • MRAB · 9 months ago
    A "split lambda" perhaps?

    records = employee.select(lambda employee):
    return do_something(employee) if cond(employee) else None
  • Fredrik Holmström · 9 months ago
    How about


    def foo():
    yield "hello"
    yield "world"


    foo() do(arg1)
    print arg1

    would print "hello" and then "\world", maybe it introduces some ambiguity I'm not aware of?
  • mark · 9 months ago
    Good and interesting summary :)

    The real advantage I see with blocks is (and I use {} most of the time actually) that it leads to very short code.
    However I think python is too far away from ruby to allow blocks. The syntax differences are very large.

    Ruby has no problem to use a notifier like @ but python seems to rather use __ or other solutions to not introduce "new" key-symbols like @

    For me, shorter code is always better. Really. This is btw what I like in python, that I can omit end.
    (I however cant stand implicit self, required () and the needed : after statements. On the other hand, I think ruby can be beat - a better language would be one that SIMPLIFIES ruby heavily. I dont like the module vs class separation at all for example. I also dont really like lambda. I can see the advantages of lambda and procs, but I think they should LIVE in a world full of objects, not in a world full of messages - messages are more important in the end, but well defined objects help my brain model everything easier IMHO)... I digress...

    (Btw you made a minor mistake... else: is not needed in ruby, the : seems to be pythonic )
  • tav · 9 months ago
    Thanks for catching the mistake Mark -- fixed.

    I agree that {} and shorter code would be better. Unfortunately I haven't been able to come with a Pythonic way of doing that.

    This proposal was about trying to come with a Pythonic equivalent.

    As for a better language, I agree -- and perhaps I should just do that -- there would be a lot less resistance! =)
  • zellyn · 9 months ago
    Ummm. I think you need better examples of compelling uses of blocks. I'm not sure exactly how select() works, but it sure looks like you're using a boolean filter-function for its side effects. Why not just use "for employee in employees.select(): …"?
  • alcides · 9 months ago
    Although I find the solution pretty elegant, I'd rather have multi-line lambdas. I love how it works in Javascript.

    I also like the blocks in Ruby and it's right now the only advantage I see in Ruby over Python. But with multi-line lambdas the same would be achievable.

    Oh, and make this a 2.x upgrade :)
  • tav · 9 months ago
    Multi-line lambdas would be nice, but one step at a time =)

    And thanks for the blog article!
  • Curt Hagenlocher · 9 months ago
    Given the straw-man implementation of __do__ you've given, there's no non-awkward way to write the "employee.select" function of the original example. And the notion of multiple competing versions of __do__ in userland is somewhat frightening.
  • Antti Rasinen · 9 months ago
    I'd love to see more usecases, please. The current ones are rather ambiguous.

    Specifically, what does employees.select _do_? It seems to be either filter or map, but neither really. The multiline example seems a particularly bad example of Ruby code. I'd imagine you'd probably use #each instead of #select.

    If select is like filter, then you could rewrite the first example as
    [e for e in employees if e.salary > developer.salary]

    The latter example is very confusing. I have no idea what I'd want to do with a list of return values from two different functions, both of
    which obviously have side-effects. Do they return boolean values? True on success, false on failure? Or true if we keep the employee?

    I'd rather see that written out explicitly as a for-loop:

    remaining_employees = set()
    for emp in employees:
    if emp.salary > developer.salary:
    fireEmployee(emp)
    else:
    extendContract(emp)
    remaining_employees.add(emp)

    There is no mention in the proposal about how the alternative
    __do__ functions are specified.

    Finally, as you said, it only implements a portion of the features of Ruby blocks.
  • Alia Khouri · 9 months ago
    There is probably no need to introduce ruby-like blocks to python where iteration comes naturally with list
    comprehensions and generators. But for the simple case of entering a block of code as one does with @contextmanager I suppose it would be nice to make a generator with a single yield statement a contextmanager by default such that:

    >>> def g():
    ... print "a"
    ... yield
    ... print "b"
    ...
    >>> with g():
    ... print "c"
    ...
    a
    c
    b

    would be equivalent to

    >>> from __future__ import with_statement
    >>> from contextlib import contextmanager
    >>> @contextmanager

    ... def g():
    ... print "a"
    ... yield
    ... print "b"
    ...
    >>> with g():

    ... print "c"
    ...
    a
    c
    b
  • Jan · 9 months ago
    So... the only difference is that you can skip the @contextmanager line? (You need the imports anyway in either version)
  • Alia Khouri · 9 months ago
    Well you wouldn't need the imports in the version below... and yes the assumption is that you can skip the @contextmanager line.
  • Eric · 9 months ago
    Good idea, I would love this addition. Here is a good example of a ruby lib that uses blocks to construct a dsl:

    http://www.pluginaweek.org/2009/03/08/state_mac...
  • Name · 5 months ago
    Dude, this is cool as shit, I love it! Without reading all the 50 comments, is there any progress on this, like a PEP?!
  • tav · 5 months ago
    Thanks man -- unfortunately I got stuck in terms of figuring out how to make exceptions raised inside blocks be Pythonic...

    Without solving that, not much use in putting together a PEP =(
  • Name · 5 months ago
    Sorry to hear that. I'm not familiar with the intrinsics of the PEP process, but maybe there is a chance of publishing what you have so far, and open the door for others to work on it. Maybe someone else has a solution.