I've been toying with the idea of trying my hand a generic pretty printer module for a while. Lately I've had to deal with cyclic object graphs and things like that, where having a dump of the data is pretty handy. Granted there is a pprint module in the standard library. But what it does is format and print iterables (lists, dicts, tuples..), it doesn't attempt to show you the contents of an object. Of course, when you're messing with objects this is very useful to have.

So I thought that I would build a recursive iterable that I can give to pprint. Here's an example:

class Node(object):
    classatt = 'hidden'
    def __init__(self, name):
        self.name = name

a, b, c, d = Node('A'), Node('B'), Node('C'), Node('D')
a.refs = [b, d]
b.refs = [c]
c.refs = [a]
d.refs = [c]

This will give you:

{'__type__': '<Node {id0}>',
 'name': "'A'",
 'refs': [{'__type__': '<Node {id1}>',
           'name': "'B'",
           'refs': [{'__type__': '<Node {id2}>',
                     'name': "'C'",
                     'refs': ['dup <Node {id0}>']}]},
          {'__type__': '<Node {id3}>',
           'name': "'D'",
           'refs': [{'__type__': '<Node {id2}>',
                     'name': "'C'",
                     'refs': ['dup <Node {id0}>']}]}]}

There are two things being shown here:

• node C is reachable through ABC and ADC.
• A takes part in two cycles: ABCA and ADCA.

It would be nice to have a way to see this from the output. So aside from the object attributes themselves there is also a __type__ attribute which tells you the type that you're looking at. And it has a marker of the form {id1}, where id1 is an identifier for this object, so that you can see where it pops up in a different part of the graph.

Now, suppose we follow A to B to C and then to A. We are now seeing A for the second time. Instead of printing the object again we print a duplicate marker: dup <Node {id0}>. The identifier is supposed to be vim * friendly, so if you pipe the output to vim, put the cursor over it and hit * (also might want to do set hlsearch) then you'll see it light up all the other instances of it in the graph.

Well, that's all for now. It's definitely not the last word in pretty printing, but it's useful already.

I thought maybe github's gists would be appropriate for something like this:

• pp module

I already mentioned Coders at work in an earlier entry. The point of this one is not to write a review, but to make a note for myself of what I've gotten out of the book. I think I could do better to read more books with a pen and a pad so I have a better chance of exploiting the content.

So these are notes to myself. I wouldn't take it upon myself to summarize a more general listing of notes that would somehow apply to the average person, because I think we're all in very different places in the universe that is called "learning to program (well)", and every person has to figure out for himself what he most needs to learn relative to where he now is.

Advice: Read code

Read other people's code, "open black boxes". This is something I never really do, I should start. Just take some codebase and check it out, get used to the practice. Reading code is not the easiest thing to get into, so here are some tips:

1. First, get it to build.
   Sometimes everything you have to do to build it already teaches you a number of things about the codebase. And once you have it built, you can start making changes to it and try out little things dynamically.
2. Read while building.
   Making builds for any codebase can be hairy and painful, so parallelize this activity with code reading. Great way to use the time you'd otherwise waste in between debugging the build.

Advice: Write unit tests for new library

You've found a library for something that you've never used before: how do you figure out how to use it? Write unit tests. Some libraries have bad unit tests (or no tests) to begin with, so it could be a way to improve it. In any case you can test your basic hypotheses of how the library works.

Ideas to investigate

1. OO and classes vs prototypes (JavaScript).
2. "There is a lack of reuse in OO because there is too much state inside". Libraries must expose too much of their innards through APIs, functional programming model should be better at this.

Pointers

Articles:

1. Richard P. Gabriel - Worse Is Better

Blogs:

1. How to read code – a primer

Books:

1. Douglas Crockford - JavaScript: The Good Parts
   In the absence of the book, Crockford's lecture series on JavaScript is probably a good start.
2. William Strunk, Jr. and E.B. White - The Elements of Style
   For writing better English.
3. Steve McConnell - Code Complete
   On software engineering process and best practices.
4. Gerald Weinberg - The Psychology of Computer Programming

Talks:

1. Joshua Bloch - How to Design a Good API and Why it Matters

I'm reading Peter Seibel's book "Coders at work". It's a collection of interviews with famous programmers. This is the kind of book I really like, it's not a technical book, but it's a meta sort of book where these people tell you what they think about various relevant issues in the industry. And not just issues that concern them directly, but general trends too. It's a very easy read, perfect for the plane or the airport.

There are 15 interviews and almost all these people started playing with computers sort of roughly before there were computers. So if there is a theme running through the book, it is this:

1. Kids today don't understand how the metal works.
2. I don't like all these layers of software.

I think it's an understandable point of view coming from people who've written operating systems and compilers and coded assembly and machine code because there was nothing else available. But I don't find it a very helpful perspective.

The basic complaint is this:

1. Things used to be simple.
2. Instead of remaining simple, they got complex, but not in a good way (ie. bad technical decisions).

I think this is an "argument from nostalgia", essentially. Back in the days, systems were simpler. Today they are very complicated. And so we wish things were simpler. But this is because some people were present more or less at the "birth" of computer science. The field went from zero and just keeps expanding. That's normal, though.

If a physicist said "I hate how when you discover a layer of particles, there's always something smaller than that!" would people nod in agreement? I remember learning about atomic orbitals and not understanding them and I kept thinking "what was wrong with the Bohr model, that one was so much simpler and nicer?"

The difference between physics and computer science is that in physics there's noone to blame for what is there. There is this sense of "nature is the goddess who bestows gifts upon us and we have the privilege to explore them". In computer science we're not trying to explain or discover anything, we make all this stuff up!

In physics there's no way you can remove the complexity and be left with a simple system, the complexity is there at all levels. But in computers you can delete everything save for the kernel and you indeed have a simple system. (Better yet, delete the kernel too and install a simpler one that you wrote yourself.)

The fundamental difference, to me, is that there is someone to blame. There is noone to blame for atomic orbitals and "why do they have to be so complicated??", but there is someone to blame for every programming language and every system. I don't think for a minute that we wouldn't do the same in physics if we had the chance, though.

What's Plan B?

Of course, the difference between the physical sciences and computer science raises the old "is it a science?" question, but at any rate it is becoming more like physics in the sense of a top to bottom system that is difficult to understand at all levels.

In physics you don't say things like "I would like to throw all this out and start over, make it simple". This is something you can totally do in computers, but chances are you're not gonna have much impact. Sometimes people bemoan how there hasn't been any innovation in operating systems in 30 years. So go write your own, see how many people you can convince to use it.

In a way, the answer is right there. The fact that there aren't any new operating systems taking over from the old ones, _means_ that the old ones have succeeded. They've successfully laid that layer of bricks that has proven to be a strong enough abstraction to move away from that layer in the system and focus our attention on something higher up. They're not works of art in terms of simplicity and purity, but neither are layers of abstraction in physics. *ducks*

Complexity is often presented as a mistake, but the fact that we have all this complexity is not really an accident, it has to be there to do the kinds of things that we want to do.

One thing that's awesome in python is having a small codebase that can fit in a single directory. It's a comfy setting, everything is right there at your fingertips, no directory traversal needed to get a hold of a file.

Flat structure

Let's check out one right now:

./frame.py
./master.py
./mystring.py
./page.py
./sentence.py
./user.py

And here's the import relationship between them:

Easy, straightforward. I can execute any one of the files by itself to make sure the syntax is correct or to run an "if __main__" style unit test on it.

Tree structure

But suppose the codebase is expanding and I decide I have to get a bit more structured? I devise a directory structure like this:

./media/book/__init__.py
./media/book/page.py
./media/book/sentence.py
./media/__init__.py
./media/master.py
./media/movie/frame.py
./media/movie/__init__.py
./media/mystring.py
./user.py

The same files, but now with __init__.py files all over the codebase to tell python to treat each directory as a package. And now my import statements have to be changed too, let's see master:

# from:
import mystring
import page
# to:
import media.mystring
import media.book.page

Nice one. Okay, let's see how this works now:

$ python user.py
page says hello!
sentence says hello!
frame says hello!
mystring says hello!
master says hello!

user imports page and then master. The first 4 lines are due to page, which imports three modules, and finally we see master arriving at the scene. All the files it imports have already been imported, so python doesn't redo those. Everything is in order.

As you can see, imports between modules in the tree work out just fine, page finds both the local sentence and the distant frame.

But if we run master it's a different story:

$ python media/master.py
master says hello!
Traceback (most recent call last):
  File "media/master.py", line 3, in <module>
    import media.mystring
ImportError: No module named media.mystring

And it doesn't actually matter if we run master from media/ or run media/master from ., it's the same result. And it's the same story with page, which is deeper in the tree.

These modules, which used to be executable standalone, no longer are. :(

A hackish solution

So we need something. The nature of the problem is that once we traverse into media/, python no longer can see that there is a package called media, because it's not found anywhere on sys.path. What if we could tell it?

The problem pops up when the module is being executed directly, in fact when __name__ == '__main__'. So this is the case in which we need to do something differently.

Here's the idea. We put a file in the root directory of the codebase, a file we can find that marks where the root is. Then, whenever we need to find the root, we traverse up the tree until we find it. The file is called .codebase_root. And for our special when-executed logic, we use a file called __path__ that we import conditionally. Here's what it looks like:

import os
import sys

def find_codebase(mypath, codebase_rootfile):
    root, branch = mypath, 'nonempty'
    while branch:
        if os.path.exists(os.path.join(root, codebase_rootfile)):
            codebase_root = os.path.dirname(root)
            return codebase_root
        root, branch = os.path.split(root)

def main(codebase_rootfile):
    thisfile = os.path.abspath(sys.modules[__name__].__file__)
    mypath = os.path.dirname(thisfile)
    codebase_root = find_codebase(mypath, codebase_rootfile)

    if codebase_root:
        if codebase_root not in sys.path:
            sys.path.insert(0, codebase_root)

codebase_rootfile = '.codebase_root'
main(codebase_rootfile)

So now, when we find ourselves in a module that's somewhere inside the media/ package, we have this bit of special handling:

print "master says hello!"

if __name__ == '__main__':
    import __path__
import media.mystring
import media.book.page

Unfortunately, importing __path__ unconditionally breaks the case where the file is not being executed directly and I haven't been able to figure out why, so it has to be done like this. :/

You end up with a tree looking as you can see in the screenshot.

I've pushed the example to Github so by all means have a look:

• python-codebase-structure

We pass the test, all the modules are executable standalone again. But I can't say that it's awesome to have to do it like this.