Archive for 2011

the Dutch close loopholes

June 14th, 2011

If it's one thing that is very evident about Dutch culture is that they are careful to close down all the loopholes. If you get on the tram and there's a ticket inspection and you say you forgot to swipe your card -- could be an acceptable excuse in Norway -- you'll still get a fine. There's no leniency, no second chances.

Gyms are horribly expensive and their pricing policies are very sleazy, because they usually don't have any short term membership options, you have to join for at least 3 months, or 6, or 12. As a PR bandaid they like to offer an "introductory lesson" or something along those lines. Basically, you can come in and use the gym once, for free, to sort of make up your mind before the point of no return. Sometimes they have an instructor accompany you to explain what is what. It's the same scam as "the cell phone for free", where they try to distract you from the 12 month contract.

At the gym I went to today they do have an introductory lesson. But this being Holland, there is an interesting twist to it. If you want to just come in once you can pay a one time fee of 15 bucks. The introductory lesson, which in theory is free, costs you 12.50. Then, if you decide to get a membership they will deduct that 12.50 from the price you pay later. So the introductory lesson is only free if it's not free, you see that logic?

This is a pretty eccentric attitude. In theory, if you wanted to bad enough you could come in once a month, try not to run into the same staff, and always go for the introductory lesson. If you could do it so that they didn't remember you, you'd get it for free. How many people would actually try that, though? One? Three? The money a gym would lose on this scheme is pretty negligable. More importantly, anyone inclined to go to such lengths probably isn't willing to pay for the gym anyway, so that money wasn't yours to begin with.

But this is a loophole-closing culture, they don't want to run that risk. In Norway the social stigma of being found out and feeling embarrassed is deemed sufficient in such cases that basically noone does it. But the Dutch are not satisfied with that, they need policies that remove any room for such creativity.

full system encryption

June 2nd, 2011

In the age of laptops I was thinking maybe it's time I finally try encrypting my disk. I've never done it before, so before going for it I needed a small approfondimento.

The common strategy seems to be roughly:

  1. Leave /boot unencrypted.
  2. Encrypt the rest of the disk with LUKS. You then have dm-crypt that provides a mapping between the partition (according to the partition table) and the corresponding unencrypted block device, which becomes a node like /dev/mapper/nodename, depending on what you call it.
  3. Use /dev/mapper/nodename as the "physical" partition which you assign to lvm and make into a volume group.
  4. Create logical volumes in the volume group, so that each logical volume corresponds to what we used to call a partition on the old model, ie. /, /home, /var etc.

lvm is practical here, because you need at least two partitions, / and swap. You could just as well create multiple partitions sda5,sda6,... and encrypt each one, but then you'd have to unlock them individually on boot, which is hacky.

The setup is a bit involved and I would rather be spared the trouble of doing it manually. The ubuntu alternate install cd has a fully automatic feature that does this, using the whole disk. If you're happy with the basic scheme, but you want more partitions or you want to size them differently, you can use the curses gui following a nice guide like this one.

So far so good, but now comes the inevitable question. Much like with compression you want to be able to not only compress but also to decompress. What if I screw up my boot sector or my fstab and I need to boot from a rescue cd? How do I mount /dev/sda5 now?

Mounting manually

First of all, in case we don't have all the tools we need:

$ apt-get install lvm2
$ modprobe dm-mod

We obviously need to know what the physical partitions actually are:

$ fdisk -l /dev/sda

Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a30a5

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13       96256   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              13        3917    31357953    5  Extended
/dev/sda5              13        3917    31357952   83  Linux

/dev/sda1 is /boot, that's easy. Then we have /dev/sda5, which is the encrypted partition and mounting it directly will not work. This is where dm-crypt comes into the picture: we are going to unlock the partition, obtaining a block device that represents the unencrypted view of the partition.

$ cryptsetup luksOpen /dev/sda5 vg0
$ cd /dev/mapper
$ ls -lh
crw-------    1 root     root       10, 236 Jun  2 16:44 control
lrwxrwxrwx    1 root     root           7 Jun  2 16:47 vg0 -> ../dm-0

We have decided to call the block device vg0, and it thus appears as /dev/mapper/vg0. Since we know this is an lvm volume group we use the lvm tools to figure out what it contains:

$ vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "vg0" using metadata type lvm2
$ vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  vg0    1   2   0 wz--n- 29.90g    0 
$ lvscan
  inactive          '/dev/vg0/swap' [1.86 GiB] inherit
  inactive          '/dev/vg0/root' [28.04 GiB] inherit

We have two logical volumes in there. But they are inactive, which means they are not visible under /dev, ie. we can't mount them. To make them active:

$ vgchange -a y
  2 logical volume(s) in volume group "vg0" now active

The device names listed before are now visible and we can mount them:

$ cd /dev/vg0
$ ls -lh
lrwxrwxrwx 1 root root 7 Jun  2 18:24 root -> ../dm-2
lrwxrwxrwx 1 root root 7 Jun  2 18:24 swap -> ../dm-1
$ swapon /dev/vg0/swap
$ mount /dev/vg0/root /mnt

ironpython + gtk

April 9th, 2011

nametrans_about_box I develop a bunch of tools for my own use that usually run in the terminal, for the simple reason that this maximizes their usability to me. It's also true that usually a terminal program takes much less code and effort to write than a gui program.

But occasionally I want to make something of mine usable to the so-called "end users" that we keep hearing about. So what I was wondering about was: What would it be like to add a gui to an existing python program and run it on .NET?

How do you find out? You try it. This is something that I've been meaning to try for a while anyway.

Pros:

  • 2.3mb download

Cons:

  • gtk# dependency
  • sluggish gui

Gtk

This is the first time I've used gtk. I've written one program in the past using winforms and it was quite painful, so I wanted to try gtk.

To build the gui I naturally use glade, which is the best gui design experience ever. I then have to find out how to connect the signals, write the handlers and all that stuff.

Mono's api docs on gtk seem quite complete, so you can usually find out what you need to know. I seem to remember looking at these in the past seeing nothing but "Documentation for this section has not yet been entered", so perhaps they have improved this recently. I have to say they do use the beloved javadoc model of having the left frame list all the classes, which makes navigation a pain. Api doc presentation is not really known to be a cutting edge field, it really could use work. Then again, the presentation of the docs can't help reflect the organization of the api to a large extent and if your api is organized as a kingdom of nouns rather than thematic modules then there's probably little magic the api doc guy can wield.

Gtk is not as discoverable as it could be, due to the fact that they have invented some terminology. A window resize event is not a ResizeEvent, it's an ExposeEvent. When I tried to discover this one I tried connecting all the signals that mention resizing in the api docs and none of them worked. Maybe this one belongs to a base class that wasn't listed in the api doc I was looking at? Could be. (You see how api docs matter?)

In general though, I get the impression that the more you know about gtk the more it makes sense, it seems a really well engineered toolkit. This, of course, in stark contrast to winforms, where the more you know about it the worse it gets. I don't know what gtk is like in c, but at least under .NET it doesn't have the manual memory management "feature" of winforms.

Gtk is hugely superior in many ways and that I don't think is even debatable. The layouting is great, internationalization support is painless, pango rendering is lovely and lets you use markup etc. One advantage that winforms has is the way it appears under Windows. But this I think is more a theming issue than a toolkit issue. Winforms looks nice under Windows and gtk looks mediocre. On the other hand, if I boot Mono's livecd with the latest mono on it, and I run the samples there, gtk looks lovely and winforms looks atrocious, even though this is ostensibly supposed to demonstrate how well both of them run on the latest mono. So I take it from this that simply not enough people care about gtk on Windows to make it look better.

For the python coder, you can apply the api docs for c# directly, because the binding exposes the same names to python. This is very helpful. There are also quite a few pygtk examples around the web you can find if you need to discover how to do something in gtk.

IronPython

Let's start with the positives. If you give it an existing python program and tell it where the standard library is, it will run your program. In this case I had to work around some platform nastiness and patch some code that's using os (throwing false positive OSErrors on file renames) that works fine on CPython/Windows but doesn't on IronPython/Windows. But in general it seems to work quite well.

Development experience

But if all you're going to do is write code using the standard library, there is absolutely no reason to use IronPython. How do you access .NET apis from Python? No, seriously, that's not a rhetorical question. IronPython ships with a number of assemblies, so where are the api docs for these? Are they browsable on the web? Are they viewable locally? The only thing I can see documentation wise is, next to IronPython.dll, something called IronPython.xml. It seems to be an aggregate of all the inline api comments in the source code. Is this it? Is this how you're supposed to view the api doc? IronPython 2.7 does something interesting. It ships a file called IronPython.chm, but only the .msi installer package includes this. As I'm sure the IronPython guys are well aware, the installer is Windows only and the .chm file too is Windows only (although I think there is something called kchmviewer that could read it).

In fact, the .chm file is basically the original Python 2.7 documentation (written in .rst, which compiles to html), renamed to IronPython 2.7, and includes a few additional pages about IronPython. Except that instead of throwing the html files on the web or including them in the download, they have compiled them into a .chm which they only give to the Windows people. And this is supposed to be a platform independent runtime? Is this a prank?

IronPython ships some examples, but it's not enough to figure out how to use the apis. To write a veritable IronPython gui program you need to produce an assembly, which has to be compiled with -target:winexe so that it does not show a terminal window on Windows. The simplest way seems to be to write a wrapper in c#, using the runtime hosting api, and execute your python in there. That's what the Pyjama project uses (they also use gtk#), so I was able to look that up. But when I needed to set up sys.argv and sys.version using the hosting apis before launching the python I had no idea how. How are you supposed to find this out without documentation?

It turns out that you need to do something like this:

ScriptScope scope = Python.ImportModule(runtime, "sys");
scope.SetVariable("version", "ironpython");

Easy, right? Yeah, when you know it, of course it is. But what if you didn't? How would you know that there is a function ImportModule in the Python namespace?

And how do you set sys.argv?

IronPython.Runtime.List lst = (IronPython.Runtime.List) scope.GetVariable("argv");

How would you know that IronPython.Runtime.List is the class that models a python list? You can use it like a python list:

lst.append("program.py");

In the end your best bet might actually be dir(). So how do you find out what's in the Python namespace when coding c#? Use python:

clr.AddReference("IronPython")
import IronPython
print dir(IronPython.Hosting.Python)

You could actually use this method to recursively dir() the assemblies and produce a map of what's in there, generating some kind of api doc in the end, but this is getting quite out of hand now.

At this point I was going to mention msdn and how that ought to host all the .NET related api docs you could ever want, but I see that the site is even worse than it used to be and I don't have the palest idea of where to find anything anymore. I can only imagine that if you install the latest Visual Studio Ultimate Premium Professional it will enhance your hard drive with untold gigabytes of xml compiled to a proprietary binary file format worth of api docs, which you can only browse in the proprietary api doc reader, which internally just renders html anyway. But actually, since IronPython was ousted by Microsoft and since it has the stigma of being an open source project, it could well be that it would be considered too tainted to include its api docs in the Visual Studio-installed doc browser.

As your final recourse, I suggest you warm up your grep, git clone this IronLanguages repo and hope for the best.

Debugging

When I try this somewhere in my hosted python program:

v = None + 1

this is what I get:

Unhandled Exception: Microsoft.Scripting.ArgumentTypeException: unsupported operand type(s) for +: 'NoneType' and 'int'

And the rest of the stack trace doesn't mention the python code at all. Better than "Segmentation fault", yes, but not by a whole lot. No filename, no line number.

This being the case I would strongly recommend developing your python with CPython first and then, once it's in good shape, build the gui so you can do most of your debugging under CPython.

Upgrade cycle

I mentioned IronPython 2.7 earlier, but funny thing, I've never actually tried it. The 2.7 release is compiled against .NET 4.0, which is not supported by the mono packages in Ubuntu. I thought that since I have everything working with .NET 2.0/IronPython 2.6 I could just throw in the newer assemblies, run my makefile and compile it against .NET 4.0, on a mono release recent enough to support 4.0 (such as the mono livecd I mentioned before). But no deal. And if you just install .NET 4.0 under Windows it doesn't come with a compiler or anything, so there's no way to try it.

IronPython + Gtk

So how responsive is an ironpython/gtk application, even a tiny one like this?

Start up speed on my Ubuntu system is something like 4s, which is just about fast enough not to notice that it runs on IronPython instead of CPython. But Ubuntu uses gtk natively, so the libraries are in memory. On Windows it varies from a warm start of 6s to 20s+, for a cold start. In the worst case the first hello message from python appears after maybe 5s, so the rest must be accounted for by loading gtk.

Once it's running, it can be a bit sluggish, in particular halting the program on Ubuntu sometimes seems to freeze the gui for a few seconds before it goes away. On Windows the issue (as usual) seems to be io. This program scans the filesystem whenever the input parameters change, which produces a list of files that is eventually displayed in the gui. But I'm not convinced that it's any slower than CPython/Windows.

nametrans: renaming with search/replace

March 25th, 2011

Keeping filenames properly organized is a pain when all you have available for the job is renaming files one by one. It's most disheartening when there is something you have to do to all the files in the current directory. This is where a method of renaming by search and replace, just as in a text document, would help immensely. Something like this perhaps:

nametrans_ss

Simple substitutions

The simplest use is just a straight search and replace. All the files in the current directory will be tried to see if they match the search string.

$ nametrans.py "apple" "orange"
 * I like apple.jpg    -> I like orange.jpg
 * pineapple.jpg       -> pineorange.jpg
 * The best apples.jpg -> The best oranges.jpg

There are also a number of options that simply common tasks. Options can be combined and the order in which they are set does not matter.

Ignore case

Matching against strings with different case is easy.

$ nametrans.py -i "pine" "wood"                                                        
 * pineapple.jpg -> woodapple.jpg
 * Pinetree.jpg  -> woodtree.jpg

Literal

The search string is actually a regular expression. If you use characters that have a special meaning in regular expressions then set the literal option and it will do a standard search and replace. (If you don't know what regular expressions are, just use this option always and you'll be fine.)

$ nametrans.py --lit "(1)" "1" 
 * funny picture (1).jpg -> funny picture 1.jpg

Root

If you prefer the spelling "oranje" instead of "orange" you can replace the G with a J. This will also match the extension ".jpg", however. So in a case like this set the root option to consider only the root of the filename for matching.

$ nametrans.py --root "g" "j"
 * I like orange.jpg    -> I like oranje.jpg
 * pineorange.jpg       -> pineoranje.jpg
 * The best oranges.jpg -> The best oranjes.jpg

Hygienic uses

Short of specific cases of transforms, there are some general options that have to do with maintaining consistency in filenames that can apply to many scenarios.

Neat

The neat option tries to make filenames neater by capitalizing words and removing characters that are typically junk. It also does some simple sanity checks like removing spaces or underscores at the ends of the name.

$ nametrans.py --neat                                                                    
 * _funny___picture_(1).jpg -> Funny - Picture (1).jpg
 * i like apple.jpg         -> I Like Apple.jpg
 * i like peach.jpg         -> I Like Peach.jpg
 * pineapple.jpg            -> Pineapple.jpg
 * the best apples.jpg      -> The Best Apples.jpg

Lower

If you prefer lowercase, here is the option for you.

$ nametrans.py --lower
 * Funny - Picture (1).jpg -> funny - picture (1).jpg
 * I Like Apple.jpg        -> i like apple.jpg
 * I Like Peach.JPG        -> i like peach.jpg
 * Pineapple.jpg           -> pineapple.jpg
 * The Best Apples.jpg     -> the best apples.jpg

If you want the result of neat and then lowercase, just set them both. (If you like underscores instead of spaces, also set --under.)

Non-flat uses

Presuming the files are named consistently you can throw them into separate directories by changing some character into the path separator.

Note: On Windows, the path separator is \ and you may have to write it as "\\\\".

$ nametrans.py " - " "/"
 * france - nice - seaside.jpg -> france/nice/seaside.jpg
 * italy - rome.jpg            -> italy/rome.jpg

The inverse operation is to flatten the entire directory tree so that all the files are put in the current directory. The empty directories are removed.

$ nametrans.py --flatten
 * france/nice/seaside.jpg -> france - nice - seaside.jpg
 * italy/rome.jpg          -> italy - rome.jpg

In general, the recursive option will take all files found recursively and make them available for substitutions. It can be combined with other options to do the same thing recursively as would otherwise happen in a single directory.

$ nametrans.py -r --neat 
 * france/nice/seaside.jpg -> France/Nice/Seaside.jpg
 * italy/rome.jpg          -> Italy/Rome.jpg

In recursive mode the whole path will be matched against. You can make sure the matching only happens against the file part of the path with --files or only the directory part with --dirs.

Special uses

Directory name

Sometimes filenames carry no useful information and serve only to maintain them in a specific order. The typical case is pictures from your camera that have meaningless sequential names, often with gaps in the sequence where you have deleted some pictures that didn't turn out well. In this case you might want to just use the name of the directory to rename all the files sequentially.

$ nametrans.py -r --dirname                                                              
 * rome/DSC00001.jpg -> rome/rome 1.jpg
 * rome/DSC00007.jpg -> rome/rome 2.jpg
 * rome/DSC00037.jpg -> rome/rome 3.jpg
 * rome/DSC00039.jpg -> rome/rome 4.jpg

Rename sequentially

Still in the area of sequential names, at times the numbers have either too few leading zeros to be sorted correctly or too many unnecessary zeros. With this option you can specify how many leading zeros you want (and if you don't say how many, it will find out on its own). Based on an old piece of code that has been integrated.

$ nametrans.py -r --renseq 1:3                                                           
 * rome/1.jpg   -> rome/001.jpg
 * rome/7.jpg   -> rome/007.jpg
 * rome/14.jpg  -> rome/014.jpg
 * rome/18.jpg  -> rome/018.jpg
 * rome/123.jpg -> rome/123.jpg

The argument required here means field:width, so in a name like:

series14_angle3_shot045.jpg

the number 045 can be shortened to 45 with "3:2" (third field from the beginning) or "-1:2" (first field from the end).

Get it from sourceforge:

making hypothetical statements in italian

March 7th, 2011

A hypothetical statement is one that is speculative. It speculates on the consequences of a hypothetical event. It states what would occur if something that hasn't happened, were to happen.

Expressing hypotheticals in Italian (il periodo ipotetico) is somewhat tricky because it requires use of the subjunctive, a verb tense that takes getting used to. It is further complicated by making the chapter on this topic that you will find in textbooks overly long and confusing.

Here I will start with two solid reference points in English, so that you can see exactly what is going on. Then we'll go through some examples to see what it looks like in real world use.

Could

The first case is the could scenario.

- You've never scored a penalty.
- Yeah, but I could do it.

- You don't play the piano, do you?
- I could if I wanted to!

Got it? Okay, so we build our hypothetical from the could.  And the hypothetical describes what would happen if you did the thing that you "could do". (Or, in general, what would happen if the thing that "could happen", happened.)

- If I scored a penalty, I would celebrate.

- If I played the piano, I would play Beethoven all the time.

Here is what you need to know:

  1. The action is described using the imperfect subjunctive (il congiuntivo imperfetto).
  2. The consequence of the action is described using the conditional (il condizionale).

- Se segnassi un rigore, lo festeggerei.

- Se suonassi il pianoforte, suonerei sempre Beethoven.

Of course, you are not limited to talking about yourself.

- If it rained, they would not show up.
- Se piovesse, non si farebbero vivi.

Could have

The second scenario is all about could have.

- You lost the game!
- Yeah, but we could have won.

- Al Capone never jumped out of a window.
- Yeah, but he could have, he was that crazy.

Once again, the hypothetical starts from the thing that "could have happened".

- If we had won the game, we would have been champions.

- If he had jumped out of a window, he would have terrified his underlings even more.

In this case the tenses are different:

  1. The action is described using the pluperfect subjunctive (il congiuntivo trapassato).
  2. The consequence of the action is described using the past conditional (il condizionale passato).

- Se avessimo vinto la partita, saremmo stati campioni.

- Se si fosse buttato dalla finestra, avrebbe fatto ancora più paura ai suoi subalterni.

A slight variation on this is used if the consequence is one that is still felt today, in which case you use the regular conditional:

- If he had jumped out of a window, he would be even more famous.
- Se si fosse buttato dalla finestra, sarebbe ancora più famoso.

Examples

If you're not reading Leonardo then you really should, the guy is hilarious. In a recent entry he made a superb demonstration of the hypothetical.

I'll just quote one sentence here, but read the whole thing so you understand what is being talked about. The short version is that Berlusconi is alleged to have employed underage prostitutes, one of whom was nicknamed Ruby. Hiring a prostitute in Italy does not constitute a crime, but if the person is underage then it obviously does. Leonardo here is satirizing the kind of statements made in support of Berlusconi by various figures who more or less owe their position to him. A classic case of scandal and farce in Italian politics.

Il sesso coi cyborg non è ancora regolamentato per legge, e quindi Berlusconi (che è impotente) (e se non fosse impotente sarebbe fidanzato) (e se avesse tradito la fidanzata con Ruby, comunque non l'avrebbe pagata) (e se l'avesse pagata, comunque sarebbe maggiorenne) non può essere punito per aver fatto sesso con un cyborg.

- e se non fosse impotente sarebbe fidanzato
- and if he weren't impotent, he would have a girlfriend

- e se avesse tradito la fidanzata con Ruby, comunque non l'avrebbe pagata
- and if he had cheated on his girlfriend with Ruby, he still wouldn't have paid her for it

- e se l'avesse pagata, comunque sarebbe maggiorenne
- and if he had paid her, she would be of age anyway