two weeks of rust

January 10th, 2016

Disclaimer: I'm digging Rust. I lost my hunger for programming from doing too many sad commercial projects. And now it's back. You rock, Rust!

I spent about two weeks over the Christmas/New Year break hacking on emcache, a memcached clone in Rust. Why a memcached clone? Because it's a simple protocol that I understand and is not too much work to implement. It turns out I was in for a really fun time.

UPSIDES

The build system and the package manager is one of the best parts of Rust. How often do you hear that about a language? In Python I try to avoid even having dependencies if I can, and only use the standard library. I don't want my users to have to deal with virtualenv and pip if they don't have to (especially if they're not pythonistas). In Rust you "cargo build". One step, all your dependencies are fetched, built, and your application with it. No special cases, no build scripts, no surprising behavior *whatsoever*. That's it. You "cargo test". And you "cargo build --release" which makes your program 2x faster (did I mention that llvm is pretty cool?)

Rust *feels* ergonomic. That's the best word I can think of. With every other statically compiled language I've ever used too much of my focus was being constantly diverted from what I was trying to accomplish to annoying little busy work the compiler kept bugging me about. For me Rust is the first statically typed language I enjoy using. Indeed, ergonomics is a feature in Rust - RFCs talk about it a lot. And that's important, since no matter how cool your ideas for language features are you want to make sure people can use them without having to jump through a lot of hoops.

Rust aims to be concise. Function is fn, public is pub, vector is vec, you can figure it out. You can never win a discussion about conciseness because something will always be too long for someone while being too short for someone else. Do you want u64 or do you want WholeNumberWithoutPlusOrMinusSignThatFitsIn64Bits? The point is Rust is concise and typeable, it doesn't require so much code that you need an IDE to help you type some of it.

Furthermore, it feels very composable. As in: the things you make seem to fit together well. That's a rare quality in languages, and almost never happens to me on a first project in a new language. The design of emcache is actually nicely decoupled, and it just got that way on the first try. All of the components are fully unit tested, even the transport that reads/writes bytes to/from a socket. All I had to do for that is implement a TestStream that implements the traits Read and Write (basically one method each) and swap it in for a TcpStream. How come? Because the components provided by the stdlib *do* compose that well.

But there is no object system! Well, structs and impls basically give you something close enough that you can do OO modeling anyway. It turns out you can even do a certain amount of dynamic dispatch with trait objects, but that's something I read up on after the fact. The one thing that is incredibly strict in Rust, though, is ownership, so when you design your objects (let's just call them them that, I don't know what else to call them) you need to decide right away whether an object that stores another object will own or borrow that object. If you borrow you need to use lifetimes and it gets a bit complicated.

Parallelism in emcache is achieved using threads and channels. Think one very fast storage and multiple slow transports. Channels are async, which is exactly what I want in this scenario. Like in Scala, when you send a value over a channel you don't actually "send" anything, it's one big shared memory space and you just transfer ownership of an immutable value in memory while invalidating the pointer on the "sending" side (which probably can be optimized away completely). In practice, channels require a little typedefing overhead so you can keep things clear, especially when you're sending channels over channels. Otherwise I tend to get lost in what goes where. (If you've done Erlang/OTP you know that whole dance of a tuple in a tuple in a tuple, like that Inception movie.) But this case stands out as atypical in a language where boilerplate is rarely needed.

Macros. I bet you expected these to be on the list. To be honest, I don't have strong feelings about Rust's macros. I don't think of them as a unit of design (Rust is not a lisp), that's what traits are for. Macros are more like an escape hatch for unpleasant situations. They are powerful and mostly nice, but they have some weird effects too in terms of module/crate visibility and how they make compiler error messages look (slightly more confusing I find).

The learning resources have become very good. The Rust book is very well written, but I found it a tough read at first. Start with Rust by example, it's great. Then do some hacking and come back to "the book", it makes total sense to me now.

No segfaults, no uninitialized memory, no coercion bugs, no data races, no null pointers, no header files, no makefiles, no autoconf, no cmake, no gdb. What if all the problems of c/c++ were fixed with one swing of a magic wand? The future is here, people.

Finally, Rust *feels* productive. In every statically compiled language I feel I would go way faster in Python. In Rust I'm not so sure. It's concise, it's typeable and it's composable. It doesn't force me to make irrelevant nit picky decisions that I will later have to spend tons of time refactoring to recover from. And productivity is a sure way to happiness.

DOWNSIDES

The standard library is rather small, and you will need to go elsewhere even for certain pretty simple things like random numbers or a buffered stream. The good news is that Rust's crates ecosystem has already grown quite large and there seem to be crates for many of these things, some even being incubated to join the standard library later on.

While trying to be concise, Rust is still a bit wordy and syntax heavy with all the pointer types and explicit casts that you see in typical code. So it's not *that easy* to read, but I feel once you grasp the concepts it does begin to feel very logical. I sure wouldn't mind my tests looking a bit simpler - maybe it's just my lack of Rust foo still.

The borrow checker is tough, everyone's saying this. I keep running into cases where I need to load a value, do a check on it, and then make a decision to modify or not. Problem is the load requires a borrow, and then another borrow is used in the check, which is enough to break the rules. So far I haven't come across a case I absolutely couldn't work around with scopes and shuffling code around, but I wouldn't call it fun - nor is the resulting code very nice.

Closures are difficult. In your run-of-the-mill language I would say "put these lines in a closure, I'll run them later and don't worry your pretty little head about it". Not so in Rust because of move semantics and borrowing. I was trying to solve this problem: how do I wrap (in a minimally intrusive way) an arbitrary set of statements so that I can time their execution (in Python this would be a context manager)? This would be code that might mutate self, refers to local vars (which could be used again after the closure), returns a value and so on. It appears tricky to solve in the general case, still haven't cracked it.

*mut T is tricky. I was trying to build my own LRU map (before I knew there was a crate for it), and given Rust's lifetime rules you can't do circular references in normal safe Rust. One thing *has to* outlive another in Rust's lifetime model. So I started hacking together a linked list using *mut T (as you would) and I realized things weren't pointing to where I thought they were at all. I still don't know what happened.

The builder pattern. This is an ugly corner of Rust. Yeah, I get that things like varargs and keyword arguments have a runtime overhead. But the builder pattern, which is to say writing a completely separate struct just for the sake of constructing another struct, is pure boilerplate, it's so un-Rust. Maybe we can derive these someday?

Code coverage. There will probably be a native solution for this at some point. For now people use a workaround with kcov, which just didn't work at all on my code. Maybe it's because I'm on nightly? Fixed!

---

So there you have it. Rust is a fun language to use, and it feels like an incredibly well designed language. Language design is really hard, and sometimes you succeed.

django girls 2015 summary

December 16th, 2015

I've never been too far removed from Django Girls. I was present at EuroPython 2014 where the very first Django Girls workshop was held. But at the time I had no idea what that even was. It wasn't until a few months later that the story of Django Girls started reverberating through the Python web.

I decided to take the plunge in September and attend my first Django Girls in Groningen. It was a great experience and I knew right away that I wanted to do more of these. I kept checking the event calendar every few weeks and eventually I settled on Bordeaux in November and Rome in December.

Stats

Groningen: 60 participants, 20 coaches

Bordeaux: 27 participants, 10 coaches

Rome: 35 participants, 11 coaches

Our combined group of 40 coaches has taught programming for a day to 120 women! And globally Django Girls is nearing 3000 participants with multiple events happening pretty much every single weekend.

What about 2016?

The big news is that we are organizing our own event in Den Haag in March!

Beyond that I'm sure I will be present at other Django Girls events too - I just don't know which ones yet.

djangogirls as a keyframe moment

December 3rd, 2015

In video compression there is a concept known as the keyframe. The simplest way to store a video recording is to store each frame (let's say it's 25 frames per second) as an image, all together in one big file. But people noticed that when progressing from frame to frame most of the image of a frame does not change between frames. That makes sense, otherwise the whole screen would flash constantly. The new model of storing video was to store frame 1 in its entirety, and then for frame 2, frame 3 and onwards you would only store what changed from the previous frame.

This gives enormous gains in terms of filesize and all common video formats use it. But we don't always view video from frame 1 right up until the end. Sometimes we want to seek backwards and forwards. If we only had one frame in its entirety and then a long sequence of frame changes it would be very expensive (and therefore slow) to calculate what frame 37500 at the 25 minute mark should look like.

That's why we also have keyframes. A keyframe is simply a frame that is stored in its entirety (like frame 1), which occurs at certain moments in the video. For instance, once every minute (every 1500 frames) we store the whole frame, and in between keyframes we only store changes or "deltas".

As it happens, keyframes also have a second purpose as an error correction mechanism. If you ever try to play back a video and all you see is the whole screen full of weird purple rectangles with something moving here and there, that's because the video player is applying changes frame by frame, to a keyframe which is damaged. So all you see is the changes, but not the original image from which the changes have been calculated. And thus it looks like the video is broken. But if you wait long enough, you might reach a keyframe. And once you have a keyframe you have the whole image again, and calculating changes from that image makes the video "work" again.1 So if some of the keyframes are missing or damaged it breaks part of the video, but not all of it. Keyframes are a mechanism to recover from a partially damaged video file.

We can even view a keyframe as a metaphor in life. A keyframe is a point where we have excellent clarity - we feel we know exactly what we need to keep doing, what we need to stop doing, what we need to start doing. But then the moment passes, we chug along, and after a while it's not all that easy to measure exactly how much progress we've made, or whether we're headed in the right direction. Not until the next moment of clarity. The next keyframe, if you will.

---

I'm heading home from Django Girls Bordeaux and as I'm sitting on the plane I realize I'm incredibly at peace with the world today. Nothing seems to bother me at all. As if I've just nourished myself on the exact right diet that my body needed. Even physiologically speaking - over the last two days I've lived on less food and less sleep than normal. Yet my body kept saying: that's all we need. Like an engine running on the perfect fuel, at peak performance.

I've coached at Django Girls once before, but this time was even better. As "meta coach" my mission was to supplement the coaches - one per team - by helping the participants whose coach was busy, or by helping with obscure problems. In practice I ended up more like a doctor doing rounds - I would quite systematically visit each team and help anyone who needed assistance. This was super awesome, since I had a chance to talk to all the participants and all the coaches.

At its core Django Girls is a really simple idea. We invite a group of women and teach them some basic programming and web development. Big deal, right? But think about the last time you learned some basic programming... What did that lead to? A career in software by any chance? A life long interest in programming perhaps? So you see, it *is* a big deal. It's actually one of the bigger deals out there.

Let me illustrate why.

On Friday night we had a drinks event for the coaches. It was basically like any group of programmers drinking beer and talking shop, with splinters of pip, pypi mirrors, wheels and devpi flying everywhere (mercifully, a critique on the state of python packaging was omitted). For me it was doubly interesting, since that's literally the first time I've ever talked about programming in French.2 But I've heard a version of that conversation any number of times, and basically it's a group of coders making fun of and complaining about stuff all night long, albeit in an entertaining manner.3 This is our culture, people. Oh and I forgot to mention that 9 of 10 coaches were male. (But that's irrelevant, right?) End of scene.

Saturday rolls around, it's the big day. The room fills with 27 participants and their coaches. Everyone's ready to roll. The participants are a little shell shocked at first, not surprisingly. How would you feel if you had to type cryptic stuff in a terminal window? But quite soon everyone is humming along nicely. The coaches aren't talking amongst themselves now, they're with their teams. And each one is doing a fantastic job. (I saw the whole thing, remember? I was walking around.) They're being supportive. They're enthusiastic. They explain things patiently. They help the participants relax and understand that they're on the right track.

Let me repeat that in case you got distracted for a second. The *participants* are making the *coaches* better. They're making *us* think and feel more positive and have more fun than usual.

And the participants? Well, where do I start? When is the last time you had the courage and the patience to spend 9 hours learning something new? Something that many people think "isn't for people like yourself"? Right, chew on that one for a minute. You see, Django Girls isn't like being in school where you mindlessly sit through one class after another. The people who participate are motivated to learn. They work through a tutorial that, despite having been written for beginners, is full of technical terms they've never heard before.4 They're expected to learn mostly through reading and performing exercises. They're expected to absorb a lot of knowledge throughout the day. They're expected to show a lot of patience and perseverance. And boy do they ever. Not only that, they are some of the nicest and most fun people you could ever spend a day with.

So you see, it's not that we're doing them a favor because one can have a nice career in IT these days. You don't have to work in IT to have an awesome career, there are many other options. "We" need "them" much more than "they" need "us".5

They are beginners. That sounds like a bad thing, but it's actually an awesome thing. They didn't come up through our computer science programs. They didn't cut their teeth on PHP back when it was just C macros.6 They come from a different background, with a different mindset, and a fresh perspective. They're not jaded like us, joking around about how broken software is. And we, the coaches, are seeing it through their eyes. Why is coaching at Django Girls so fulfilling? Because wow, that's what programming is *supposed* to be like!

Some highlights that stuck in my mind:

  • The participant who said "so a=6 and b=4, but how do you get 24?" Thirty minutes later not only is assignment under control, she says "oh so len and str are functions I can use just like in Excel"?
  • The participant who would explain back to me what she had learned, in her own words, and her explanation was dead on every time. Later on she got into a discussion with her coach on the merits of Django vs Wordpress and what you could do/couldn't do with either one.
  • The participant who early in the day was almost too focused to talk. By the afternoon she'd racked up enough wins that she was listening to music on her phone, reading the tutorial out loud and nodding along, totally getting it.
  • The participant who finished the whole tutorial with time to spare (trust me, it's really long) and was browsing sites on code bootcamps while telling me about a site she was planning to build to serve files for use at the library where she works.

And that, to make a long story short, is why someone who's been a programmer for 20 years is raving about how incredible it is to coach at Django Girls. A moment where everything comes together and everything is awesome. A keyframe moment that reminds me that *this* is what I want my life to be about.

What are the keyframe moments of your life?

---

  1. I say "work" because as far as the video player is concerned a frame is just a bunch of pixels with colors. It has no way of knowing what the frame is supposed to look like.
  2. The whole idea of coaching in French was a massive YOLO. It was tough, but it was super fun!
  3. I know it extremely well, because in my previous team that was literally my favorite way to pass the time with my programmer buddies.
  4. I worked through it myself. Last week I didn't know fichier from dossier. I made a stack of flashcards for all the technical terms you encounter in the tutorial and crammed them. Thank you for saving my skin, Anki!
  5. The "we" vs "they" rhetoric is pretty painful to read, isn't it?
  6. Or whatever your claim to fame is.

no metro de lisboa

August 4th, 2015

O metro de Lisboa tem um sistema de passagens eletrônico. Você tem que comprar o seu bilhete numa máquina da estação antes de viajar. O bilhete mesmo custa 50 cêntimos e depois você precisa carrega-lo com crédito suficiente para a sua viagem.

Então, em Lisboa tem uma companhia de metro, e múltiplas companhias de comboios.

A coisa louca é que o crédito que é valido para uma companhia não pode ser utilizado por outra. Assim você tem de carregar o seu bilhete com crédito para a primeira etapa, pois na estação seguinte tem de o carregar de novo pela viagem seguinte.

Se você pretende viajar primeiro de metro, tem de carregar para o metro, e só depois para o comboio. Não é possível carregar só uma volta para 20 euros e viajar muitas vezes com isso. Porque não? Não sei, isso não faz sentido.

Me lembro de que um dia um senhor aproximou-se de mim a perguntar "você conseguiu?". Eu não estava a entender. Consegui o que? Ah, comprar o bilhete? Claro que sim. Mas porque o senhor me faz esta pergunta? Ele era um homem de cinquenta-sessenta anos. Acho que era de Lisboa, então eu era turista. Então o que este cara quer? Quer dinheiro?

Logo percebi que ele não conseguia comprar o bilhete porque não é fácil compreender a máquina. Só queria que eu lhe ajudasse a comprar o bilhete. Que bizarro. Nunca antes vi um turista ajudar um residente porque este não compreendia o sistema de bilhetes.

a little help with bitwise operators

August 3rd, 2015

Binary numbers are easy, right? You just do stuff with bits.

But invariably whenever I code C I can never remember how to actually set a bit or test a bit, I keep getting and and or confused.

So I made a cheat sheet I can look up any time. These are the key ones:

All the others are available on the cheat sheet.