Every detective story or for that matter, spy novel, has the same overreaching goal: to give you an interesting puzzle to solve. A story is being told - you have to pay attention, and while you're taking notes you're also continuously assembling a repository of information and scouring it for relationships and patterns. And once you link two facts together, you've understood a part of the puzzle. A couple of factors determine why this is fun. First of all, the content of the puzzle should be fun, that is if you're putting together a puzzle of the Eiffel Tower, you will have fun if you like the Eiffel Tower. Then there are different ways of creating a puzzle, deciding what the pieces will be, how clear or faint the connections are going to be and so on. And finally there are different ways of giving you the pieces, at a certain pace, in a certain order. In as much as there are dependencies between the pieces, such that one will not help you without having a certain other piece, you can craft a complicated scheme. All these factors vary from story to story, some puzzles are fun, some are not.

This process of solving a puzzle is actually very intuitive. It's the same process by which we learn things. The difference is that when learning we're trying to understand an incidental truth, which isn't governed by any person's capricious decisions to make it so. And it could well be beyond your ability to solve. It may also be that no one has solved that puzzle. A man made plot, on the other hand, is a puzzle created specifically for our enjoyment, where the creator makes all the decisions, limits the scope appropriately, and deliberately sets the difficulty at a certain level.

A spy novel is always about deception upon deception, and as a reader you participate. Not only is the character in the story being deceived, you are as well. The author will be meticulous about telling you facts that point to a certain conclusion, but it's actually a decoy. Equally, he will mention facts that don't have any apparent use at the moment, but they will explain something further along, and if you're careful, keep you from walking right into a trap. It's really just a mind game.

The Ambler Warning was actually published in 2005, based on an unfinished manuscript Ludlum was working on when he passed away in 2001. In other stories of his that I've read he generally likes to run a chronological race, with very economical use of facts. This requires efficient reading, because rarely is anything said twice. The socio-political setting of the story also lets certain bits of information permeate from the surroundings. And when it comes to giving up information, Ludlum keeps you from seeing very far into the horizon. Most of the time he lets you guess and suspect things that won't be revealed for a long time yet. This puts you into a mode of forensics, trying to make what you know go as far as possible. It culminates in the economical resolution of some question, solving it with the least amount of facts possible. At other times he likes to keep you starved of new facts for a while, causing you to go back over what you already know and dig for new deductions in vain, before he finally drops something significant that has a big impact, but which changes so many existing variables that you get overwhelmed by the amount of information you have to process.

In The Ambler Warning we get a different kind of story telling. For one thing, only pieces of the timeline are accounted for, the rest isn't told. But it's enough to make sense of, again playing on the economical use of information. Secondly, the pieces are not told in chronological order. There is a leap from chapter one into what chronologically is the third chapter, but told in a different order.

This changes a lot of the assumptions of a continuous story, the fact book keeping takes on a different model. Since it's not chronological, you can't keep the chapters completely self sufficient, you have to make some modest references to chapter three in order to maintain a smooth reading experience. And as shown in the figure, this builds certain expectations about chapter three. But it does mean that you have to stack up certain facts from chapter two whose meaning you don't yet know, and won't, until you reach chapter three. This is a different mode of puzzle solving than I've seen in other Ludlum books, but it's pretty interesting.

The plot and the characters are actually a bit below par. I wonder if this is the influence of the editor who shaped an unfinished manuscript into a novel. And the conspiracy holds up nicely until the resolution. In fairness, this is very common. Authors build expectation with a meticulous use of detail, but then fail to make the conspiracy live up to those expectations. It's a difficult thing to do, and what they pull off is impressive enough.

You probably remember ethereal, right? Yeah, the gtk packet sniffer that has been around forever, but people like me who are pretty lame about networks tried it and didn't know what to do with it. In principle it's all very simple: you capture traffic and then you look at the data. Erm... kay. Well, in practice I had a tough time getting something out of that. Even a simple http request when split into an array of tcp packets doesn't really enlighten me that much when I have a little chunk of it here and the next there and so on. There are so many of them that I'm looking for the needle in the haystack.

This is a lesson in user interface design. ethereal is now called wireshark, and it's the same exact gui window that it always has been (with some minor incremental improvements). Here's the difference though. They figured out that data visualization goes so much further when you actually give a thought to representation.

The before picture, googled for an old screenshot of ethereal:

The window is split into three panes. The top one is the one you'll look at first, this is the list of packets we captured. Notice how small the scrollbar is and just imagine we have a large chunk of traffic in front of our eyes. Scroll up, scroll down, sort by protocol, whatever. In the other two panes you have info on the packet that is selected right now. In the middle one you have nicely structured info, in the bottom you have the raw data. This is the way to look at packet content. Btw tcp packets can be up to 1.5kb in size, so it's not exactly a sizable amount of content.

Gear up for the after picture:

Yeap, colors. Really, that's all I had to say. People underestimate what a difference it makes to have the same exact output in contextual coloring. The same data just jumps up at you if you know it's gonna be in red and you don't have to parse the whole chunk of text to get to it. Notice how the old version and this one have the protocol column just the same, but how much easier is it now to spot different types of traffic? It makes a world of difference.

wireshark has another brilliant addition that I don't remember from years ago. You right click on a packet, like the tcp packet selected in the screenshot above that belongs to an http stream, and go Follow TCP Stream. What you now get is a new window with aaaaall the packets that belong to this conversation assembled and you can observe the whole thing in one place. Fabulous. You can also filter the data (on message level, not packet level mind you) on host if you like. It's fantastic.

If you thought *that* was bling, this will really blow you away. wireshark does not only assemble http conversations, it assembles a lot more protocols. For instance, I saw a talk the other day where the guy was demonstrating how he sniffed a voip conversation and then reassembled the whole thing to a simple audio file with a few clicks!

These two features, packets colored by protocol and packet assembly, alone make wireshark about 200% more useful than it used to be. And think about what has changed here. It's still the same data. The only difference is how you represent it.

It's always fun to start coding something from scratch, because it has no faults yet, you get a clean slate.

Being there from the beginning of the software life cycle also means that you see the whole thing up close. Baby's first steps, first school play, it's an adventure. I've noticed something that happens to my perception, though.

In the very beginning I have a clear vision of what I want to do and how I'm going to realize it. But the code isn't there yet, and the code that has been written up to this point isn't ready either. So when I'm coding and testing I find a lot of bugs. Some are things I didn't realize at the outset, and some are just errors in bad code. Both get fixed, but in the throes of this process I'm very in tune with how raw this thing still is. It's unstable: push it two inches beyond where it's supposed to be and it self destructs.

At this point I'm excited about the project, because it's new and I'm in the middle of it, enthusiasm running high. But I'm also conscious of how unstable it is presently. So although I'd like to, I'm not eager to give it to people. I don't want to expose them to this that I very well know is unlikely not to crash while they use it. I want to protect them from that. I also don't want someone to come back with a desk and bash me over the head with it. It's great, it's just not ready to be used yet.

The perception also depends on the type of bugs I find. There are both incidental and structural bugs, although the latter feeds the former much of the time. If I've coded a similar kind of thing before, I probably did something really stupid the last time, and I may have learned from that. Unfortunately, the space of mistakes is huge, so by the time I've exhausted all the mistakes and I'm guaranteed that from now on I'm not making any more, I would have aged considerably. But at least I can get it right when I'm starting with a clean slate. That or I can hack it up and redo the structure if I'm not too far into the process yet.

So when I'm in the midst of development and the code is visibly unstable, it sometimes feels like this is not only the current station but also the final destination. It's always going to feel like this.

But over time something unexpected happens. I run the code 50 times, it crashes 3 times. When I start to close in on feature completeness and say 80% code completion the code actually starts to be dependable. Structurally the system is almost ready, and what remains is to fill in the blanks. If the structure is decent the incidental bugs start dropping off too. So it's turning into a piece of code I can actually use, just what I had in mind in fact. :cap:

Without quite knowing when or how I begin to gain confidence in the code. And when I bounce off a few pieces of bad input and a couple of unforeseen scenarios get handled gracefully, my mood almost points toward celebratory.

I know I promised you a curve. Well, I lied, I don't know what the curve looks like. :P But it's there, be sure of it!

When I was 15 I spent a few weeks at a camp in the summer. We were sailing. Up in the top right corner of Poland there's a region called Masuria, a region of lakes. It's pretty much the only place in the country you really wanna go sailing. It's vast and quite uncontaminated by urbanity and worse: industry. You'll find small picturesque towns scattered between the lakes and the way of life is to stay on the lake most of the day and set up shop at night in tents.

So where was I going with this? Oh yes, sailing is fun, but it's hard to fill 14 consecutive days to the brim with it. There was a lot of down time, and plenty of boredom as well. I didn't know any of the people and I felt quite alone. I was also the youngest. There formed cliques, but I wasn't in any of them. I shared a tent with a guy who was 18 and we didn't fraternize. And anyway he was busy with his girlfriend.

Toward the end of the camp it got a little better for me. I had nothing to do, but I noticed my tent partner started reading a book he'd brought with him. I wasn't big on reading back then, but when you're that bored, a book will do nicely. One time he was out I got curious. It was a nicely bounded volume and I started looking through it, reading a few words out of the middle. The characters appealed to me, I kept on reading. Before long I was caught up in the book.

I guess my tent partner was a little put off, but well he knew I had nothing to do and most of the time I was indeed not doing anything, so he just let me. Anyway, it was his book, he could read it anytime, and his girlfriend was more interesting to him. Somehow word got around that there's a good book in the camp and several people got interested. The camp was coming to a close and I started to worry that, especially with this new interest, I wouldn't finish it. But despite the hype I kept on reading it pretty much all the time, and most people aren't the type to rip an item you're holding out of your hands, so I was in control. On the trip back from camp I finished the book with about 2 hours to spare.

Unfortunately, I forgot the title and years later I remembered the incident and tried to figure out what it was. That turned out to be harder than I expected. All I could remember was that it was a story about prisoners of war held by the Japanese during World War 2. And I vaguely remembered the title, but not accurately. I also didn't know what it would be called in English.

At one point I thought it might be Lord of the flies, the title sounded close enough. But that turned out to be a children's book, and a bad one at that. Eventually I tracked down the story, called King Rat, which is a literal translation of the title in Polish.

I read it a decade ago, and when I read it today the story is still captivating. It's a tale of Allied troops being held in a POW camp in Singapore during the war, under strict Japanese rule. Their living conditions are difficult to imagine, and prisoners have regular afflictions of fever and disease, those that are surviving. Food is scarce, drugs are lacking. They know nothing of the outside world, nothing about how the war is going, no news of home.

These are the circumstances in which survival is a privilege. From a sociological point of view it's almost like.. a controlled experiment. Prisoners realize they cannot survive alone, so they form groups of 2 of 3, accomplices they can trust, and take care of each other. They share food, help the sick, and look out for trouble. Everyone is putting some food aside just in case it might get even worse.

And this is the world of the King. A lowly American corporal is the one man who knows how to survive, nay how to live, in these surroundings. The only one not malnourished, the only one not diseased, the only one not clad in rags. He is.. a business man. Those prisoners who have somehow managed to hide their last possessions are keen to sell them when they have no other choice. Watches, jewelry, things that can buy a wealth of food to last weeks. The King is the only man who can arrange such a trade. He trades with the outside, clandestinely. The Japanese overlords strictly forbid it.

And this is the gripping story of a not-so-distant past.

Tabs. Mouse gestures. User-agent switcher. Dedicated transfer window. Pop-up blocking and javascript abuse filtering. Integrated search box. Page zoom. Session saver.

Chew on those features. We'll be coming back to them.

Let me take you back in time to a year I like to call 1996. Those were the murky days of 28.8kbps dial-up modems and "personal websites" that said "Welcome to my website! (under construction) Here's my email address." It was also the prime of Netscape Navigator, the new browser that had taken the fast expanding internet by storm and become the successor to the venerable Mosaic. Netscape was about as ubiquitous as you can imagine, for all intents and purposes it was the world wide web. Anyone privileged enough to have access to the web (chiefly in universities) had Netscape running. Microsoft (already then king of the desktop), having famously discarded the internet as a fad, had nothing to offer and Internet Explorer in 1996 was a complete joke.

This is the climate in which a little Norwegian software company decided to launch their precocious new product: a web browser. It was very much the right time for that. The web browser was by no means an established product, it was a very fresh concept. Netscape was putting in all sorts of new features and had no competition (yet). So in 1996 Opera launched their Opera web browser 2.0 and the game was on. The project was a success: a couple of years down the road a lot of people knew about Opera. And it was cool to have a Norwegian company out there in the arena - one of the most hotly contested applications even to this day, the browser.

Of course, there were hard times ahead for Netscape. Microsoft made serious progress with IE and bundling it with Windows (adding to the ongoing internet revolution where more and more home users got connected) meant that Netscape's position was threatened. Little by little it was becoming apparent that Netscape was a dinosaur next to IE, which loaded quicker, ran faster and crashed less. Netscape was not blind to this, but their counter strategy turned out to be the cyanide pill in the cocktail. They decided to scrap the existing code and start a rewrite. And thus, give or take, we never heard from Netscape again. Over the next couple of years IE pushed out the old Netscape installations (and with no new releases, IE won by default), and basically captured the whole market. This was a time when IE actually was the best product. (I know how incredible that must sound.)

Where was Opera in all of this? Catching up, it would seem. But by 2000 Opera had caught up quite nicely and Opera 4 was a very slick browser. It wasn't as complete as IE, but it looked good and both loaded and ran faster than IE5. This seems like the first time Opera was in a position to start competing with IE. It had momentum, it had speed going for it, it had new features. Hot on the heels came Opera 5, and then Opera 6 in 2001.

This is where Opera set an important precedent. With the benefit of hindsight, knowing how incredibly difficult it has been for Firefox to unseat IE, things could have been different for Opera. I recall using Opera on and off in this period. I liked the product, I liked how lightweight it was and still worked just as well, but there was just something... off about it. It didn't quite feel right. As a user of IE, I didn't feel at home in Opera. The user interface was not just different, it was too different. Then and there, I realized that Opera would not be my first choice, in spite of everything it had going for it. Purely because of the user interface.

It wasn't just me. Opera failed to build a user base. It had adoption among technology enthusiasts, but it utterly failed to break into the realm of average users. We are talking about a browser with an attractive interface, with tabbed browsing, and faster page loading. Not only that, it was more solid than just about any Microsoft application: it just did not crash. IE5, meanwhile, was on a downhill stability slope where the crashes and freezes would only get worse and worse with the proliferation of pop-ups and various nasty advertising gimmicks that the web was becoming infested with. Opera handled this so much more gracefully. But Opera was #3 (still behind Netscape) and light years behind IE in user base, without really making progress.

In 2002 Netscape returned from the dead. The rewrite everyone had long since forgotten about was released into the open, under the name Mozilla. It had little in common with the old Netscape now, the rendering engine (Gecko) was new and the user interface had been replaced (phew). Meanwhile, it was IE's turn to stagnate, IE6 was released in 2001 just before Windows XP and there ended the trail. By 2003 I felt Mozilla was so overwhelmingly superior that I wrote an advocacy text in favor of it, to wean people off the (by now) awfully backward IE. Firebird became Firefox and a year or two later the popular Firefox revolution began for real - suddenly everyone and their grandma was using Firefox.

Today IE is the dinosaur next to Firefox. The dinosaur still dominates the market, because of the unfair advantage of being pre-installed. And for the sake of completeness, grab a can of Microsoft Anti-Competitive Practices and sprinkle the whole historical period generously. But it's completely obvious that Firefox has long since won in every dimension, on technical merit, and in folklore.

Browser market share is a notoriously contested metric, but the general consensus (to the extent that there is one), is that IE remains first, then Firefox, then Safari (they basically pulled an IE on the Mac), finally Opera at #4.

With the release of KDE 4.1 on Windows and Mac (expected in July), Konqueror (which seems to be gaining ground on Linux) will be available on all platforms. If KDE adoption on Windows goes well, which I think is entirely plausible, Konqueror (now with the webkit engine) could relegate Opera to #5.

What is wrong with this picture? Let's return to the features I mentioned at the start.

Tabs. Mouse gestures. User-agent switcher. Dedicated transfer window. Pop-up blocking and javascript abuse filtering. Integrated search box. Page zoom. Session saver.

Here is the big question: which browser was first to include these features? Opera. Opera. Opera. Opera. Opera. Opera. Opera. Opera. That's right, Opera prototyped all of these things. And it would actually take years before other browsers could be persuaded that these were good ideas. For heaven's sake, Firefox didn't include saving your tabs until version 2.0 in 2006! Futhermore, Opera has taken certain ideas from others and improved upon them. Firefox was first to save your passwords, but it works better in Opera.

Opera also predates every browser in common use except IE, its contemporary. And yet Opera has failed to make a real impact, why? It's not because the technology isn't good enough: Opera is still super stable and faster than anything else. Performance wise Opera completely dominates the embedded market, that should be sufficient proof. And it's not for a lack of ideas, clearly.

To put it bluntly: why did people rally around Firefox and not Opera? Is it because Firefox is technically superior? It isn't. Is it because Opera is closed source? No, I really don't believe the average user understands the distinction, or cares about it. Is it because of the extensions? It is definitely a great sales pitch, but again I don't believe that's the reason. "Power users" adore them, but does grandma really care? I sort of doubt it. Is it because Firefox is more "secure"? It isn't. Nor is it any less portable, you get the same Opera on every platform, just like Firefox. What's more, I don't think Opera has an unfair reputation on any of these points. It's just that Opera is the browser I'm going to use if this one crashes. It is "the alternative". It isn't the first choice.

Taking a stand to be different is bold, and deciding to make your application different and expect users to adapt is even more bold. It doesn't matter if your way is better. If there's one thing you absolutely have to know about software engineering, it is this: people hate change. If you are first to the party, you set your own rules. But if you're second, alas you have to play by someone else's. Opera is not competing for the theoretical masses of people who don't have a browser and are looking for one. They are ostensibly fighting to convert people from IE and Firefox.

I first heard about Opera through Norwegian media in 1999. I was excited about it and I've kept an eye on it ever since. I have made several attempts to adopt it (most recently because Firefox&adobe-flash is such an explosive combination, pun intended), but they've all failed. It is little things. How I cannot satisfactorily reproduce my Personal Bookmark Folder toolbar the way I have it set up in Firefox. Or how when I click to close a tab in Opera the tab that becomes active is not the one I want, I want the Firefox behavior. Or how I can't use extensions that I have come to expect (nowadays it is Firefox setting the standard for browsers). Or how when loading pages from slow servers they seem to get stuck loading and never time out, which doesn't give me a conclusive answer as to whether it's just slow or it can't connect. Or how when I want to configure something I can never seem to find that specific option amid a lot of other options I don't care about. Or how the caching mechanism seems to work slightly differently, so that I have to deliberately reload pages more often. Or how the fonts until very recently (before 9.50) have been scaled different and looked wrong. A dozen little things like that and it adds up to "I don't quite like this application".

When I've spoken to people about Opera I have never heard complaints about technical shortcomings. What people always say is "it's too different". And it was too different from IE when Firefox came along as it is too different from Firefox now. User interface is a very delicate problem. People's perceptions of how things are "right" or "wrong" are very subjective and entrenched, much like how in one city you feel at home and in another you don't. Opera has stayed true to their user base over the years by doing things slightly differently. I think this is also how they have taken themselves out of the running in the browser wars, despite having a highly competitive product at least since 2000.

That isn't to say they have failed. Opera Software is a thriving company and while they have 1% or less of the desktop market, they have made a successful land grab on the PDA/smartphone market with Opera Mobile, which is the same browser. While I find the proposition of putting a browser on a 200 pixel display is a horrible idea, I would pick Opera any day for that role.

Still, one could easily imagine a status quo where Opera is more of an influence. I think had they been more reluctant to redefine the browser this could have been the case. Ironically, most people will remember Firefox to have innovated the concept of tabbed browsing. Just like Apple "innovates" by recycling old ideas. By now Firefox has taken just about every good idea left from Opera and put it into a product and a form that people are more eager to use. That is to Opera's credit, and yet it is a bit sad that that Opera hasn't gotten the credit for it by attracting users.

I do actually think Opera has been converging on a lot of these little points of friction over the last couple of years, so it makes a stronger case than it used to. They have adopted some Firefox-bling that Firefox users expect to have, like an Adblock-lookalike, themes, about:config etc. Unfortunately, there is also more competition these days, from Safari (which sort of sucks, but not enough for people to rebel) and soon perhaps Konqueror.