Playgrounds

I can still remember the day when two colleagues had an argument about a piece of code similar to this one:

    while (i-- > MAX_VALUES) {
        int k = 0;
        // Do some computations that
        // modify 'k'.
        ...
    }

While one of them argued that ‘k’ was initialized upon every loop iteration, the other claimed that the variable was only initialized (to zero) once. After almost an quarter of an hour, they wanted to hear my opinion on this matter.

I didn’t tell them the answer, but instead asked: “Why don’t you just try it out? It shouldn’t take more than a minute…”. They looked baffled, so I did it on my machine while they were watching. Granted, it actually took me two minutes, but, hey, I was not used to work in front of such a big audience.

legobricks.jpgDuring a typical development day, lots of questions arise. Some are simple, some are not so simple and some of them are really tough. In many cases it is necessary to look up the missing information in a manual, in other cases it is best to consult a friend. But very often, one should just try things out in a safe place. That’s learning by doing, or — ideally — learning by playing, which is probably the best way of acquiring knowledge.

Playing should be fun, but at the very least it has to be painless — otherwise it won’t be done. If you have to close down/restart your IDE and/or have to go through an eight-step wizard to create a toy project before you can start, you wont do it. Experimenting shouldn’t be disruptive — it should have as little impact on your main work as possible. So what we need is safe place that is easily accessible and fun to work in — a playground.

Let me tell you about my solution, my playground. A virtual desktop manager (VDM) is at its center and I wouldn’t want to live without it anymore.

Normally, you only have one desktop, but by using a VDM you get many, as many as you desire. Most Linux desktops like KDE and Gnome already support virtual desktops out-of-the box and when I’m doing development on a Windows machine, I use VirtuaWin (virtuawin.sourceforge.net).

The main benefit of VDMs is that they allow you to separate different activities: you can go from one activity to another without closing/reopening applications; restoring context is effortless: just press the corresponding shortcut to switch from one desktop to another.

For me, four virtual desktops are sufficient: desktop 1 is for my main development work, desktop 2 for any urgent, higher-priority interrupts (like fixing the build breaker that I checked-in last night — damn!), desktop 4 for mail, instant messaging and Internet browsing. Desktop 3, if you haven’t guessed it by now, is my playground.

On desktop 3 there is always a terminal waiting for me and its current directory points to ~/pg, short for ‘playground’. Within this directory there are several subdirectories:

    bash/
    c/
    cpp/
    flex/
    java/
    make/
    perl/
    python/
    qt/
    sqlite/
    tcl/

Each of these directories contains various code snippets, results from experiments that I did to find out certain aspects of a programming language, tool or environment.

In most directories I keep a write-protected file named ‘pg’, which serves as a default template to save me from having to type the same boilerplate code over and over again. As an example, here is the ‘pg’ file of the ‘cpp’ subdirectory:

    #include <string>
    #include <vector>
    #include <algorithm>
    #include <iostream>
    #include <cassert>

    using namespace std;

    int main(int argc, char* argv[])
    {
        return 0;
    }

With a playground like this in place, tackling the loop initialization problem becomes easy:

    cd cpp              # Change to cpp playground.
    vi loop_init.cpp    # Open a new file in editor.
    :r pg               # Read in playground template.

    // Modify main
    int main(int argc, char * argv[])
    {
        int i = 42;
        while (i-- > 0) {
            int k = 0;
            cout << k << " ";
            k++;
        }

        return 0;
    }

    :w                  # Save file.
    g++ loop_init.cpp   # Compile.
    a.out               # Execute and view results.

Actually, it’s even easier, as I have defined some shortcuts that let me compile and run code from within my editor; that is, without having to alt-tab to my bash console.

But this is just my playground, a place that gives me lots of joy. Maybe you work with Microsoft’s Visual Studio or Eclipse. In that case you could place links to template projects on your desktop; or, you place batch files in your playground directories that launch a new instance of your IDE and load a template project. Set-up your playground such that it is a fun place for you, but make sure you have one.

Like I said earlier, I keep my experiments for future reference and I suggest you do so, too. Over time, you not just improve your programming kungfu, you also build-up a valuable knowledge repository.

Single-step your code!

watch_your_step.jpegI was so proud: after I had gotten rid of some minor compile-time issues (ie. typos), my unit tests ran over my newly written code without any errors. Granted, the changes I made comprised less than 500 lines, but still, it meant something to me. Feeling happy and content, I hummed R. Kelly’s “The World’s Greatest” while carrying on with other work.

A few days later, I wrote some black-box tests and — to my big surprise — I got a couple of “fails”. After some debugging, I found more than five bugs in the code that had passed my unit tests so nicely. I was completely puzzled. What went wrong? Why didn’t my unit tests catch these trivial bugs?

As it turned out, I forgot to register my test code with the CppUnit test framework, so my tests were not executed at all! Once I had added the missing line

CPPUNIT_TEST(TestDemodulationCoupling);

to my test suite class, all five bugs surfaced in an instant. I was so angry! My first reaction was to curse CppUnit: with JUnit this would not have happened. I would have used the @Test annotation and my test would have been auto-registered — unless I had forgotten to tag it with the @Test annotation…

Later that day, I realized that the actual mistake was a violation of Steve Maguire’s powerful principle: “Step through every line of code that you added or changed with your debugger”. Had I set a breakpoint in my code, I would have seen that it was never executed.

Years ago, I used to be a passionate follower of this principle, but somehow unlearned it, largely — I presume — due to the rising unit testing hype. Don’t get me wrong: I think that unit testing is great (and black-box testing is great, too), but it is no replacement for single-stepping through your code.

Reviewing your own code is good, but actually stepping through your code is much cooler. The cursor showing the next statement to be executed focuses your attention and you really experience the program flow instead of having to make guesses about it. Further, you have all the data available and you can even modify it. You can invoke functions from your debugger (e. g. ‘call myfunc()’ in gdb), play with different combinations of parameters, member variables and the like, re-execute just executed code without restarting the debugger by setting the “next statement to execute” a couple of lines up. Probably the biggest benefit is that you get a deeper understanding of your code: maybe you step over a library call that works as expected but takes two seconds to execute; or you observe that you unnecessarily visit the remaining elements of a collection after you found what you’ve been looking for — no unit test would give you this kind of insight.

Often, it is difficult to unit test for certain failure causes, like malloc() returning NULL on out-of-memory conditions:

if ((p = (char*)malloc(2048) == NULL) {
    // Handle out-of-memory.
    ...
}

How would you unit test that? Such error handling code is usually left untested and is the reason why so much software crashes under heavy load. While you’re in a debugger, testing is easy: just set the “next statement to execute” to the error-handling code (right before stepping over the call to malloc), step through it and convince yourself that it works as expected. Again, how would you unit test that? Answer: factor out the error-handling code:

void HandleOutOfMemory(/* context */) {
    ...
}

if ((p = (char*)malloc(2048) == NULL) {
    HandleOutOfMemory(/* context */);
}

Now, you can call your error handling code from your unit tests. Still, testing the code by using the debugger is easier, doesn’t require any context set-up and gives more insight.

It helps, of course, if you write your code such that debugging is as painless as possible. A line like this is fine, of course:

double convReading = convertSensorReading(Sensor.current().reading(),
    scalingFactor());

but writing it like this is (probably) more readable and you can inspect (and alter) intermediate values in your debugger:

Sensor currSensor = Sensor.current();
double currReading = currSensor.reading();
double convReading = convertSensorReading(currReading, scalingFactor());

If you think this is too much typing, get better at typing and/or get yourself a better editor. If you think this wastes code, bear in mind that we don’t live in the 1970s anymore. If you think that you can always step inside convertSensorReading() and inspect/change the parameters there, you are right, at least as long as you have access to the source code of the function you want to step into.

Macros are bad since you cannot step into them. Use them only if you have no other choice; instead prefer (inline) functions and template functions: they come with the same efficiency advantages and you get type-safety and debuggability as a bonus.

And, speaking of the preprocessor, stop using #define’d symbolic constants: all preprocessor symbols are inlined during the preprocessor phase and I don’t know of any debugger that can resolve their values. Instead, use enums or, even better, const variables:

#define MIN_COUNT 23        // bad
const int MAX_COUNT = 42;   // good
...
if (MIN_COUNT <= count && count <= MAX_COUNT)

Mouse over MIN_COUNT in your debugger and you will see nothing; mouse over MAX_COUNT and you will get “the answer” ;-)

Automated unit tests are great, but stepping through your code gives quick feedback and a lot of insight into what is happening at run-time. Sometimes, hard-to-write unit tests can be avoided by consequently following the “step through all of your code” paradigm. As a simple guideline write unit tests — if you like — before starting with the implementation. Then single-step your code by executing your unit tests in a debugger and watch your step.

The Answer To The Last Question

coke_can.jpgToday is towel day, but due to higher priorities I have to celebrate this important day all by myself. I can’t make it to Innsbruck this year, but I swear to you that I’m wearing my towel around my neck while I’m typing this blog entry, which I dedicate to Douglas Adams.

Few people know that his idea about “The great question of life, the universe, and everything” (to which the answer is, as everybody knows, “fourty-two”) was in fact a little bit inspired by Isaac Asimov’s great short story “The Last Question“, where generations after generations build more powerful computers to find out how to reverse entropy and thus prevent the universe from becoming an infinite starless nothingness. “The Last Question” is a great read with a surprising end. I won’t spoil it, don’t worry.

While it is impossible for humans to stop “real” entropy from increasing (let alone reversing it) it is certainly doable in the software world. But how?

It’s not by carrying out the big refactorings and redesigns that nobody wants to do and that no profit-oriented organization can afford: for months valuable resources are so busy cleaning up instead of implementing cool features customers are willing to pay for. It’s the small stuff that counts: the teeny-weeny improvements that you do on a regular basis. Like James O. Coplien said: “Quality is the result of a million selfless acts of care”

I very much like Uncle Bob’s boy scout rule analogy: “Always leave the campground cleaner than you found it”. This principle is helpful for life in general and software development in particular. If the system you are working on is a complete mess, don’t resign. Even if you just improve a comment or rename a badly named variable, you have made the system better. Then, if everybody acts like you, software entropy will be reversed.

Masters, Not Managers

plane.jpgIf you watch one of those famous chefs on TV, you get the impression that cooking is both, fun and easy. A chef knows instinctively how to combine spices and how long the meat has to stay in the oven to be just perfect. That’s a clear sign of true mastery: masters have become one with their tools, materials, and processes; to a layman everything they do looks so simple.

What the audience usually fails to realize is that these chefs are not just great cooks and entertainers: they are also entrepreneurs, managers, and coaches. They usually run at least one big gourmet restaurant, they do the marketing and goods procurement, they write books, they hire people and train apprentices such that they someday can become great cooks, too. To sum it up, they are experts at their craft and possess other important skills at the same time.

Contrast this to a traditional software company, were you have managers managing people and developers doing the programming. Managers usually don’t do any coding and probably haven’t done much coding in their life. Most likely, they are the least-qualified to write solid, production-quality code. And yet they are in charge of strategic decisions, including hiring new developers. A master chef, however, is the most-qualified person to do the cooking and because of this is the most-qualified person to make strategic decisions.

In my view, a traditional non-coding software manager is a relic of the industrial age. The obvious waste is that such a manager does not actively contribute to the main asset, ie. working software. But there is more: not being a skilled software developer with first-hand up-to-date experience, a manage-only manager will face difficulties in many areas: first in finding and selecting talented people; second, in being a mentor and role-model for developers and third, in convincingly educating upper management about intrinsic software-related problems. Especially the latter weakness is a constant source for disappointment and grief in many companies.

The classic apprentice/journeyman/master model has proven timeless over centuries. It promotes people who master their craft and have the right management and social skills. Possibly the biggest advantage is that in such a system managers (”masters”) are still allowed to pursue their craft. Being able to do what one loves to do the most is good from a motivational point of view and leads to continuous dissemination of knowledge and best practices.

The Code is the Design, Baby!

What exactly is the “design” of a software system? Is it a detailed UML diagram? Is it 300 pages of prose? I recently came across an article from 1992 by Jack W. Reeves, mentioned in an interview on the subject of TDD; in his article, Jack argues that the design is actually the source code itself.

What!? The code is the design? This proposition is surprising at first, but not really if you think about it: only the tested and debugged code is a precise description of how the system should behave, how it should be built. And the process of “building” a software system is merely the feeding of the code to tools like make, Ant, compilers and linkers. Thus, “designing” software is hard, building software is straightforward.

Often, people demand that the design should be a UML diagram, something that is easy to read and comprehend by anyone. However, for any non-trivial system, the diagram(s) would become at least as complicated as the real code. It’s always the same: you start out with a couple of classes that have only a few relations, and everything looks nice; but when you finally have 200 classes you suddenly cannot see the forest for the trees anymore. UML is good for highlighting certain parts and aspects of a system, but a complete specification, one that can be handed down to the “code monkeys” is a myth. Even the author of the most successful UML book ever, Martin Fowler, is very skeptical about UML as a blueprint design tool. As Fred Brooks pointed out long ago: “software is not inherently embedded in space and has no ready geometric representation” (like hardware or buildings do).

There are technical issues, too. How do multiple “designers” collaborate on a UML design? How do they create parallel versions and merge conflicts? How do they keep track of variants? Countless tools exist for designs based on plain-text (aka. source code); next to nothing exists for UML.

Why do so many folks fall prey for this “everything-has-to-be-designed-in-UML” idea? Some (most likely managers or sales people) simply confuse programming with typing. Others (most likely developers that never got the hang of programming), want to introduce a white-collar/blue-collar developer system. Yet others think that there must be a method (in this case UML) that would render a complex system in such a way that it can be understood fully by anyone (including managers, sales people and developers who never got the hang of programming).

The design of a modern airplane comprises hundreds of thousands of pages. Nobody can and must comprehend everything. But even laymen can look at the 3D model and understand a lot: ah, this is where the engines are, and yep, here we have the anti-lock brake system and, boy-oh-boy, here we have the flight controller.

Such design overview diagrams are very important for comprehension and communication and no source code can ever replace them. It’s important to have such a diagram (call it high-level design or system architecture diagram or whatever you like). If you want, you can draw it in UML but I suggest you don’t: UML is way too boring. Have a look at the Google Android architecture diagram — it’s shiny, it’s been reduced to the absolute minimum, it’s freakin’ awesome!

Is there a need for designers, anyway? I definitely think so! You need somebody who takes care of the big-picture technical decisions, like the high-level composition of a system, the coding conventions, the hardware decisions and so on. Let’s call this person the systems architect. And, of course, you need somebody who takes care of the look & feel, the user-interface, a Steve Jobs kind of person. And, last but not least, you need many small-to-medium scale “designers” who iteratively detail the “design” through so-called “programming” such that compilers, linkers and the like can build the rest easily.

Poor Man’s DIP

Sometimes a lower-layer component needs to invoke a service on a higher-layer component. Consider, for example, a timer component (T) that periodically calls a handler function in a user-interface component (U). Component T is probably part of the OS kernel and thus clearly “lower” than component U.

In this setting, there is an upward dependency from T to U; such upward dependencies are undesirable, at least if they are bound at compile-time. Implemented naively, there is a hard-coded call to the UI component like this:

// file: OS/Timer.c

#include "UI/UIManager.h"  // Bad!
...
static void OSTimerNotify(void) {
    ...
    UIManagerNotify10msTimeout(); // Bad!
    ...
}

Dependency lines that point up in a component diagram are not just ugly: they denote that the lower-layer component cannot be independently reused and tested.

The classic dependency inversion principle (DIP) is usually applied to solve this problem: instead of having a hard-coded function call in the timer to the handling component, the timer calls back on a function pointer that is set to the timer-handling routine in the initialization code of the higher-layer component:

// file: UI/UIManager.c

#include "OS/Timer.h"

static void UIManager10msTimout(void) {
    // Do something ever 10 ms.
}
void UIManagerInit(void) {
    ...
    OSTimerSet10msCallback(&UIMananager10msTimeout);
    ...
}

// file: OS/Timer.h

typedef void (*TIMER_10MS_CALLBACK)(void);

// file: OS/Timer.c

#include "OS/Timer.h"

static TIMER_10MS_CALLBACK s_10msCallback;

void OSTimerSet10msCallback(TIMER_10MS_CALLBACK callback) {
    s_10msCallback = callback;
}
static void OSTimerNotify(void) {
    ...
    if (s_10msCallback != NULL)
        (*s_10msCallback)();
    ...
}

Note that there is still a T to U dependency, but now this dependency is only present at run-time, which is OK, as this doesn’t hinder reuse and testability. The U to T compile-time dependency is quite natural and doesn’t violate any design principles. So, the undesirable compile-time dependency has been successfully inverted. The classic DIP recipe looks like this:

1. In T export a callback interface
2. In U implement the callback interface
3. In U (or some init/startup code) register the implementation with T
4. In T call back on the interface

When you are working in a constrained environment like embedded systems, you often cannot afford the memory and performance overhead that accompanies such late (run-time) binding, so you might try what I call the “Poor Man’s DIP”: simply export a “callback interface” as a function prototype and “implement” it by defining the function in the upper-layer component:

// file: UI/UIManager.c

#include "OS/Timer.h"

// Implementation of the call-back interface.
void OSTimerNotify10msTimeout(void) {
    // Do something ever 10 ms.
}

// file: OS/Timer.h

// Call-back interface.
extern void OSTimerNotify10msTimeout(void);

// file: OS/Timer.c

#include "OS/Timer.h"

static void OSTimerNotify(void) {
    ...
    OSTimerNotify10msTimeout();
    ...
}

This pattern gives you most of the advantages of the classical (run-time bound) DIP but doesn’t incur any overhead. It can (and should) be applied whenever there is a dependency from a lower-layer component to an upper-layer component that doesn’t need to change at run-time but stays fixed throughout the lifetime of the application.

Breaking the Limits!

running.jpgHave you ever heard about Cliff Young? In case you haven’t here is his story…

For those who think that running a marathon is not challenging enough, there is the Sydney to Melbourne ultra-marathon: 875 kilometers across the hot Australian desert. This race is obviously only for the best of the top athletes and it usually takes them six to seven days to complete. Having gone through the pains of preparing and running a “normal” marathon myself I can only try to imagine what a race like that means.

Despite of these apparent tortures, in 1983, a 61-year-old farmer decided to take part in this race, wearing an old army jacket and gumboots. Spectators were worried that this poor guy wouldn’t survive the first hours of the race but when he left the stadium they gave him a big cheer anyway.

The race started and as expected, Cliff lagged way behind. After 20 hours, the first runners arrived at the camp, where they took a bite, got a massage and slept for about two hours (they typically stopped for a four hour break for every 20 hours of running). When Cliff arrived at the camp three hours later, he didn’t stop — he just continued to run. When a reporter asked him whether he didn’t want to have a break like the other athletes, Cliff responded that he was here to run and not to sleep.

He ran and ran. On the third day he decided to “sleep” for 25 minutes. Believe it or not: he arrived at the finishing line in Melbourne almost two days ahead of number two.

Amazing story, isn’t it? There a various slightly different versions of this story out there, but what they all have in common is that they stop here. What usually isn’t told is that the next year, 17 out of the 18 runners even exceeded Cliff’s record — all of a sudden, they had realized that it is possible to run the race without sleeping at all.

Cliff Young won the race because he challenged existing limits. Not really hard limits, like laws of nature, but arbitrary limits.

So the question is this: is something truly impossible or do we just think it is impossible because we haven’t tried before?

Regular Expressions — Sweetest Poison

It’s amazing how much time you can save by using regular expressions; it’s even more amazing how much time you can spend getting them to work correctly.

Because they are so powerful and easy to use, regexps can easily be misused, for instance by applying them to problems that are not “regular”, that is, where balancing is important:

    if (a > b) {
        ...
        if (x > y) {
            ...
        }
    }

Parsing problems like this are not suited for a regular expression matcher, as you need to retain state information and regexps simply cannot keep track of which blocks or braces are open or closed. In cases like this, what you really need is a parser. Period.

Alas, often people can’t be bothered writing a true parser, even if lex/yacc-like tools greatly simplify the work. And I’m guilty of this myself. Years ago I wrote a profiling tool for embedded systems. Since the embedded C code that I wanted to profile had to be instrumented (each function required enter/exit logging calls to get out the execution timing data) I needed to write a tool to do the job. I was not particularly interested in this job — hacking the actual performance analysis code was much more fun — so I decided, well, to go for a heuristic “parser” based on regexps.

In less than one hour I had cobbled together a little script that seemed to work fine. Over the next couple of months I had to spend endless hours fixing all the nasty corner cases; even today it doesn’t work in all circumstances! But I’ve learned my lessons: don’t use regexps when you need a true parser. Again, period.

But even if the problem is regular, people often define regexps sloppily. Look at the following example that checks if a .cfg file appears anywhere in a given string:

    while (<>) {
        print “Found: $&\n”
            if /\w:\\(\w+\\)*\w+\.cfg/;
    }

So let’s see what we’ve got here. We are obviously looking for a Windows-style absolute path: a single drive letter, followed by a colon and a backslash, followed by n optional directories (each of which followed by a backslash), followed by a mandatory filename that has a .cfg extension. Looks really neat…

These are the regexps people love to write and I don’t know how many times I’ve had to fix one because of this pathological “simplicity”. It might work today, but it is far from future-proof. Sooner or later the surrounding context will change and this regexp will match much more (or much less) than was intended.

Here a some of the major shortcomings:

- Using word characters \w is way too restrictive. According to the Windows long filename specification, a filename may contain any UTF-16 character, but for all practical purposes \w is really only a shortcut for [a-zA-Z0-9_]. If a filename contains a blank or umlaut, the expression won’t match anymore.

- Actually a corollary of the previous item: you cannot have partial relativity within an absolute path, e. g. C:\files\services\base\..\items\main.cfg would not match because the \w character class does not allow for dots.

- The regexp is not aligned on a word boundary, which means that if your editor happens to create backup files like C:\config\user.cfg~ they’ll match, too.

Often — but not always — using regexps means striking a careful balance between accuracy and convenience. It makes little sense to implement the complete Windows filename spec in a regexp. But investing a little energy to tighten them up usually pays off in spades. How about this?

    while (<>) {
        print “Found: $&\n”
            if /\b[a-zA-Z]:\\([^\\]+\\)*[^\\]+\.cfg\b/;
    }

At the cost of being only slightly more difficult to read, this solution is much more resilient to change due to the use of some good practices. First of all, it is easy do define a set of valid drive letters, so I used [a-zA-Z] instead of \w; second, the whole regexp is aligned on word boundaries, which means no more regexp over-/underruns and third, by stating that everything between separators (backslashes, in this case) is a series of non-separators we won’t run into “strange character” problems.

Next time you write a regexp think this: “I know that by using regexps I’m saving hours of development time, so I can afford to spend another 10 minutes to make them more robust”.