How To Attract Great Software Developers, Part II: The Natural Habitat


“We have nothing that is really our own;
we hold everything as a loan.”
― Nicolas Poussin

In the previous installment of this series, I explained why many software companies attract charlatans instead of great developers. I summarized the problem by postulating the first rule of hiring great software developers:

To be able to attract truly great developers, a company has to be truly great itself

Thus, if companies can’t attract (or retain) great software developers they are most likely not great themselves. Great, from a great developer’s point of view, mind you. There are many companies out there who are probably considered “great” by ordinary people (and even investors), but this doesn’t mean they are a great place to be for great software developers.

Today, I want to explain what great developers look for when selecting a company. However, keep in mind that no two great software developers are the same. Still, in my experience, most of them share a common set of expectations.

1. Challenging projects that matter

The main goal of great software developers is working on challenging projects where they can demonstrate and hone their skills. Implementing a mission critical embedded system from scratch (like the mars rover) most likely is. Adapting an accounting system such that it’s in line with the latest tax regulations most likely isn’t. It’s the hallmark of great software developers that they always look for new ways of doing things and learning new techniques and technologies — that’s why they’ve become great, after all. When you look at a great developer, you see a snapshot of a life-long journey towards excellence.

The most attractive projects are projects that have top management or even better media attention. Don’t be fooled by the fact that most great software developers are introverts — as every human being, they still want to feel needed and also like to show off every now and then. This is a manifestation of positive pride: pride in workmanship.

On the perfect project, they would work together with other like-minded great developers, ideally recognized experts in the field from which they can learn and to whom they can look up to.

2. Freedom and the absence of friction

Great software developers are individuals who turned their hobby into profession. They want to be ever productive and spend as much time as possible doing what they love most: writing awesome code. Anything that distracts them from coding is thus abhorred. To them, having to fill in time sheets, write meeting minutes, apply for admin rights, attend regular status meetings is creativity-killing utter waste of time.

Great developers love to freely choose their development tools and when and from where they work. Forcing them to use a particular IDE or to commute daily in order to work on-site in a noisy open-plan office is diametrically opposed to what they look for.

3. Decent pay

Money does matter, even to introverts. But it matters less to great software developers than to average people. More important are challenging projects and a frictionless environment. By paying way-above-average salaries, even mediocre companies may be able to lure in great software developers every now and then. They will, however, always lose them, regardless of whether they stay or leave.

So there you have it: feed them great projects, give them freedom and keep bureaucracy out. Pay them enough that they feel comfortable. The latter is the easiest to achieve. Offering a steady stream of challenging projects that matter and providing a hassle-free environment is hard in the long run, especially for a successful company that grows due to its success. The bigger companies get the more they become driven by fear — fear to lose what they have accumulated in terms of wealth and reputation. They fight this fear by adding more and more creativity-stifling policies to maintain the status quo.

Thus, it’s the sad fate of every successful company to become bureaucratic and boring in the long run. Regardless of whether great software developers stay just for the money (at the expense of losing their greatness), or leave — the second law of hiring great software developers can’t be escaped:

In the long run, great software developers cannot be retained

How Long Is A Millisecond?

“Time is relative; its only worth depends upon what we do as it is passing.”
― Albert Einstein

Different trades and professions have their favorite units of measurement. For astronomers it’s light years, for farmers it’s acres, and inches for carpenters. For us systems programmers it’s clearly milliseconds.

Images from a video camera running at 60 FPS arrive every 16.7 milliseconds. An operating system’s time-sliced task scheduler might switch tasks every 50 milliseconds. During full-breaking, an anti-lock breaking system typically applies/releases breaking pressure every 75 milliseconds. Milliseconds seem to be perfectly suited for close-to-the-metal programming and for smoothly controlling systems.

But how long is a millisecond, actually? Sure, everybody knows that it’s one thousandth of a second, but I’m rather curious as to how much work can be accomplished in a millisecond. Is it, for instance, a waste of CPU cycles if a thread wakes up every 5 ms to poll a sensor?

People often use “dog years” to better understand the behavior of dogs. We can apply the same idea and use “CPU years” to better understand CPUs.

For us humans, a second is a very important unit of measurement as the human heart beats (roughly) once every second. Let’s use a fictitious modern-day RISC CPU that runs at a clock rate of one GHz and requires one clock cycle to execute an instruction as an example. If we view the clock frequency as the heartbeat of a CPU, then a CPU’s heart beats every nanosecond[1]. Thus, one millisecond corresponds to one million CPU hearbeats.

One million human heartbeats equal roughly 12 days (i. e. 1,000,000 / 60 / 60 / 24), a time-span in which a human being can get a lot of work done[2]. Likewise, for our CPU, a millisecond corresponds to 12 CPU days. A full second corresponds to — lo and behold! — 32 CPU years (i. e. 12,000 CPU days).

Now, you might be coding for an embedded system that runs at “only” 100 MHz, so in this case a millisecond is “just” 1.2 CPU days. Or, you might have a Raspberry Pi 4 running at 1.5 GHz whose Cortex-A72 CPU sports a 3-way out-of-order superscalar pipeline, in which case you can probably multiply the 12 days by three.

While not perfect, this “a millisecond is a dozen CPU days” rule-of-thumb is a nice reminder that on a modern CPU, a millisecond really stands for “a lot of time”.

________________________________

[1] A nanosecond is such a mind-boggingly short period of time that even the fastest thing in the universe (i. e. “light”) can only move one foot (30 cm) in a nanosecond.

[2] At least theoretically. Definitely not if you work for a large corporation and your schedule is packed with meetings.

People Patterns In Software Development: The Major Tom

“Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away.”
– Antoine de Saint-Exupery

In this day and age, especially in large corporations, you can find a special breed of software developers who just can’t KISS. For every problem, even the simplest, they dream up a solution that is so complex, so over-engineered that when you look at it, it’s next to impossible to tell what problem it’s trying to solve. When you delve down the layers of abstractions and technologies you soon get lost and forget where you started from.

In 2001, Joel Spolsky coined the term “Architecture Astronaut” to describe folks that love to complexify. However, according to his definition, “It’s very hard to get them to write code or design programs, because they won’t stop thinking about architecture”. To me, however, his definition of an Architecture Astronaut sounds more like a particular variant of a Non-Painter.

The individual I want to talk about today bears a strong resemblance to an Architecture Astronaut, at least on the surface. Contrary to the Architecture Astronaut, he does design programs and does write code — actually, he’s a quite prolific coder. However, what such a developer really does is building overly complex, not-from-this world systems and hence I call him “Major Tom”, in reference to the astronaut character “Major Tom” in David Bowie’s song “Space Oddity“. This song, “depicts an astronaut who casually slips the bonds of the world to journey beyond the stars”. Later in the song, it’s revealed that Major Tom is forever stuck in space and will never return back to earth.

Here’s an example that gives you insight into the thought process of a Major Tom, which can be helpful for diagnosing one.

Let’s assume you have two processes running. One of them (process A) depends on the other (process B). For reliability, process A wants to regularly check whether B is still alive. The initial and obvious choice would be to have process B emit a “live tick” message at regular intervals, potentially including a running counter. A Major Tom’s journey to the stars (and beyond) could look like this:

1. Let’s use a Unix pipe between the processes to exchange the live tick message.

2. But what if the processes are distributed across different machines? Pipes don’t work across machines, so let’s use UDP (datagram) sockets.

3. But with UDP, datagram delivery is not guaranteed, so let’s better use a TCP (stream) socket instead.

4. But what if the machines are based on different architectures that use a different byte order (i. g. big-endian vs. little-endian)? Wouldn’t the counter value be interpreted wrongly? Let’s better use a platform-independent data exchange technology like Protocol Buffers on top of it. This has the additional benefit of supporting backwards compatibility should the message format change and one never knows what additional messages need to be exchanged between process A and process B in the future.

This reasoning goes on and on, let me spare us the rest. In this hypothetical example, a possible final design could look like this:

N. All involved processes register the services that they offer (a “live tick” service, for instance) with a central service registry. Service consumers discover available services offered by service providers. A dedicated, load-balanced broker connects service providers and service consumers which exchange JSON objects via rest API calls over a triply redundant TLS connection.

Now, every software developer is — to some extend — familiar with such lines of thinking and sometimes also leaves solid ground. But while healthy developers stop early and descend back due to the pull exerted by the YAGNI principle as well as time and budget considerations, Major Toms only know one direction: up, up, up in the sky, which translates to more, more, more complexity.

APPEARANCE

Major Toms don’t have any special appearance. They come in all shapes and sizes.

PERSONALITY

Why do Major Toms have such a strong desire to inflate everything? Let’s explore the main drivers, which depend on the particular personality of the Major Tom.

One huge source for not-of-this-earth complexity is the boredom felt by a super-smart developer. I’ve observed many times that developers with a very high IQ have a hard time doing menial work, and sooner or later every task becomes menial to them. The only way to calm down their insatiable mind is to constantly feed it with complexity.

Another cause for cosmic designs can be software engineers with a strong scientific background, especially computer scientists holding PhDs. Some of these individuals are all about research and insist on always creating something novel. Keeping things simple, going for the absolute necessary, and using “mundane” technology is often beneath them.

Last, but not least, there’s a possibility that people create uber-complex designs for job security reasons. Especially Fake Surgeons are known for using strategies like this in order to maintain a massive head start over the rest of the team, thinking “if you can’t comprehend my complex code, you’re not going to replace me.”

RATING

According to the Q²S² framework, a Major Tom’s rating is 4/1/3/3.

(The second Q-rating is so low because of the incidental, unnecessary complexity that causes poor maintainability and poor performance. Hence, the value of what a Major Tom creates is of low, sometimes even negative value to the company.)

POLAR OPPOSITE

The Wolf

CONCLUSION

Major Toms have a passion for making mountains out of molehills but not for shipping production-grade software within budget, in time. Instead of adhering to the trusted XP principle of “doing the simplest thing that could possibly work”, they program in the future tense and constantly ask “but what if?”. The result is often something that resembles a Rube Goldberg machine that nobody (apart from the Major Tom who created it) is able to maintain.

Sometimes it’s possible to cure a Major Tom, and since they are smart and prolific, it’s always worthwhile trying to redirect their energy into more valuable channels. Here are some tactics that might work:

1. Explain that software development is a learning process, that great systems must evolve from simple to complex, just like an oyster grows a pearl from grain of sand, like a vase is made in traditional pottery.
2. Explain that a simple first implementation doesn’t preclude a more elaborate implementation later, once the need for additional complexity has been proven.
3. Explain that a simple design is not a sign of weakness but a sign of strength, that simple design is often the result of hard thinking and discarding of non-essential parts.
4. Explain that doing simple things extremely well can be as satisfying as working with hip technologies.
5. Explain that a company’s main job is to generate money to feed the hungry mouths of its owners and employees, so all successful commercial software development must be done within time and financial constraints.

If your attempts to ground a Major Tom are fruitless, it’s time to go separate ways. Fortunately, when confronted with your down-to-earth ideas, pathological Major Toms usually launch their jet packs and head off for another planet.

Circular Adventures IX: The Poor Ring Buffer That Had No Tail

“If you hold a cat by the tail, you learn things you cannot learn any other way.”
― Mark Twain

The typical use case for a ring buffer is a producer consumer scenario, where a producer sometimes produces data faster than the consumer can consume. Eventually, however, the consumer will catch up. This typical use case can be summarized as “short term buffering of bursts of data”.

But what happens if buffered data is not consumed fast enough and the ring buffer becomes full? Answer: adding another element will drop the oldest element, as can be seen by looking at the implementation of the ‘add’ method:

After the new element is added to the buffer, the head index is incremented (modulo ‘BUFSIZE’). If it now points at the tail index, the tail index is incremented (modulo ‘BUFSIZE’) as well, which effectively removes the oldest element from the ring buffer, making space for a new ring buffer element to be added. Here’s an example of this happening to a ring buffer that already holds 9 elements (why not 10? The head index is ‘exclusive‘ and points at the location where the next added element will go):

Calling ‘add’ in this situation advances the head and the tail index by 1:

This will be continue for every subsequent ‘add’. Here’s how the ring buffer looks like after six more calls to ‘add’:

Why am I telling you all this? Because there’s a ring buffer use case, where a producer endlessly produces data that is never consumed. I call this the “history buffer” use case where record is kept over the last N events, similar to how a flight recorder operates.

If a ring buffer is used as a history buffer, it is always fully loaded. Thus, the ‘add’ operation is unnecessarily inefficient, as the ‘if (head == tail)’ conditions is always true. So let’s get rid of it:

This naturally leads to another optimization: the incremented value of ‘head_’ is the current value of ‘tail_’. Thus, we just set the new ‘head_’ to the current value of ‘tail_’ and only increment ‘tail_’:

That’s even better, isn’t it? But there’s more! The tail is usually only used when consuming elements, which we don’t do for the history buffer use case. Why maintain it’s value at all? Plus, if we need it later (as we shall see shortly), we can always compute it from the head, since it is one element ahead of head (sounds a bit weird, doesn’t it? The tail is always ahead of the head):

Adding to the ring buffer has definitely become simpler and cheaper, but you probably ask yourself how to retrieve the history data stored in our ring buffer. As usual, there are many ways, but the one that I find most useful is through the standard library’s way of doing things, namely iterators. ‘begin()’ and ‘cbegin()’ simply return the tail, which is — as we already know — one element ahead of ‘head_’:

‘end()’ and ‘cend()’ just return ‘head_’.

In the simplest case, the iterators returned are forward iterators, which means that they only provide ‘operator++()’. As limited as forward operators are, they allow you to use many useful algorithms from the standard library, including:

But there’s probably another nagging question on your mind: what if the ring buffer is not fully loaded? What if, for instance, only 3 elements have been added so far to a 10-element ring buffer holding doubles? In this case, the 7 unused elements will hold default-initialized doubles, which can be easiliy skipped over:

Most of the time, this extra work is unnecessary in history buffer use cases, as a) the ring buffer is quickly filled and from this point on always fully loaded and b) default constructed elements don’t harm in typical history buffer use cases.

I like to call this particular variant of ring buffer “Manx buffer”, named after a peculiar breed of cat, called Manx, whose salient feature is the lack of tail.

Choosing a Manx buffer over a regular ring buffer may make sense in the following cases:

  • The elements added are never consumed but rather kept to have chronological, historical data.
  • The size of the element type is small, such that the cost of maintaining the tail is significant compared to the overall cost of adding an element.
  • Elements are added at a rapid rate.

An example of when a Manx buffer can prove useful is a data collection system for an aircraft that records the last 10 minutes of accelerometer readings for roll, pitch, and yaw as double values.

You can find my 70 lines implementation of a Manx buffer here.

How To Attract Great Software Developers, Part I: The Man In The Mirror

“Everything is energy and that’s all there is to it. Match the frequency of the energy you want and you cannot help but get that reality. It can be no other way”
— Albert Einstein

These days, almost every software company is looking for software developers. Not just any software developers, mind you, it’s great software developers that everybody’s craving.

Such companies know that — as has been confirmed again and again in many studies over the last decades — great software developers are easily 10 times as productive as average software developers. Productive as in “able to craft quality products”. It’s not about intelligence, nor the talent to give great talks, write awesome research papers, file patents, implement one prototype after another. It’s not about ranks, titles, grades, number of diplomas or PhDs. It’s all about the great developer’s ability to ship money-generating products on time. Those people are few and far between.

The sad truth is that the main problem of such companies is often not that they have a shortage of great developers — they suffer from far too many below average developers (again, in terms of money-generating productivity). These companies don’t ship awesome products, not to mention on time.

As a countermeasure, they recruit for great developers. What they often forget to do is take a critical look in the mirror. If they did, they would understand that what they try to do is akin to a hobo courting a beautiful princess. It’s nothing more than an exercise in futility! To attract someone attractive, you have to be attractive yourself. It’s that easy!

Realistically, the great developers that such companies are after are already with a highly attractive employer. Why should they change down?

Desperate as these companies are, they employ tricks that are as old as mankind: making themselves appear more beautiful than they really are. They dye their hair, they apply tons of makeup, but it’s just a cover up for something that’s ugly or not there at all.

They try to lure great developers in by telling them that they will be working in a modern workspace, be part of a global, agile team of the brightest minds, that they will shape the future of a whole industry, and so on.

Obviously, everybody, not just great developers, knows that this is pure BS and that a company which resorts to such measures must be a) far from great and b) really desperate.

The latter naturally attracts a special breed of developers. Con artists, who also apply tons of makeup to hide their shortcomings, which I usually refer to as Non-Painters; and so the vicious circle continues for the poor company. Birds of a feather flock together. Pretenders attract pretenders. But the converse is equally true.

Let’s conclude this installment by postulating the first rule of hiring great developers:

To be able to attract truly great developers, a company has to be truly great itself.

In the next installment, we’ll take a look at the hallmarks of great software companies and what great developers are attracted to.

Avoid Static Class Members As Much As You Can

“Most people want to avoid pain, and discipline is usually painful.”
— John C. Maxwell

Ah, static class members! They pop up in every C++ project. In most cases, however, their use is not justified. I try to avoid static members as much as possible, and so should you. Here’s why.

Let’s start by revisiting the mother of all static member text book examples. It demonstrates that by using the ‘static’ keyword, state can be shared among all instances of a class:

The reasoning in these text books usually goes like this: The total number of employees obviously doesn’t belong to a particular employee instance. Such information is shared by all instances, so let’s use ‘static’ here.

Don’t write code like this. It’s so flawed I don’t even know where to start. But I’ll try anyway.

If you believe that something is not the responsibility of an instance, you should not add this responsibility to the instance’s class either. Calling the static member function like so:

Is still a call on Employee, so it’s still a responsibility of the class. Instead, assign the responsibility to keep track of employees to a higher-level entity, like a Company or an Employer class, which manages Employees in a container, a std::vector, for instance. Such a higher-level class would provide a regular instance method that returns the total number of employees:

With this new design, in order to find out how many employees work for a company, you do the natural thing — you ask the company, not a static member function of class Employee. (Also note that in class Company, employeeCount is declared as ‘const’ to signify that it doesn’t update any data. There’s no such thing as ‘const’ for a static member function.)

What else is uncool about the original Employee class? Like every non-const static member ‘s_employeeCount’ needs to be defined exactly once outside the class, which is typically done in the corresponding .cpp file.

Further, manually incrementing and decrementing the count is cumbersome — the std::vector in Company will do this automatically for us. On top of that, the original design is not exception-safe. Imagine that initialization code in the constructor body of Employee throws an exception after the count has already been updated. The Employee instance will have never officially existed but the count indicates otherwise. You need to add extra exception handling code around the count increment to safeguard against such cases.

The original design is also sub-optimal as static members impede unit testing. After every unit test case, you have to make sure that the static data is cleaned up (or reinitialized) for the next unit test case. Since the static data is usually private, one often has to add extra public static initialization or setter methods to do the job, thus complicating the class interface even more. In general, static member functions can’t be mocked (at least if you’re using Google Mock) so it might be difficult if not impossible to simulate particular behavior of a static method to obtain code/branch coverage in a client component.

So are there any legitimate uses of static members in C++ classes? In my view, there’s only one: public (and protected) symbolic constants:

Private constants or private utility methods that do not touch instance data are usually better defined as anonymous namespace members within the class’ .cpp file — this avoids cluttering up the class definition in the header file with things that bear no relevance to the user of a class (aka implementation details).

But what about public utility functions that don’t use instance data? Shouldn’t they be declared ‘static’?

The official answer is probably ‘yes’, and there are compilers and tools (like clang-tidy) that suggest that you should declare a method ‘static’ in such situations. These days, however, I rather tend to ignore/suppress such warnings based on this reasoning: if a function is declared ‘public’, it’s part of a class’ interface. The fact that its implementation doesn’t touch instance data (today) is just an implementation detail that doesn’t need to be conveyed to the user of a class. I’ll pick a regular (virtual) method that I can mock over a static method any day:

This design clearly differentiates between interface and implementation and achieves loose coupling between the class and its clients. At runtime, clients can replace one implementation with another (e. g. optimized for speed, with caching, with logging, a mock and so on) without having to change the client code. All this would not have been possible had ‘factorial’ been declared ‘static’.

To sum it up:

  • There’s only one legitimate use-case for static members: symbolic constants
  • State shared by all instances of a class should not be shared in that class, but rather in an instance of a higher-level class
  • Static members prevent late (i. e. runtime) binding
  • Static data and static member functions make unit testing difficult