erwan's code blog

Sitting in a warm bath this morning, enjoying the peace before the storm, I came to realize a striking thing: software development goes very much like the law of the jungle. To survive in a changing ecosystem, software either mutates or is replaced. It has a darwinian growth.

Consider the ecosystem of a program as made of its running environment: software, hardware and fleshware, both users and developers, the features it must implement to fill its function, the extra features that make it more likable than other similar programs and the features it will have to implement in the future to meet emerging user needs.

Like a real ecosystem, a software ecosystem can remain unchanged for years. The business logic is well defined and users don't need to change the way they do whatever their business is. The volume of work remains constant. The developer in charge of maintenance gets acne upon hearing the word "refactoring". And so on. No change in the ecosystem allows for no change in the software. And that's fine.

But more often than not, software ecosystems are highly changing environments. Developers come and go and have very different opinions on how things should be implemented. The business evolves and requires new ways of working and thinking, and a brand new software logic. The volume of data to process grows and grows. A generation of new developers comes around and trashes the code beyond recognition (wow, we even have the equivalent of real-life cancer in this ecosystem!). All these natural events have one of two consequences on a program: either the program evolves or it dies. Much like a living organism :)

Ok, program evolution does not (hopefully) come from random code mutation. A developer is normally to blame. But this kind of controlled mutation is just a variant of the darwinian one, except more purposeful and efficient. In the end, the ecosystem still decides whether to keep the program alive or let it rest in peace. And how many time haven't we seen developers struggling to keep some code alive only to be beaten out by a competing program that just does the right things better. Programs rise and fall. Developers move on.

Considering this setup, no wonder that programs show signs of organic growth! Code evolve under the same darwinian rules as all things alive!

And developers are to programs what random mutations are to living beings. Great. A real kick to my self-esteem.

It seems hard to keep the motivation and engagement of a team of programmers at its top when their software's growth requires the team to expand. There is a fine line to walk there, along which striving to preserve individual creativity might be a key to maintaining balance.

Most software products grow out of the joint effort of a small number of developers, usually something like 3 to 5 persons. A small group like that can be a fantastic catalyst for individual energy. Everyone gets space to exert his or her own creativity. The product being only at its birth stage, team members are there for fun rather than for profit. There is a positive competition between team members since individual energy can focus on the challenge itself rather than on the hierarchical and managerial issues around it. This list could go on for a while. What I am trying to tell is that in my experience, the most creative and exciting development teams to work in are small ones.

That's an idea emphasized in agile methods and scrum who heavily believe in strongly focused, self organized and SMALL teams of developers.

In 'Peopleware', such teams are called 'jelled teams'.

When the number of developers in a team reaches a critical limit, somewhere around 5, the dynamic of the team changes. Roles are introduced, such as architect, lead-developer, CTO, boss. Decision making gathers within some roles and escapes others. An unhealthy competition grows when people strive to get those roles. People get bossed around. Developers feel they have no space left for their creativity. The fun factor tumbles.

Of course, team scale is related to product growth. It's because the software is growing that you need more developers. But if you insist on keeping them in the same team, you will get over the critical mass for jelled teams and loose the positive creativity in the team. I guess a way to avoid that would be to break overgrown software into smaller components and grow a jelled team around each of them. But it's easier said than done and can be hard to justify for management.

Team growth is not always directly related to code growth though. At places where management has taken over the engineers to decide over the product's future (which means at more or less every mature software company), management can cause teams to grow in completly artificial ways that are not respectful of the current balance of the jelled teams.

Let's take an example. A company has a jelled team of 5 that has been develloping a product over the last 5 years. The company also hosts a number of parallel teams working on related products. Management decides that the future of their product line requires a complete redesign of the product map, so they hire a number of software architects to decide on what to do. They hire those architects from outside the company because they worry that in-house developers promoted to the architect role would remain too loyal to their team and product and hence have a biased view on what to achieve. Those decisions make sense from their perspective, but out of mistrust for their most productive developers, this management has just given a death blow to its jelled teams.

Developers in jelled teams have given their souls to make their product a successful one. The very act of them doing so is usually what made the product successful, but that's something that non-technical management tends to be blind to. To maintain this level of engagement, you have to leave enough space for each developer to make design decisions and use his creativity. When you hire external architects, they will want to make design decisions. That's their job and that's what they enjoy doing. A conflict will occur between architects and developers on who gets to decide. Most likely, the architects will get the last word, because design is their role and the developers' role is implementation, at least in the management's simplified understanding of the craft of software development. Members of the jelled team will fill robbed of their freedom of creativity, and leave.

Management probably won't notice this chain of events. A few developers will leave. New ones will be hired. The new ones will do what the architects will tell them to do and everything will look good. But a process of natural selection has occured that will keep a certain kind of highly efficient and engaged developers away from this project. A few years later, the product will need a complete rewrite because no developer took on him to refactor the code, and no architect noticed the need.

All this boils down to the belief that successful software development is about getting the right developers. Some developers are just way more efficient than others. It has been proven in many ways and has grown into a widespread belief in the software community but it still remains somewhat of a taboo in many circles because it strongly implies that there is a kind of elit of software developers. It's unfortunate, because elitism has very little to do with what makes those developers be so productive. They usually do what they do for fun, seek no power and are little interested in career competition. To get those people, you don't even have to pay them much or to lure them with bright career plans. But you do have to offer them an environment in which they can create freely and together with people like them. Break this balance and they will leave.

Daniel J. Bernstein, the author of qmail and djbdns, is wellknown and respected for the exceptional quality of the software he has written. He recently wrote about the process of writing secure server software in this paper, where he shares his experience of implementing qmail:
http://cr.yp.to/qmail/qmailsec-20071101.pdf

This is worth reading!

Have you ever played this game: taking a large piece of paper, and sitting down beside it together with a couple of small children, and all together try to draw, say, a fairytale landscape?

If you are in it for the fun, you're gonna get loads of it.
But if your aim really is to draw a fairytale landscape, like those on fantasy books hardcovers, you are going to experience a great deal of frustration.

Now, a similar effect awaits you when writing software in a group: if you are a code perfectionist, you are going to be frustrated. Simply because normal developer teams are a mix of people with varying ambitions.

At some point, a developer has to let go his own feeling of responsibility toward the code, especially toward the code he has written himself. Some day, someone else will start changing your code, making it look weird and clumsy, introducing new ideas, taking charge of your code. It will happen and it is fine (as long as they don't break your regression tests).

But in my experience, letting go of your code is hard, especially code that you have polished during many hard-working hours. It's in human nature to care for what you create.

Once, during a conference long ago, I heard a developer call that 'ego-less' programming.

I am still trying to get better at that. I even wrote 'ego-less' on the corner of my desktop screen, to stare at it every day. But this is a typical example of self-refactoring: it takes time...

This evening I attended to a seminary given by Andy Hunt, the co-author of 'The pragmatic programmer', one of my favorite computing books.

He gave a one hour talk starting with the title 'how hard can it be'. Funnily enough, this happens to be the motto of one of my colleagues who likes to fire it each time some truly impossible assignment falls on us...

The talk was entertaining. A lesson in how to make professional presentations: concise slides, tons of jokes and a truly enjoyable speaker. Having read (and reread) (and read again) the said book, my expectations were high. So I got a bit disappointed at the slow tempo and low rate of fresh ideas. Still, it is honey to my heart to hear rehearsed what I so sincerely believe in...

The talk boiled down to: software development is truly really hard because it has become the center of the universe. Indeed, everything nowadays depends on chips, and more precisely on the programs running on those chips, and thereby on programmers.

Programmers have to translate into code precisely everything man has thought of, which implies that programmers have to understand in details a scale of things that stretches from the extremely technically arcane (bits,
hardware design, os guts and floating point arithmetics) to the extremely humanly arcane (laws, stock exchanges, poker rules, the inside of customers heads). Of course, it is impossible. That's why things get so hard.

Nonetheless, programmers have to cope with amazing complexity. And some of it is self-induced: bad habits, wrong abstractions, ill-adapted tools... To fight complexity, one has to strain to keep things simple, which starts by choosing the right tools.

This is a nice conclusion. Nobody would argue against it.

Yet choosing the right tools raises a number of issues, not all of them technical in their nature. Assuming you actually know which tools are the right ones for the specific problem you are facing, how will you convince others in your team to convert to them?

As Andy Hunt stated (free adaptation): "I can refactor myself a bit, but I really don't know how to refactor others". And truly, refactoring others is hard. Otherwise, communism would have succeeded, which it didn't. Quod erat demonstrandum.

So what would the pragmatic programmer do?
I actually asked Andy that question and was eager to get his opinion. His answer was: convince by example. Try the new tools, new languages, new abstractions on your own, as a side project, and show the result to your colleagues. Don't push them. Let them come to you.

This answer was really sweet to my hears, because as a matter of fact it is exactly what I have been doing at work for the past years. And it has worked well.

It is quite amazing how little software development really is about developing software.

While cleaning up a few old directories, I came upon a text I wrote on what must have been a particularly glad day at work. Taking the risk of sounding almost mystical, here it comes, naked and raw:

***BEGIN***

In the eyes of uninitiated people, coders may usually appear like boringly silent and introverted people, folded onto their inner world. For them, a coder is a statue with moving fingers.

Such a wrong picture, so far from the truth...
If only those people could catch just the slightest glimpse of the amazing landscapes a coder contemplates while exercising his art.

Concepts take a very real shape, structures change into elegant skyscrapers, bridges, buildings of all styles and sizes. Functions are like busy creatures moving around this city landscape, altering facades and forms, like alien-shaped architects of a virtual world.
It is a world in full creation that is spreading silently under the Coder's inner eye. Worlds being born and collapsing, designs spreading their curves up to the sky, fighting against each other, reaching compromises, strengthening each other in their embrace.

And then, there is beauty.
The unspeakable, striking beauty of perfected designs. Well balanced, smooths and gracious, simple like light itself, they redefine the meaning of harmony by their mere presence. Their utter simplicity is an hymn to the achievement of the minds behind, those clear-seeing minds who could extract meaning and order out of sheer complexity.

Coders are artists and gods at the same time. They create in a universe whose rules they decide upon. They can design the language that best reflects the landscapes they have dreamt of.

Achieved coders are a pure delight to see at work. Out of chaos they create harmony, out of obscurity, simplicity. Less experienced coders contemplate their art in awe. Behind their silent faces, tears of bliss are running: they can appreciate the beauty of the Code.

Those coders are of a particular species. Coding is more for them than just implementing someone else's new tool. To them, coding is a mind dance, an inner experience, a form of meditation, an initiatic path that leads beyond the skies and on which each step shall reflect harmony.

In a way, coders are monks on a spiritual journey.

***END***

As a software developer, one spends most of one's time thinking and writing code.

This process is recognized by many as closely related to the work of artists. It follows similar patterns: one needs a vision and tries to express it as code; Code changes and evolves through the pressure of new ideas and growing understanding. In the end code grows into a thing of beauty, something the developer sees with pretty much the same feelings a sculptor sees her art or a poet her poems.

A developer facing code does not just see a bunch of text files and directories. He sees a complex and beautiful mechanic stretching through layers of abstractions and functionalities, with both ugly defects and shining zones of near perfection. He sees the ideas beyond the code that expresses them. He sees those ideas changing and maturing. He sees a whole world.

There is a problem though: this whole world he sees resides only within his head.

Common artists produce material things that anyone can later look at and admire. Most artistic creations are by nature reachable to others. Not everyone may be able to appreciate everyone else's art, but at least they are aware of the art being there and can acknowledge the effort of creation.

Like any artist, a developer crafting his code would appreciate sharing the beauty of his craft with others, but there is nothing material to show. And where there is nothing to show, there is no acknowledgment to expect.

Most people do not even realize that programs might be beautiful. At work, most of the persons a developer works with see the code from a completely different perspective: as a product, as part of a process, as a black box to use and test, or as the mysterious fulfillment of a set of specifications. But seldom do they see the code itself.

Fellow developers are more likely to get a glimpse of the code, but in practice it seldom happens. Not all developers appreciate code beauty, nor might they see beauty in the same way.

So as a developer, I spend most of my time thinking about the code I write, maturing it, perfecting it, then sitting in front of screens and writing it, filled with a growing sense of joy and sometimes pride when managing to produce
code I sense as beautiful. And I have no one else to share this achievement with than myself, or maybe indirectly the developer who will next maintain my code, and if I am lucky one or two fellow developers having enough free time to look at my code, understand it and appreciate it.

It is frustrating that so much of one's energy goes into something that won't even be acknowledged.

Once a friend asked me what I am doing at work. A rather usual question, but I find the standard answers ("developing software", "writing code") so empty. So I told him something like "I am building a starship, a piece of really advanced engineering, but no-one knows about it".

A couple of years ago, I started releasing code that I was developing during working hours as open source. That was the first step on a journey that helped me clear my mind on a number of issues surrounding open source.

There are still people around there who look down at open source with a shade of fear and disbelief in their gaze, so I wanted to share my modest experience on the matter.

I am a corporate developer. Mostly. My free time is filled enough with real life activities so that I can't afford the luxury nor have the drive to continue developing at home. Yet I love coding. So I do it at work instead. I mention those private details so you understand that I am not one of those many half-blinded linux-worshiping microsoft-hating slightly fanatic open source zealots.

Yet I do believe in open source.

My conviction has grown out of one simple fact: open sourcing code makes code better.

That simple statement is the result of a rather complex mechanism, blending psychology, group dynamic and efficient knowledge sharing. But through the years, I have had multiple occasions to verify this statement.

There are a number of things happening when you prepare yourself to release code as open source.

First, you are going to release code with your name on it. And internet being as it is, what is released does not disappear. It stays available for everyone to see. Everyone: people living 12000km under your feet, people you will never meet, but also people who will start communicating with you, sharing ideas with you, building a social and professional network around you. People you will meet at courses and conferences. People who will become your colleagues. Head hunters. Your boss. Recruiters who will consider hiring you, or not, depending on what you just released.

To make a long story short, releasing open source code is not just about writing code. It's an important social activity with significant consequences.

So you don't want to mess it.

This leads to what may be the single most important effect of releasing open source code: you are going to be extra careful with what you publish, trying to write elegant, well designed, well documented code. You will try to follow standards to avoid getting flamed publicly. You will write tests to feel more confident. You will react quickly to bug reports so your stats looks good. You will refactor your code when you discover ways of improving it.

And in the end, your code will stand a higher level of quality, for everyone to see.

Now, compare that with what usually happens with code that is released only internally, close source. If you write ugly corporate code, google won't know anything about it. In most cases, your fellow colleagues won't even notice: they are too busy trying to match their own deadlines to take time to look at your code. That is, until you resign, move on to a better paid job and live them with the joy of deciphering your production.

Only your closest colleagues use your code, so you will cut on documentation, since they just have to raise their head and ask you when they wonder about something. They probably wouldn't even read the documentation if there was one: asking is so much easier...

Your future may be influenced by the quality of what you write, but only as long as you work for the same employer. And according to statistics, you are most likely to get another job within a couple of years anyway.

So while the social aspects of code release tend to improve code quality in an open source context, it can almost have the opposite effect in close source environments.

But enough with the social aspect of open sourcing. There is one more consequence to open sourcing that has a significant effect on code quality: your code gets used.

At your office, you may (if you are lucky) have a team of testers dedicated to hunting bugs in your code. But most of the time, the only tools you can rely on to improve your code are your own regression tests, your profiler, your code coverage tools and your own imagination. Also, you will run your code on a couple of computers running one operating system. And that's it.

When you release code as open source, it gets tested, reviewed, commented, discussed. People will send you patches to fix bugs they found. People will run your code on their fridge running operating systems they have hacked together by themselves. Your code will be twisted, ported, executed down to the very last instruction in the darkest area of the less used branch condition from the most forgotten subroutine. And every time a bug is found and fixed, your code gets better.

I could go on a while like this, but I suppose you got the point anyway.

Now to the skeptics around us:

Yes, developing open source code does take more time and resources than close source one. That's because quality requires time. And therefore, this time is not wasted.

Yes, some of this time will go to development that your company won't benefit of. On the other hand, you will get free testing and sometimes even bugfixes and new features. It's a choice to make.

Yes, your pointy-haired boss will have difficulties understanding why you should start publishing code, as open source, since he himself never did that when writing fortran for Big Inc International 30 years ago. Not to mention the fact that he just read an article in 'computer weekly' stating that open-source development may cause fingertip cancer. Be patient and explain about the gain in quality. Or change job. Or wait until your pointy-haired boss is replaced by an other one and try again.

A recurring argument against weakly typed languages is that the risk introduced by their lack of compile time type checking is so tremendous that no mentally sane developer should consider using a weakly typed language (such as Perl) to implement a piece of critical software (like, say, the guidance software of a nuclear warhead).

The common answer of weakly typed language zealots sounds like: static typing is just a special form of testing, and can be compensated with various methods for catching type exceptions at runtime, and they usually mention unit testing and test coverage (See Strong typing vs. strong testing by Bruce Eckel for a discussion on the subject). Aside from developer techniques, the language implementation itself provides some strong means of runtime checking, such as type inference, array boundary checks, etc. Every dynamic language implements a subset of those techniques.

One argument sometimes raised is that static type checker cannot catch all exceptions. Some exceptions will happen at runtime that no static type checker (at least in any of the major languages) could have foreseen, such as crossing array boundaries in C or failing in external code called inline from Python.

Those questions have been central issues at my work over the past year or so.

I develop a large financial accounting software that places pension saving into funds. The core of this software is written in Perl, which gives us all the advantages of high level programming, but at one cost: the lack of static type checking. The lack of types, in fact. Of course, Perl provides some type checking at runtime, especially on objects, but it is rudimentary.

The problem is: this software must not fail. If any technique can help us writing safer code, we try it. We use techniques such as systematic regression testing, test coverage, paranoid design, defensive programming, asserting everything, strict code convention, and more.

But after a while, I started wanting something to compensate for the lack of static type checking in Perl.

This need emerged from a fact: every subroutine in the code tended to start with a bunch of lines that simply asserted the subroutine's input arguments. Sometimes, similar assertions could be seen at the end of the subroutine, upon returning results.

That made the code heavy and hard to read. So I tried to move those assertions outside the subroutine's body, and place them into what in fact was a form of programming contract for subroutines. I would define a syntax for describing outside the subroutine a number of constraints on the subroutine's input arguments and return values. This formed a kind of contract. This contract is compiled at compile time into an anonymous subroutine (a bunch of code generated at runtime) that gets wrapped around the contracted subroutine. Each time the subroutine is called, its input arguments are checked against the constraints defined in the contract. The same happens when the subroutine returns. If an argument fails a constraint, a runtime exception is sent.

The idea has evolved under the pressure of practice and is currently mutating into a Perl module called Sub::Contract, hosted on sourceforge.

Along the way, I realised that I was in fact implementing in Perl5 something like the constraints or refinement types found in other languages like Perl6, Haskell or ML.

In fact, such contracts enforce a stronger form of type checking than static types, since contracts not only specify the type of arguments expected, but even their state.

Consider a function that takes one argument: a date, implemented as strings with the format 'YYYY-MM-DD'.

In Java (statically typed), this argument would have a type ShortDate, specified in the function's signature. In Perl (weakly typed) this argument would just be a scalar and you would have to explicitely check that it has a valid format. Or you would define a contract for this function, and specify in this contract that the functions expects one and only one argument and this argument must validate a constraint. An easy way to implement constraints is by using code references. In Perl, a constraint validating the date format 'YYYY-MM-DD' could therefore look like:

     is_shordate($_[0])

where is_shortdate is a subroutine that returns true if its argument is defined and matches /^\d\d\d\d-\d\d-\d\d$/.

Now, imagine that our function not only expects a date in argument, but further expects this date to be of the year 2006. In Java, you would have to write an assertion for that in the function's body. In Perl too. But if you use contracts, you would just refine the constraint into something like:

     is_shordate($_[0]) && substr($_[0],0,4) eq "2006"

Which saves us one line of assertion...

Those *contracts* have proved to be very useful for catching runtime exceptions at earlier development stage. Though to be really efficient, this technique requires that the developer write unit tests and uses code coverage quite systematically.

Curiously, I have not seen contract programming used as an argument in the 'static vs. strong typing' discussion. Hence this post :)

My point here is that I believe that a lack of static type checking does not make a language less safe, provided that you use proper runtime checks. Function contracts are one of them.

Perl is evolving into an interesting direction in this respect: Perl6 will have types and roles (a kind of constraint), all of which enforce safer coding. I won't have to implement pseudo-contracts into the language anymore, they will be implemented in the language core, better, cleaner and safer :)

We have heard of extreme programming and agile development, but I don't think I have heard anything about storm development. Yet it exists, and is widely practiced.

Most development methods strive to achieve higher degrees of coordination. Storm development is about lacking coordination.

Most development methods are about rationalizing, chosing the most logical way, contemplating facts, taking distance. Storm development is about rushing blindly into one direction, with your guts gnawing their way through your braincells.

So what is storm development then?

Well, how many times haven't you felt irritation at having to repeatedly go through the same manual steps, wishing to have a tool doing it for you? Or how many times did you have a tool, but one so badly designed or out of purpose that it made your work harder than having no tool at all? The same for a code api?

When placed in a situation like that, there comes a time when you as a developer have to act and solve the problem and (re)write the piece of crap. For some developers it will happen at a very early stage, for others take more time. And for some it never will: they are those who find it easier to twist their mind than to straighten their world.

And now, confess: in which state of mind were you when you started (re)writing the aforementioned piece of crap? Yes? remember the cold fury? the deadly sense of purpose? the rush of adrenaline?

There it is!
You were storm developing.

Storm development is a developer's primal scream of rage against abherent obstacles. It grows out of accumulated frustration and is characterized by a tremendous albeit usually shortlived burst of coding energy.

Surfed properly, this wave of energy can bring rapid improvement to a project.

But most of the time, the whole momentum of it gets lost because the developer didn't take time to think out properly where he was going before wrecking havoc on the code.

In the past years I have become better at recognizing in myself the symptoms of a storm development crisis before it happens. That's a powerfull tool, really. When I know I am close, I can still take a bit of distance and try to plan for it. That makes the ride more likely to succeed.
And sometimes I may even realise in time that the storm would lead nowhere, manage to calm myself and avoid wasting time!