How to know if a docker container runs on an EC2 instance or not?

August 25th, 2016

So you have a docker container running on a number of hosts: sometimes on a developer laptop when it is being tested, sometimes on an amazon instance when it is running in a live environment.

Now, for some reason, you want that container to know whether or not it is running on an amazon EC2 instance (live environment).

Note that the mere fact that you need to know it probably means that you have failed separating configuration from container image, and you should address that. But hell, this is real life and software is never perfect.

From within the container, you could do a reverse DNS lookup on the host’s external IP address and see if it looks like an amazon fqdn. Huh… Heavy. And prone to error since a container running on an EC2 instance inside a VPC won’t have an fqdn.

There is a much simpler solution: containers on EC2 instances inherit from the instance’s metadata! So for example:

curl http://169.254.169.254/latest/meta-data/

will return stuff if you are inside an EC2 host, or nothing if you aren’t. In python, this turns into:

def is_ec2_instance():
    """See if 169.254.169.254:80 is open to server instance meta-data,
    to see if host is on an ec2 instance"""


    # Note: this code assumes that docker containers running on ec2 instances
    # inherit the instance's metadata, which they do as of 2016-08-25

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.settimeout(0.2)
    try:
        s.connect(("169.254.169.254", 80))
        return True
    except socket.timeout:
        return False

I am on youtube!

September 5th, 2014

Well, at least some of my body parts are :-)
A few of my limbs and the back of my head indeed appear a couple of times in this video:

World fame, here I come!

The end of growth

September 3rd, 2014

I am a regular reader of the Post Carbon Institute’s various publications (among a long list of blogs, books and resources analysing the defects (ecological, financial, economical, social…) of the society we live in…).

I just must link to that video they made: it sums up the whole situation so neatly! It tells in 6 minutes what all this reading has gotten me convinced of.

Global warming

February 8th, 2014

A video that says it all (source: nasa)

Outcome for a debt-fuelled economy facing the end of growth

February 5th, 2014

Finally!
I have spent at least a year searching for the answer to that question: What happens to an economy that is based on debt accumulation when growth stops?

Isn’t it a simple question?

I mean, once you have read a bunch of resources about the end of economic growth (like this or that) and been convinced by their rather compelling arguments, and once you have recognized that most of the world’s economy is fueled by ever increasing-debt, this question comes very naturally to mind.

Indeed, loans are made on the premise that future growth will make it one day possible to repay both loan and interests, and hopefully do so with a margin. Take a glance at the exponential-looking curve of most western governments debts since the seventies and it is clear that the world’s economy is seriously depending on future growth. So what if that growth never comes? What if the assumption of future wealth behind most accumulated debt was but an illusion? What happens with that mountain of debt never to be repaid?

A simple question indeed.

Still, it took a year of steady tribulations across blogs, books and newspapers to come upon a thorough, well-structured answer to that very question. Why? Mostly because the conventional economic debate sees the end of growth as a kind of taboo. People coming in trumpeting about resource scarcity and the end of fossil fuels are still seen by economists as doomsday weirdoes riding vegan unicorns. But I suspect the root cause lies deeper, in the way the human mind tends to fossilize its knowledge: if all you learn at economic schools is how to navigate an economy whose sole purpose is growth, thinking outside that box becomes really hard.

I stumbled across my answer in ‘The end of growth, adapting to our new economic reality‘ by Richard Heinberg.

His discussion is based on research from the economic historian Niall Ferguson who identified 6 ways of resolving a debt crisis:

  • 1. Increasing growth
  • 2. Lowering interest rates
  • 3. Offering bailouts
  • 4. Measures of economic austerity
  • 5. Creating money and injecting it into the economy
  • 6. Arranging for debt to default in a coherent way

Option 1 becomes obsolete in an economy devoid of growth. Interest rates are already so low that option 2 can be discarded. Option 3 works to some extent as when the US government bailed-out Fannie Mae and Freddie Mac in 2008. But it does not scale up. Economic austerity is what the IMF and EU are applying in Greece and Spain, and it’s painful. Option 5 is what the US have been doing since 2008 with Quantitative Easing. Option 6 is more funky since it requires some form of debt-jubilee. That’s what Iceland did when it gave the financial finger to its creditors upon going bankrupt.

In the context of a worldwide systemic failure of economies due to the cessation of growth, the first 4 options are not realistic. That leaves money creation (and hence inflation which leads to relative reduction of debts) and debt-jubilee. Both solutions are highly controversial but, hey, at least they make it possible to avoid a total economic meltdown.

Now I am happy, I got my answer and the future looks less mysterious :-)

Time to think

January 29th, 2014

I am escalating my parental leave, passing from a 32 hours work-week (for the past 4 years) to a 15 hours one (for the whole coming year), and spending the rest of the time at home, caring for my second child.

Being on a parental leave has an interesting side effect, something I remember experiencing already the first time: it leaves me with time to think. Quality thinking time. Unstressed, unhurried time to catch ideas, massage them thoroughly, fry them on every side and let them sprout new ideas all over. And out of that time grows a thirst for knowledge and understanding, as well as a deep pleasure at seeking them.

And I don’t mean time to think in the way I do while working as a software developer. It’s a completely different thing to focus one’s brain on a work assignment, while being interrupted all the time and constrained by a team’s agenda, versus letting my brain freely explore topics of its own interest. Time spent at work passes in a sort of focused thinking trance woven into a rather intensive social dance, that leaves little space to relaxed, self-motivated exploration. And here I must emphasize the ‘relaxed’ aspect of it. No good thinking can ever be hurried up, nor arise under stress.

Caring for my second born gives me plenty of that kind of relaxed time, especially while I am walking with her strapped on my belly or sleeping in her stroller. Time during which I can ponder whatever I have been reading and let new questions arise.

One frustrating point, though, is that all those questions that pop up during the day have to wait for answers until all children have gone to bed and fallen sound asleep. Trying to google for articles while in charge of children just does not work. It only leads to pointless frustration. So lately I have developed a pattern of reading in the evening and pondering during the day.

And what a liberation to suddenly have time to think! I am seeing the world again! New horizons are opening! During my first parental leave, I submerged myself into buddhism, studied corporate finance and vegetarianism. I daydreamed stories that I fancied I would write down someday. This second time, same thing: I am swallowing quantities of books and articles about peak oil, climate changes, renewable technologies, models for collapsing societies, finance and debt-driven economics. And when bored, I swap to yogic philosophy.

It’s a very exciting time, the like of which I haven’t experienced since my student years.

And it strikes me now that many of the persons whose blogs and articles I am reading, people who distinguish themselves by doing a lot of research and coming up with well-grounded theories of their own, have just the kind of lifestyle that leaves space to quality thinking time. No silver bullet here. If you want to think outside the beaten path, you have to arrange space for your thinking.

Expectation testing bash scripts

November 24th, 2013

This post is about a few practical techniques for writing regression tests for a bash script.

More precisely, this is about writing expectation tests in Python for a script written in bash. Most bash scripts contain little logic and a long list of calls to other system commands, so expectation testing fits them rather well.

But first a word of warning: if you find yourself in a position of having to write expectation tests for a bash script, you are in a bad place to begin with. Alarms should scream in your head and you should seriously consider rewriting your script into a more appropriate language. Yet, if a rewrite is not an option at the moment, writing tests for a bash script remains a lesser evil than having none at all.

For the sake of mental sanity, I will however implement the tests in a more appropriate language. Python in the present case. That will also ease the path towards a future rewrite of the bash script to Python.

So let’s consider an average bash script such as the one below. And let’s assume that we want some tests that validate that its behavior remains unchanged.

#!/bin/bash

ARGS=$@
PKGS=
DOPUPPET='true'

usage() {
    cat << EOF
$0 [--no-puppet] --packages foo,bar
Install or update some packages after optionaly running the puppet agent.
EOF

}

parse_args() {
    while [ "$1" != "" ]; do
        case $1 in
            "--packages")     PKGS=$2; shift;;
            "--no-puppet")    DOPUPPET='false';;
            "--debug")        set -x; DEBUG='true';;
            "-h" | "--help")  usage; exit 0;;
            *)                echo "Unknown argument '$1' (-h for help)"; exit 0;;
        esac
        shift
    done
}

parse_args $ARGS

if [[ "$DOPUPPET" == 'true' ]]; then
    puppet agent -vt
fi

aptitude update
aptitude install $PKGS

Had this script been writen in Python, I would have used a function to proxy all the system calls. Then in the tests, I would have mocked this function and asserted that it got called in the right sequence, and with the right parameters.

Well, we can do pretty much the same in bash :-) The key here is to have a proxy function, run() in this example, that does all the external calls, and whose behavior we can control. Here is the code:

#!/bin/bash

ARGS=$@
TEST='false'
PKGS=
DOPUPPET='true'

usage() {
    cat << EOF
$0 [--no-puppet] --packages foo,bar
Install or update some packages after optionaly running the puppet agent.
EOF

}

parse_args() {
    while [ "$1" != "" ]; do
        case $1 in
            "--packages")     PKGS=$2; shift;;
            "--no-puppet")    DOPUPPET='false';;
            "--test")         TEST='true';;
            "--debug")        set -x; DEBUG='true';;
            "-h" | "--help")  usage; exit 0;;
            *)                echo "Unknown argument '$1' (-h for help)"; exit 0;;
        esac
        shift
    done
}

run() {
    cmd="$@"
    if [[ "$TEST" == "false" ]]; then
        eval $cmd
    else
        echo "RUN: $cmd"
    fi
}


parse_args $ARGS

if [[ "$DOPUPPET" == 'true' ]]; then
    run puppet agent -vt
fi

run aptitude update
run aptitude install $PKGS

If the script runs with the –test flag, run() only prints the command it would execute to stdout. We can then, from the Python tests, catch the output of the script and validate that it contains (or not) the expected commands. A simple case of expectation testing.

And here are the tests, runnable with nosetest:

from subprocess import Popen, PIPE

# You probably want something like:
# os.path.join(os.path.dirname(os.path.realpath(__file__)), <relativ path to script>))
PATH_SCRIPT='deploy.sh'

def run(st):
    cmd = ['/bin/bash', '-c', st]
    p = Popen(cmd, stdout=PIPE, stderr=PIPE)
    (out, err) = p.communicate()
    print "Executed [%s]. Got out: [%s] err: [%s]" % (cmd, out, err)
    return out.strip(), err.strip()

def test_help():
    out, err = run("/bin/bash %s --help" % PATH_SCRIPT)
    assert "Install or update" in out

def test_default():
    out, er = run("/bin/bash %s --test" % PATH_SCRIPT)
    assert "RUN: puppet agent -vt" in out
    assert "RUN: aptitude update" in out
    assert "RUN: aptitude install" in out

def test_no_puppet():
    out, er = run("/bin/bash %s --test --no-puppet" % PATH_SCRIPT)
    assert "RUN: puppet agent -vt" not in out
    assert "RUN: aptitude update" in out
    assert "RUN: aptitude install" in out

Voila!
Horrible, ugly, hard to maintain, counter-intuitive, not future-proof… but it works.

(omg, i can’t believe i am actually publishing this…)

Yoga at work

October 15th, 2013

I do yoga at work.

And no, I don’t mean that I am going to a yoga class during working hours, or over lunch time. I mean I do yoga in the office, in plain sight of (some of) my colleagues.

Did your mind just go blank?

Mine did too, when I first pondered the possibility. Rolling my mat in some discreet corner of the office. Half hiding, but not quite. Displaying myself in sweaty training clothes. Making a show of myself in weird and rather intimate positions. All this was a kind of mental no-go.

But on the other hand, I am addicted to my yoga practice. I need it to live more fully. And since my parental duties keep me from popping up at the ashram every morning at dawn, I have to find other times and places to practice. I am at work five days a week, so that’s the next best match.

Once I am on the mat, I quickly find back some of my focus. But practicing at work has a tensed quality that can be disturbing.

I figure that I am practicing in an office full of people who know close to nothing about traditional Ashtanga and have all sorts of ideas and prejudices about yoga in general. That always makes me feel a bit off balance. Part of me is bothered by what I imagine people are thinking about me. Though when I put my worries through a reality-check and ask people what they think of me training at work, they are usually quite enthusiastic. Well, that also tells something about the open-mindedness of folk working at Spotify.

Yet, the other day, a colleague of mine saw me going around with my mat and started chatting about yoga. He does a lot of Bikram yoga himself. His first question was “how good are you at yoga?”, his second was “what asanas can you do?”, and his third “so, can you teach yoga?”. I was a bit over-sensitive from practicing in a public place and I immediately put myself in a defensive mental stance. My mental dialog went: How good I am? Well, I have more experience than the average gym yoga teacher but I am still decades away from the teachers I train with, and yes I can do some non-beginner asanas but does it really matter? and so on. Each of his questions felt like a probing to my ego.

 

That situation was interesting: here was a colleague showing friendly interest in my practice, and I got all defensive. Why?

I am normally delighted to talk about yoga, though I am usually careful to assess how genuinely interested my interlocutor is before flooding him or her with over-enthusiastic statements, less they take me for some weird sect member. But  I felt scrutinized and judged. In other words, my ego took the stage and I got caught in its game.

And that’s exactly where practicing at work gets challenging: it brings forth the ego. With all its habit patterns of defending its self-image and asserting itself in a social context. For me, a good practice will leave me ego-less, and it is a state I seek. But practicing at work sometimes triggers just the opposite effect, leaving me off-balance and in the grip of a grasping ego-centric mind.

 

 

 

 

Burn frustration, burn!

June 17th, 2013

For me, practicing yoga has at least one major effect apart from keeping my body fit: yoga nurtures my mind.

Yoga practice is like a fire that burns through my mind, and leaves ashes where frustration, anguish and compulsions grew before. Zen master Thich Nhat Hanh often describes the mind as a garden full of seeds. In this metafor, seeds are habit-patterns of thoughts and behaviors, that grow and evolve within you. By mindfully applying love and care to those seeds, you can cultivate the good ones and refrain bad weeds from invading your inner landscape. Since I am not a buddha yet, my mind gets regularly invaded by those bad weeds. I grow irritated, start clinging to my cravings, forget to take distance from whatever story my mind is weaving, and slowly but surely loose my inner balance and life sucks again.

This is where yoga practice comes in. I get onto my mat, me and this little crazy jungle in my head. I do my practice. And when I am done, the jungle is gone, burnt to the ground by the tapas, the energy of practice. I find myself centered again, aware of my mind and the surrounding reality, but not caught by them.

I honestly don’t know how this works, only that it does work, and it’s reliable. Any time I feel anxious or my mind starts cramping, I know I can run to my mat, and sweat it away.

Coming soon!

May 18th, 2013

I am changing my blog engine. This will be down for some time…