Test anything (with Perl)

Erwan Lemonnier

Premiepensionmyndigheten

$Revision: 1.3 $

Ever wrote some code?

• A program?
• A shell script?
• A piece of PHP/Javascript/ASP?
• Any form of code?

So...

How do you know it works?

Wrong answer 1

"I never write bugs"

Wrong answer 2

"I have read the code over and over X times"

Wrong answer 3

"It compiles without warnings..."

Wrong answer 4

"It runs without crashing"

Wrong answer 5

"I have a team of testers testing it for me"

Getting closer

"I wrote unitests for every method/function/combination of input argument. I have 100% test coverage and 1 billion beta testers running on every existing operating system and sending me automated feedback. 10000 programmers are reviewing every single release of my code. All tests are automated and run periodically."

The shocking truth

You cannot prove code to be bugfree

Testing software

• Testing software is an art (seldom taught)
• Testing code is the responsability of the developer

Testing code is hard (1/4)

Code follows a specification

• Is this specification complete?
• Have you thought of all side cases? Dependencies?

Testing code is hard (2/4)

Code evolves

• You will want to refactor your code...
• ...but will you remember the specification then?
• ...will you understand the code then?

Testing code is hard (3/4)

Code has changing dependencies

• Operating system
• Third party products (database, web browser...)
• Standards evolve

Testing code is hard (4/4)

Developers evolve

• They forget
• They get better at writing software
• They change job

Mastering change

Test must be automated and reproducible.

Software must be fully tested each time:

• the code is changed
• dependencies change

How?

• write programs to test code
• run all those programs at every code change
• and run them periodically

Writing tests...

...is easier than you think.

let's illustrate with Perl...

Perl!!

TAP

Test Anything Protocol

Can test:

• Perl code
• Any command line program
• Any network server

Example

let's test /bin/mv

Example: testing /bin/mv

    $ perl test_mv.t
    1..3
    ok 1 - error if no arguments
    ok 2 - old file removed
    ok 3 - new file created

	

test_mv.t (1/2)

#!/usr/local/bin/perl
use strict;
use warnings;
use Test::More tests => 3;
	

test_mv.t (2/2)

ok(`mv 2>&1` =~ /missing file argument/,
   "error if no arguments");
die if (-e 'foo');
`touch foo`;
`mv foo bar`;
ok(!-e 'foo', "old file removed");
ok(-e 'bar', "new file created");
unlink 'bar';
	

Better tests... (1/2)

• Test all possible arguments (unknown arguments, weird formats, string overflow...)
• Simulate faults (missing files, missing target dir, file system timeout, open() failure...)

Better tests... (2/2)

• Maximize code coverage of tested code!

Code coverage

• Language specific
• Tells you what to test
• Example: test_mv.t

Regular test runs (1/2)

• A crontab!
• as simple as:

ls *.t | xargs -n 1 perl > ./results

Regular test runs (2/2)

• or use /usr/bin/prove:

> prove *.t
test_cp....ok
test_mv....ok
All tests successful.
Files=2, Tests=6,  0 wallclock secs ( 0.10 cusr +  0.16 csys =  0.26 CPU)

prove.sh

# run perl tests in DIRTESTS, save results in DIRLOG
# and send email alarm if tests fail
LOG=$DIRLOG/my_tests_`date +%y%m%d.log` # osx only
cd $DIRTESTS
prove *.t | tee $LOG

if [ `cat $LOG | grep PASS | wc -l` == '0' ]; then
    (   echo "FROM: prove"
	echo "TO: you@somewhere.com"
	echo "SUBJECT: test failed"
	cat $LOG ) | sendmail -t
fi

Regular test runs

• store results in db
• add a web interface
• stats and reports
• alarms...

Continuous integration

• Run tests when change commited in version control system (ex: CVS hook)
• Alarm if some unit test fails
• Fast feedback!
• Ex: Smolder for TAP
• Ex: CruiseControl for Java and .NET

Thinking Agile

• Writing test code affects sofware development
• Develop in small increment
• Write tests at each step
• Design code for testing

More Examples

testing a web server

test a web server (1/3)

• Perl has modules for everything: CPAN
• WWW::Selenium
• WWW::Mechanize

test a web server (2/3)

use Test::More tests => 26;
use WWW::Mechanize;

my $m = WWW::Mechanize->new();

$m->get( "www.foo.com/login.html");
ok($m->success, "got login page");
	

test a web server (3/3)

$m->submit_form(
       form_name => 'login',
       fields    => {
           username => 'bob',
           password => 'secret',
           }
       );
       
is($mech->title, "Welcome bob", "logged in");
...
	

More Examples

• Testing a database: DBI
• A program with command line UI: Test::Expect
• A TCP server: Test::Expect and netcat
• A shared C library: Inline C
• A Java class: Inline Java

Test a shared C lib (test_strcat.t)

use strict;
use warnings;
use Test::More tests => 1;

use Inline C =>
    Config =>
        LIBS => '-lm',
        ENABLE => 'AUTOWRAP';
	
Inline->import( C => <<END_HEADERS );
char* strcat(char*,const char*);
END_HEADERS

ok(strcat("foo","bar") eq "foobar");
	

Test a shared C lib

> perl test_strcat.t
1..1
ok 1
	

Advertisment

For more: Perl Testing: A Developer's Notebook (O'Reilly)

The bleeding edge

Some things are almost impossible to test...

• Multithreaded applications
• Self modifying code
• Exceptions in external dependencies (OS, shared lib...)
• Hardware failures

Questions?

Thank you!