Abstract

We explain how to take advantage of the Debian build daemon infrastructure to
find bugs before they hit the distribution, look into the archive or use Google
Code Search to uncover recurrent bugs. We then present the most common Debian
QA tools and explain how to create new test rules for common issues. Finally,
we present ways to test the software once compiled, either at build time using
automated testing procedures or after installation using fuzzers and sandbox
testsuites.

Full Description

[full description will be polished and expanded later if possible]

It is possible to find bugs in packaged software a while before users report
bugs, sometimes even as soon as the software is built. The Debian build daemon
infrastructure offers a great opportunity for datamining and recurrent bug
seeking.

At source level, bugs can sometimes be found using the buildd logs. For
instance, the gcc -Wall flag exposes issues such as strict aliasing rule
breaking, dangerous integer/pointer casts or wrong argument casts due to
implicit function declarations. We discuss a few additional warning flags that
are not part of -Wall and expose ways to activate them without breaking build
rules, especially in automake projects. We then present Google Code Search
and its powerful regex search engine and list a few queries that allow to
rapidly find bugs that are caused by a given pattern. Archive source datamining
techniques are also discussed.

Bugs in binary Debian packages are also common. Tools such as lintian
and linda help find common issues that do not require the package to be
installed. We explain how to create new rules for these tools, how to make the
lintian.debian.org service more useful and how to dig into related hidden bugs.
Specifically checking for regressions appears to be a more effective strategy
when looking for bugs than mere addressing of each lintian error or warning.

Runtime bugs are the ones that the user is expected to be confronted to. But
there are ways to automatically test the software in a Debian environment
before the users needs to use it. The packaged software's testsuite may be
inserted into the build process, as well as custom automated tests for common
tasks. The "expect" tool can be used to test commandline tools without user
interaction. We show how to do the same with GUI applications using Xvfb and
additional tools. Eventually, we show how fuzzers can find bugs more quickly
than the average code review, present the Hachoir fuzzing framework and the
tiny zzuf general purpose fuzzer, and give more examples of specific fuzzers
(LDAP fuzzers, web application fuzzers, etc.)