.Dd August 11, 2012
.Dt NUMBENCH 1
.Os
.Sh NAME
.Nm numbench
.Nd Benchmarking tool for Gentoo
.Sh SYNOPSIS
.Nm numbench
.Fl conffile
.Op Ar options ...
.Nm
.Op Ar -h , --help

.Sh DESCRIPTION
.Pp
numbench is a tool for compiling and benchmarking numerical
libraries. It is useful to find out the implementations of common
libraries that run faster on the machine and are more accurate.
The script makes use of the portage features in order to download,
compile, install and test the packages.

.Ss Configuration file
.Ix Subsection "Configuration file"

.Pp
The configuration file is a XML document that contains the information
about the tests that have to be performed. In particular, it is stated
which module must be called, its arguments (usually the operations the
will be performed) and which test cases are to be run. By "test cases",
we mean the packages that contain the software that must be benchmarked
and the configuration the package must be compiled and run with.

Like every XML document, the configuration file must have a root element
which is called "numbench". Within the root element, two elements must
be present: "operations", which defines the module name and its arguments,
and "testcases", which defines the test cases. The element "testcases"
contains many "case" elements which describe one particular test case.

The "operations" tag contains an attribute called "module" that specifies
the module (e.g. "blas", "lapack"). The content of this tag lists the
arguments that will be passed to that module during its initialization.

The following is an example of a correctly written "operations" tag for
the module "cblas" where some operations for vectors and matrices
computations are listed:

  <operations module="cblas">
    axpy matrix_vector matrix_matrix
  </operations>


Each test case has many configuration options which are specified in the
elements within the tag "case". The tag "case" must contain an attribute
"id" which specifies an unique identification string for the test case.
Only alphanumeric characters are allowed.

The only required tag is the package name that goes in the content of the
subelement "pkg". Optionally, a description of the test case can be written
within the subelement "descr"; currently, this describtion, even if present,
is ignored; in future, it could be possible to display this informative
string in the report.

As every package can install many implementations of the software that is
being benchmarked, the user has faculty to tell numbench to skip some of
them. This can be achieved using the subelement "skip". By default, the
content of this subelement is interpreted as glob pattern, allowing one
to use wildcards "*" and "?". If the element has the attribute "type" equal
to "regexp", the content will be interpreted as regular expression. One can
add as many "skip" tags as desired. If the name of an implementation matches
a glob pattern or a regular expression given in a "skip" tag, that
implementation will be skipped, avoiding waste of computational
resources and making the plots more clear.

For each test case the user can specify up to four different environments:

\fBdependenv\fR: this environment is used by portage when emerging
dependencies of the package

\fBemergeenv\fR: this environment is used by portage when emerging the
package itself

\fBcompileenv\fR: this environment is used by numbench while compiling the
benchmarking program (not every module makes use of this environment)

\fBrunenv\fR: this environment is used by numbench when running the
benchmarking program

In order to specify the variables defined in one of these environments,
place a subelement with the corresponding name inside the test case tag;
within this subelement, place as many "var" tag as needed: each "var" tag
must have an attribute called "name" (with the name of the variable) and a
content, which will be the value of the variable. Example:

  <emergeenv>
    <var name="CC">gcc-4.6.2</var>
    <var name="CFLAGS">-O2 -funroll-loops</var>
  </emergeenv>
  
There are other ways to define environments. If the environment contains
many variables, it could be cleaner to put the definitions in a separate ASCII
file as key=value pairs, one on each line, and linking to it. For this, use
the attribute "file", whose value is the path to the file containing the
variables. For example, we could write the file "myenv.txt":

  CC=gcc-4.6.2
  CFLAGS=-O2 -funroll-loops

And then write the following in the configuration file:

  <emergeenv file="myenv.txt"></emergeenv>

The only drawback of this method is the fact that the file is static.
Instead, it is possible to write a more general script which is executed
and whose final environment is taken. Say, for example, that we have a script
called "myscript.sh", which exports the variables CC and CFLAGS according to
the most recent compiler that is available on the system; then, we can use the
attribute "script":

  <emergeenv script="myscript.sh"></emergeenv>.
  
These three ways of defining variables can be mixed:

  <compileenv file="mystaticfile.txt" script="mydynamicscript.sh">
    <var name="CFLAGS">-O3 -fopenmp</var>
  </compileenv>

In this case, the specifications present in the static file override the
variables defined through the script, while the variables defined in the
XML document override both.

Still, one could want to use her/his current environment for one or more of
the steps. In this case, the attribute "append" with value "1" must be used:

  <runenv append="1"></runenv>.
  
This is the weakest way for setting variables: they are overriden by other
variables specified in scripts, files or subtags.