To view the directory of supplemental files, click here.

The subdirectory data contains the responses from individual experimental trails (algorithm runs), with filenames of the form ssss_aaaa_pppp.csv, where:

• ssss is the SUT (bestMove, nischneu, cArcsin, fct3);
• aaaa is the algorithm (DM- or DM+); and
• pppp is the type of experiment (untuned - parameters before tuning, tuned - parameters at predicted optimal parameters, or RSM iteration/phase in the form ixpy where x is the iteration and y the phase).

In these files, the first column is the response (logarithm of the adjusted number of SUT executions as described in the paper). For RSM phase 2 experiments only, the second column is the distance in steps along the steepest descent path. The remaining 9 (for DM-) or 10 (for DM+) columns are the coded parameters at which the experiment was to run (converted back from the actual parameter values to account accurately for rounding).

The subdirectory suts contains source of the SUTS, apart from that for FCT3 which we unable to share. The macros beginning _AST, and the additional function arguments elementCountArray, outputs and elementCount, implement the instrumentation.

To view the directory of supplemental files, click here.

The directory contains:

• Source code for each of the three SUTs.

The function itself is in the header file and includes macros that enable instrumentation to detect branch coverage. Building the corresponding *.c file (e.g. simpleFunc.c) produces an executable without instrumentation, while building the *_inst.c file (e.g. simpleFunc_inst.c) produces an executable that outputs instrumentation data to stderr. For both executables, function arguments are passed as command line parameters.

• simpleFunc: simpleFunc.h simpleFunc.c simpleFunc_inst.c
• bestMove: bestMove5.h bestMove5.c bestMove5_inst.c
• nsichneu: nsichneu.h nsichneu.c nsichneu_inst.c
• Data for each of the four experiments.
• Experiment A

expt_A_*_response.txt contains the response (probability lower bound) for each trial. Each row corresponds to a separate trial. The columns are explained in expt_A_response_desc.txt; in particular, note that '1' in the second column indicates a 'succesful' trial, and '0' an 'unsuccessful' one.

expt_A_*_timing.txt contains the timings (as CPU user time) for each trial.

• Experiment B

expt_B_*_response.txt contains the response (mutation scores) for each trial. The columns are explained in expt_B_response_desc.txt. Each probability distribution is uniquely identified by a distribution id. Statistical testing distributions have non-zero ids; the uniform distribution used for random testing has an id of zero. (The file expt_B_bestMove_response_150.txt is an additional run of the experiment for only the uniform random distribution for test sizes up to 150. The data are used to determine the test size at which a mutation score of 0.55 is achievable using random testing, in order to perform a more realistic cost-benefit analysis than would be possible with the original experimental data that consider test sizes up to 50.)

expt_B_*_coverage.txt determines the number of samples taken from each distribution until the coverage criterion (all branches) is achieved. The columns are explained in expt_B_coverage_desc.txt. Each row corresponds to the equivalent row in expt_B_*_response.txt in that the same samples, in the same order are taken from the distribution. Note that the data for uniform distributions (zero id) are not used in the analysis described in the paper (and is incomplete for some SUTs).

• Experiment C

expt_C_*_response.txt contains the response (mutation scores) for each trial. The columns are explained in expt_C_response_desc.txt. Each probability distribution is uniquely identified by a trajectory iteration id.

• Experiment D

expt_D_*_response.txt contains the response (mutation scores) for each trial. The columns are explained in expt_D_response_desc.txt. Each probability distribution is uniquely identified by a combination of diversity constraint and distribution id (i.e. distribution id is unique only within the same setting of diversity constraint). A value of '1' in the second column indicates a diversity constraint was used during the search, and a value of '0' that it was not.