The DBench project will define a conceptual framework and an experimental environment for benchmarking the dependability of COTS and COTS-based systems. It will provide system developers and end-users with means for

1. characterising and evaluating the dependability of a component or a system,
2. identifying malfunctioning or less weak parts, requiring more attention,
3. tuning a particular component to enhance its dependability, and
4. comparing the dependability of alternative or competing solutions.

The two final measurable objectives that will be produced at the end of the project are a report presenting the concepts, specifications and guidelines for dependability benchmarking and a set of dependability benchmark prototype tools. The prototypes will be made widely available (e.g., through the web whenever possible) to promote their adoption by an audience as wide as possible.

The work will be performed in four steps:

1. definition of the conceptual framework for system benchmarking,
2. identification and evaluation of the enabling technologies,
3. specific benchmark definition and application to pilot experiments in order to develop, experiment and validate the benchmark prototypes and
4. consolidation of the conceptual framework with the experimental results.

The conceptual framework will settle the foundations of dependability benchmarks. It addresses several important aspects such as the need for an overall and global system viewpoint to correctly select the measures to be evaluated and interpret the results. To put into practice the conceptual framework, enabling technologies will be investigated and adapted in some respects. They address: measurements to be performed on the target system, fault representativeness and workload and faultload selection. The selected enabling technologies will be evaluated. The experiments will be performed to achieve a comprehensive work: different dependability benchmark prototypes will be defined and developed for two major application-areas (embedded and transactional applications). The benchmark prototypes will actually be implemented in two different families of COTS operating systems (Windows and Linux), allowing a cross evaluation of the concepts and the enabling technologies in a true dependability benchmark context. The final goal of the experiments is the validation of the dependability benchmark prototypes, in the sense of assuring that the benchmarks results represent a practical and meaningful characterisation of the dependability properties of the target systems, both from the end-user and the system developer’s point of view. Finally, the consolidation of the whole set of results will allow to push further the enabling technologies and to finalise the benchmark concepts and prototypes.

The work is composed of four workpackages:

1. conceptual framework,
2. enabling technologies identification and evaluation,
3. benchmark definition, experimentation and validation and
4. consolidation.

The titles reflect the activities to be completed and reported in the associated deliverables. The two final objectives to be produced at the end of the project are a report presenting the concepts and guidelines for dependability benchmarking and a set of dependability benchmark prototypes.