Charter:

Promote the research, practice, and adoption of benchmarks for computer-related system dependability. In particular, four areas are considered to be "in scope":

• Exchanging ideas about dependability benchmarking among researchers and practitioners (including participants from universities, industry, and government agencies).
• Documenting the state of the art for dependability measurement and benchmarking
  • Including similarities to and differences from benchmarking for other attributes such as performance
  • Publicizing (facilitating access to information about) available tools and techniques
  • Producing durable publications recording research in this area
  • Organizing special meetings, workshops, conference "tracks"/sessions, or other avenues for sharing results and experiences.
• Create lists of issues that must be resolved to advance dependability benchmarking to a mature science
  • What are the goals of a dependability benchmark?
  • What are appropriate units of measure? How realistic is it to have a "single" metric result?
  • What should the scope of such a benchmark be (Should it include hardware, application software, operating systems, and infrastructure? How much should it extend beyond "fault tolerance" and into security, usability, safety, etc.?)
  • What constraints must be overcome and what constraints must we be resigned to accept? (this includes technical, organizational, economic, political, and other constraints)
  • How can such a benchmark be made to measure artifacts (computer-based systems) themselves, rather than be inferred from development process metrics or hindsight-based field experience for any particular system.
  • How can such a benchmark account for the effects of system administration (and users in general) compared to the "baseline" dependability of the underlying system. These issues include the quality of the human interface (how likely it is that people will make mistakes) as well as the skill of the humans (whether the people "too many" mistakes given a reasonable user interface).
  • What are needs and perspectives the various stakeholders (system vendors, system architects, system administrators, end users, general public?)
• Eventually (not during the first meeting!) propose a mechanism and agenda for a group to propose dependability benchmarks
  • Should this go through a standards committee? Or be an industry-sponsored entity? Or simply a published set of software maintainted by volunteers?
• As appropriate, create collaborative publications. A potential goal is to create a White Paper on dependability benchmarking as the result of this SIG's efforts.

Authority:

This Special Interest Group was created under IFIP's Working Group 10.4 (pursuant to IFIP ByLaw 4.3.9). Nothing in this web site should be considered to supercede any policy, law, or bylaw of IFIP, nor should it be considered an official statement nor endorsement by IFIP, WG 10.4, or anyone but the particular authors as individuals.

Home page

koopman@cmu.edu 9/20/99