Charter:
Promote the research, practice, and adoption of benchmarks for
computer-related system dependability. In particular, four areas are considered
to be "in scope":
- Exchanging ideas about dependability benchmarking among researchers and
practitioners (including participants from universities, industry, and
government agencies).
- Documenting the state of the art for dependability measurement and
benchmarking
- Including similarities to and differences from benchmarking for other
attributes such as performance
- Publicizing (facilitating access to information about) available tools and
techniques
- Producing durable publications recording research in this area
- Organizing special meetings, workshops, conference
"tracks"/sessions, or other avenues for sharing results and
experiences.
- Create lists of issues that must be resolved to advance dependability
benchmarking to a mature science
- What are the goals of a dependability benchmark?
- What are appropriate units of measure? How realistic is it to have a
"single" metric result?
- What should the scope of such a benchmark be (Should it include hardware,
application software, operating systems, and infrastructure? How much should it
extend beyond "fault tolerance" and into security, usability, safety,
etc.?)
- What constraints must be overcome and what constraints must we be resigned
to accept? (this includes technical, organizational, economic, political, and
other constraints)
- How can such a benchmark be made to measure artifacts (computer-based
systems) themselves, rather than be inferred from development process metrics
or hindsight-based field experience for any particular system.
- How can such a benchmark account for the effects of system administration
(and users in general) compared to the "baseline" dependability of
the underlying system. These issues include the quality of the human interface
(how likely it is that people will make mistakes) as well as the skill of the
humans (whether the people "too many" mistakes given a reasonable
user interface).
- What are needs and perspectives the various stakeholders (system vendors,
system architects, system administrators, end users, general public?)
- Eventually (not during the first meeting!) propose a mechanism and agenda
for a group to propose dependability benchmarks
- Should this go through a standards committee? Or be an industry-sponsored
entity? Or simply a published set of software maintainted by volunteers?
- As appropriate, create collaborative publications. A potential goal is to
create a White Paper on dependability benchmarking as the result of this SIG's
efforts.
Authority:
This Special Interest Group was created under
IFIP's
Working Group
10.4 (pursuant to IFIP ByLaw
4.3.9).
Nothing in this web site should be considered to supercede any policy, law, or
bylaw of IFIP, nor should it be considered an official statement nor
endorsement by IFIP, WG 10.4, or anyone but the particular authors as
individuals.
Home page
koopman@cmu.edu 9/20/99