BENCHMARKS (longish)

From: André Dolenc (Helsinki University of Technology)
Date: Tuesday, July 26, 1994

From: André Dolenc (Helsinki University of Technology)
To: RP-ML
Date: Tuesday, July 26, 1994
Subject: BENCHMARKS (longish)
     I would like to raise a topic for discussion in this mailing list:
                        BENCHMARKS
     My interest in this topic is to learn what is it that people want to measure in a benchmark in 
RPT, and is it being done properly. I will share with you what I've learned about benchmarks in a 
database seminar by Jim Gray, and the material below was taken from the material distributed 
and my personal notes. At the end, I have a few questions.
1. The importance of benchmarks
     Vendors design their systems to obtain good benchmark results. It follows that benchmarks 
should reflect reality, so that there is some guarantee that vendors will improve their systems for 
the benefit of the users.
2. Benchmarking is hard.
3. What makes a good benchmark standard?
   a. RELEVANT:   characterizes the typical usage.
   b. SIMPLE:     it is understandable
   c. MEASURES:   gives a clear metric
   d. PORTABLE:   can move to new technology
   e. SCALABLE:   allows more than one device
   f. RECOGNIZED: vendors ANF users embrace it
4. How benchmarks are used: good stories
   a. accelerates progress: designers and users can compare approaches.
   b. gives performance agenda: measures release-to-release progress, sets goals, provides 
something managers can understand.
   c. Benchmark definition forms a community/vocabulary: cross fertilization of people/ideas, 
sharing of knowledge (which some people resist).
5. How benchmarks are used: bad stories
   a. Benchmarketing: make up a benchmark that makes you look good, and publicize it. Or, 
worse yet, publicize the results, but hide the benchmark (procedure).
   b. Benchmark Wars:
      - Big benchmark
      - One side wins,
      - other side re-runs it with "A-team",
      - other side re-runs with 1-star gurus,
      - other size re-re-runs with developers,
      - other size re-re-runs with next release,
      - other size re-re-re-runs with special release,
      - ad infinitum.
     It is my opinion that all this applies to RPT with some minor modifications.
     The first questions I have are:
1. Is there a clear understanding on how to benchmark RP processes?
2. Is there a consensus on how to benchmark RP processes?
3. If not, isn't time we knew how to do this?
4. The test objects being used in benchmarks should be publicaly available. Are they easy to 
obtain? How does one obtain them?
Regards,
Andre'


Previous message | Next message
Back to 1994 index