Wave Issue 9711 5/9/97 Article 2-01

***Opinion: The Difficulties of 3D Benchmarking
by Roy Latham

Graphics pros would like board and chip vendors to succinctly state the
vertex capacity, the rendering capacity for polygons having different
attributes, the significant bottlenecks of the architecture, and which
software has been tuned. It would then be fairly easy to figure out which
applications are well-suited to the particular hardware. There are two
problems with this approach: (1) the detailed information would not be
helpful to the average consumer, and (2) only the top performing vendors
benefit from making performance attributes clear, so the information is
rarely made available. In fact, since there probably is no overall "best"
hardware, with all the variations in price, performance, bottlenecks, and
tuning, many, many products will be "best" at some application. So
instead of making the overall performance attributes clear, each vendor
finds it advantageous to either cite the benchmarks at which they excel
or just publish unqualified mostly-meaningless numbers.

As a practical matter, while putting together the graphics accelerator
survey for our professional newsletter "Real Time Graphics" it was
difficult to even get information as to whether polygon performance was
for individual triangles, strips, or meshes. This was not usually
deliberate obfuscation. Often, the marketing department did not
understand what we are taking about, and engineering assumed that
whatever it was they did was obvious and the standard way of doing it for
the industry. Engineers may assume that the bottlenecks in their design
are the "normal" and "necessary" ones, and hence not worth commenting
upon. In fact, they vary considerably from product line.

The bottom line: use benchmark data that is as close as possible to a
particular application, understand what the benchmark means, and if it is
really important, do you own tests or have an evaluation done by a pro.

Roy Latham is the editor and publisher of the Real Time Graphics
newsletter. It is a newsletter for professionals working in the fields of
simulation and virtual reality. It covers all aspects of the technologies
of virtual environments, with special emphasis on image generators and
displays, and associated software for real time operation.

www.cgsd.com/rtg.html