***Review: ZD-BOP's 3D WinBench 97
by David Lohse
OVERVIEW
3D Winbench 97 was released by Ziff-Davis Benchmarking Operations (ZD-
BOP) on April 25, 1997 to coincide with the opening of the Computer Game
Developers Conference. As the successor to MonaLisa, their unreleased
preliminary effort to which it bears little resemblance [see WAVE #704,
2/28/97], 3D WinBench 97 brings 3D benchmarking to a new level.
Previously, 3D benchmarks available to the industry have tackled only one
or a few aspects of 3D accelerator performance, and none of them have
adequately addressed the important issues of visual quality. 3D WinBench
97 leapfrogs all other available 3D benchmarking software currently
available to the industry by providing detailed and comprehensive testing
of many of the issues important to 3D accelerator performance.
The most notable shortcoming of 3D WinBench 97 is that it is focused on
consumer-level accelerators. It is based on Direct3D and is not supported
on the Windows NT platform; without support for OpenGL or WinNT, most
accelerators intended for the professional market are excluded, although
this is beginning to change as Win95 and WinNT converge and as D3D moves
onto an increasingly number of accelerators. However, with OpenGL already
available on the Windows 95 platform and with growing interest in using
it for consumer applications (re: games), it would be very beneficial to
include OpenGL support in the benchmark.
INTERFACE
3D WinBench 97 has been designed to present the same easy-to-use
interface as the other benchmarking software developed by ZD-BOP such as
WinStone. The standard function window presents the same layout as their
other benchmarks, with buttons that let the user run a selected test,
save results, or compare results, as well as the usual "help" and "about"
functions. The help feature follows the standard Windows help format and
is fairly comprehensive.
3D WinBench 97 is split into three different types of testing
functionality:
* 3D Quality
* 3D Triangle Tests
* 3D WinMark
Each of these categories is comprised of many individual tests. The user
can run either individual tests for specific features, an entire category
of tests (one of the three categories), or all of the tests
consecutively.
3D QUALITY
The addition of this test suite to the benchmark elevates it to the next
level above other existing benchmarking software. The 12 tests included
in this suite are used to test the effects of specific rendering options
on visual quality, including:
* Alpha transparency
* Color key transparency
* Dithering
* Fog table
* Fog vertex
* Fog vertex and alpha
* Fog vertex and color key
* Linear texturing
* Mipmap linear texturing
* Perspective correction
* Specular highlights
Each test runs a short 3D scene utilizing the specific feature being
tested. When the short animation is over, it presents the user with a
snapshot of the scene for reference and compares it to snapshots of the
same scene as it should look ideally rendered and as it would look if the
feature is poorly implemented. The user is prompted to choose whether or
not the scene has been rendered mostly correctly or not; if the user
chooses "no" than that feature is turned off during subsequent WinMark
tests run.
Shortcomings
Although this test suite provides an excellent mechanism for testing the
visual quality objectively, it could use several improvements. Most
notably, the user must choose a "yes" or "no" answer as to whether the
scene was properly rendered. In most cases, the feature being tested was
obviously "working" but looked better on some boards than on others, and
this could not be directly addressed. Instead, a ratings scale would be
much more useful in comparing different accelerators since side-by-side
implementations of testing are not usually practical.
In addition, the quality tests did not address quality issues outside of
the score of the individual tests. That is, each test asks the user to
determine whether the specific quality being tested is working properly,
but is not concerned with other problems. For example, in several of the
tests we ran, the specific feature being tested was fine, but other
unrelated visual artifacts would significantly reduce the quality of the
animation. These issues could also not be directly addressed within the
benchmark.
Also conspicuously absent from the options was trilinear texture
filtering, which ZD-BOP plans to add to future releases.
3D TRIANGLE TESTS
19 tests make up this suite, which tests the triangle throughput for
various rendering techniques:
* 640x480x16, flat
* 640x480x16, Gouraud
* 640x480x16, Gouraud, linear
* 640x480x16, Gouraud, nearest
* 640x480x16, Gouraud, nearest mipmap linear
* 640x480x16, Z, flat
* 640x480x16, Z, Gouraud
* 640x480x16, Z, Gouraud, dithered, nearest
* 640x480x16, Z, Gouraud,dithered, specular, fog vertex, linear
* 640x480x16, Z, Gouraud,dithered, specular, fog vertex, nearest
* 640x480x16, Z, Gouraud,dithered, specular, fog vertex, nearest
mipmap linear
* 640x480x16, Z, Gouraud, fog vertex, nearest
* 640x480x16, Z, Gouraud, linear
* 640x480x16, Z, Gouraud, nearest
* 640x480x16, Z, Gouraud, nearest mipmap linear
* 640x480x16, Z, Gouraud, specular, nearest
* 640x480x8, flat
* 640x480x8, Z, flat
* Z-buffer clear
The tests may be run either individually or the entire suite may be run.
The results are given in triangles/sec for triangles of size 5 and 50
(pixels) and pixels/sec for triangles of size 50 and 1000.
Shortcomings
Although this suite succinctly tests the most important configurations
and features for consumer 3D applications, it lacks flexibility. It is
limited by the 19 test configurations in the suite, and does not allow
the user to define their own testing parameters. In addition, it would be
useful to be able to run tests over ranges of triangle sizes, although by
running tests on triangles sized 5 and 50 the benchmark does demonstrate
results for the important "small" and "moderate" sized triangles most
commonly used. It is also limited by 640x480 screen resolution; although
this is the most commonly used resolution for consumer applications,
testing for higher resolutions would be beneficial.
Again, trilinear texture filtering was conspicuously absent.
3D WINMARK
This suite tests the accelerator's performance at the scene level. The
results are given in frames/sec, and it also allows the user to observe
the visual quality of the scene - if side-by-side tests are feasible for
a user, this suite would allow them to make accurate visual comparisons.
It consists of five different 3D scenes, on which 10 different tests are
configured. Each test is configured with various "important" rendering
options enabled. In addition a user can define their own scenes based on
one of ten available databases along with the rendering options they
specify. The options available for user configuration are:
* Scene: 10 available
* Frames: starting frame and number of frames to play
* Full screen (on/off)
* Gouraud shading (on/off)
* Specular highlights (on/off)
* Dithering (on/off)
* Fog: Off, On (any mode), On (vertex mode), On (pixel/table mode)
* Screen resolution: 320x200 up to 1600x1200
* Color depth: 8, 16, 24 or 32 bits
* Lighting: None or one of 10 different types
* Texture mapping (on/off)
* Texture filtering: nearest, linear, mipmap nearest, mipmap linear,
linear mipmap nearest, linear mipmap linear
* Transparency: Off, Any mode, Color key, Alpha
* Perspective correction (on/off)
Shortcomings
The most evident shortcoming of this suite is the 3D scenes available.
Unfortunately, they are in the proprietary ".sdl" format, so users cannot
import their own databases from other formats, and are therefore limited
by the 10 databases included with the benchmark. The .sdl file format,
which is only partially documented, is reportedly similar to Microsoft's
.X DirectX file format. It would be much more beneficial if a more
"standard" file format were used.
HARDWARE INTERATION AND USER CONFIGURABILITY
Hardware Detection
Like ZD-BOP's previous benchmarking software, 3D WinBench 97 extensively
tests the user's hardware configuration and capabilities. In our
preliminary tests it correctly identified all of the major hardware
components such as host processor and memory, bus, operating system
build, CD-ROM drive, sound board, and of course the display adapter.
Less significant settings were not always detected, such as the display
adapter's BIOS or RAM, the refresh rate, the host processor's cache or
BIOS, etc. However, the user is allowed to enter this information
manually, and in any case these settings do not affect the test results.
Also like their previous benchmarking software, 3D WinBench 97 examines
the system to determine if it is optimally configured for testing (e.g.,
if other apps are running, etc.) and if it is capable of running them
(e.g. if Direct3D 3a is installed, etc.).
The benchmark also tests the installed D3D accelerator and determines
what features it supports (presumably by polling the card's D3D
capability bits). This is described directly below under "Hardware
Capability Configuration."
User Settings
Important test settings are also configurable by the user, including:
* Selectable DirectDraw adapter (primary display driver in most
cases)
* Hardware acceleration: enabled or disabled
* MMX DLLs: enabled or disabled
* Rendering buffer: front buffer, back buffer or double buffered
Hardware Capability Configuration
In addition, this benchmark offers the unique capability to turn off
features supported by the hardware or to "force" hardware acceleration of
unsupported features. The benchmark determines whether the hardware
supports the following features, and allows the user to either "force on"
or "force off":
* Dither
* Fog Vertex
* Fog Table
* Specular Highlights
* Color Key
* Alpha
* Fog vertex and Color Key
* Fog Vertex and Alpha
* Perspective Correction
* Nearest
* Linear
* Mipmap Nearest
* Mipmap Linear
* Linear Mipmap Nearest
* Linear Mipmap Linear
RESULTS AND REPORTING
Results are displayed in the common ZD-BOP format, with both numeric
results and colored bar charts, which can be displayed in either absolute
(actual numeric score) or normalized (% fraction) mode:
* 3D Quality Tests - Results display a table listing the parameters
tested, indicating whether or not each one was correctly
implemented (as decided by the user)
* 3D Triangle Tests - Results are given in triangles/sec for both
triangle sizes of size 5 and 50 and pixel fill rate for triangles
of size 50 and 1000
* 3D WinMark - Results are given in frames/sec as well as a single
"3D WinMark" score, although it is not clear what the intrinsic
value of this number is or how it is derived
The results may also be compared to other results stored in results logs
on the system (including comparative bar graphs). Five sets of results
are included with the benchmark for comparison purposes:
* MMX RGB Software Emulation
* Matrox Mystique
* Non-MMX RGB Software Emulation
* Orchid Righteous 3D
* Rendition Verite 1000 Reference Board
The results may be saved in a standard log format, as well as exported in
table or chart form to several formats including text, MS Excel, and
Windows Metafile format. Unfortunately, this does not provide much
flexability; the charts must be imported into an application that
supports the WMF format and cannot be easily edited.
Along with this review are the attached results of preliminary tests we
ran on a Permedia-based accelerator board, along with the results for RGB
software emulation and the Orchid Righteous 3D for comparison purposes.
It appears that the Permedia tests did not calculate a WinMark score due
to the fact that some of the required features were either not supported
by the card or turned off during the 3D Quality tests. Running the
complete suite of tests on the Permedia-based card on a 166 MHz Pentium
PC took about 40 minutes.
CONCLUSION
3D WinBench 97 finally brings 3D benchmarking and performance
characterization to a standard reference point and expands on past 3D
benchmarking efforts already present in the industry. As ZD-BOP's first
effort at releasing a 3D benchmark, they have presented impressive
results and maintain their spot as the authority on consumer-level
benchmarking. Although 3D WinBench 97 could use improvement in many
areas, such as those described above, it presents an important first step
towards becoming "the" definitive 3D benchmark that the industry has been
waiting for.
www.zdbop.com
Wave Issue 9710 5/8/97 Article 3-01