SPC Releases Storage Benchmarks to General Acclaim and a Little Dissent


This week, the Storage Performance Council (SPC) unveiled preliminary
results of SPC-1, the first in a series of industry benchmarks for
enterprise storage systems. On Monday, the group will release more
details about the scores, meant to help customers compare the
performance of vendors’ storage systems in OLTP (online transaction
processing) database environments.


“This is the beginning of the ability to provide a level playing field
and demonstrable methodology,” maintained Walter E. Baker, SPC
administrator and auditor, in a teleconference this week.


During the teleconference, speakers drew frequent comparisons between
SPC-1 and server benchmarks released over the past few years by the
Transaction Processing Performance Council (TPC).


IBM, Sun, and LSI are the first three vendors to publish their SPC-1
results. Storage giant EMC previously belonged to the SPC, but
resigned.


Information made available by the SPC this week provides preliminary
metrics, or raw data, for the three tested storage systems. The
results include maximum throughput, measured in IOPs; average or
optimal response time, measured in LRTs; total capacity in GB; and
data protection levels.


On Monday, though, the SPC plans to post full disclosures and
executive summaries on its Web site (http://www.storageperformance.org
). The new information will provide product pricing, as well as
configuration details.


When full results become available from more vendors, customers will
be able to use the information to help narrow down their product
choices, according to Baker. “It’s a screen to get to a short list,”
he said.


The SPC has been testing vendors’ storage systems under the new
benchmark for the past three months. In the highly fractionalized
storage industry, though, the benchmark took four-and-a-half years to
get off the ground.


The initiative started with a year of informal discussions back in
1998. “It’s been a lot of work,” acknowledged SPC Founder Roger Reich,
who is also senior technical director at Veritas Software.


Now, though, the SPC has arrived at a “complete package,” including a
specification, test kit, tools, and benchmark results, according to
Reich. “We have industry analysts on board,” he added. The SPC has
named two “Preferred Analysts” – Ideas International and Evaluator
Group – to help customers sort through the benchmark results.


In June, the SPC expects to release a multi-platform SPC-1 test kit in
June to users, manufacturers, integrators, and academic institutions.


Also in the third quarter, the SPC will release Goliath, a “superset”
of tools that will include the test kit, along with a performance
analysis environment.


SPC member Hitachi Data Systems (HDS) also took part in the initial
SPC-1 benchmark tests, but hasn’t released its results yet. “We
supported Sun in completing theirs. We do support the benchmark,
though, and we plan to release our results in a very few weeks. We
literally moved the earth in two-and-a-half weeks,” contended Robert
Ward, Hitachi’s director of product management.


Corroborated Leah Schoeb, chairman SPC and a staff engineer at Sun:
“We are a major reseller of HDS. They were a big help to us.”


Other current SPC members have been “in the middle of product cycles,”
according to Baker. “The date didn’t match up with their product
lifetime. I would anticipate (other vendors’ results), however, over
the next four to six weeks,” he said.


In previous industry bench-marketing initiatives, there have been “many
cases where it would be several weeks before you even had one result,”
Baker added.


Customers will be able to use the SPC-1 results to arrive at
price/performance ratios by dividing the price of a system by IOPs or
LRTs. In making buying decisions, though, many other factors also
should be weighed, including needs for cache and redundancy, SPC
members suggested.


Initial results from IBM, Sun and LSI “cover a good spectrum,” said
Reich. “Each (storage system) demonstrates a value proposition.”
Product comparisons, however, “will get a good bit more barnacled”
when more vendors enter the SPC-1 ring, according to Reich.


The SPC hasn’t yet decided which project to work on
next. Possibilities already raised include a sequentially oriented
benchmark, for streaming applications; a mass storage benchmark, and
extensions to SPC-1 to “make sure it represents SAN performance,”
Baker said.


Baker applauded SPC members for the “unusual” degree of collaboration
shown during the benchmarking process. In a Q&A session, one
participant noted that when industry player EMC left the SPC, it
claimed the SPC-1 specification doesn’t reflect “real world”
conditions.


Baker said he encountered similar kinds of industry dissension in his
own previous work with the TPC. “No matter what, there was always a
feeling (by someone) that (a specification) wasn’t representative of
the real world,” according to Baker.


Aside from IBM, Sun, Hitachi, and Veritas, other members of the SPC
include Hewlett-Packard; Compaq; Dell; Adaptec; NEC; and Unisys.


The SPC has also been talking with storage switch specialist
Brocade. Brocade, though, probably won’t see much reason to join
unless the SPC decides to benchmark implementations that include
switches, according to Baker.


»


See All Articles by Columnist
Jacqueline Emigh

Latest Articles

Follow Us On Social Media

Explore More