SPC Releases Storage Benchmarks to General Acclaim and a Little Dissent
It's taken four years, but the Storage Performance Council has finally unveiled online transaction processing benchmarks as dissenters claim flawed reflections of real world performance.
This week, the Storage Performance Council (SPC) unveiled preliminary results of SPC-1, the first in a series of industry benchmarks for enterprise storage systems. On Monday, the group will release more details about the scores, meant to help customers compare the performance of vendors' storage systems in OLTP (online transaction processing) database environments.
"This is the beginning of the ability to provide a level playing field and demonstrable methodology," maintained Walter E. Baker, SPC administrator and auditor, in a teleconference this week.
During the teleconference, speakers drew frequent comparisons between SPC-1 and server benchmarks released over the past few years by the Transaction Processing Performance Council (TPC).
Information made available by the SPC this week provides preliminary metrics, or raw data, for the three tested storage systems. The results include maximum throughput, measured in IOPs; average or optimal response time, measured in LRTs; total capacity in GB; and data protection levels.
On Monday, though, the SPC plans to post full disclosures and executive summaries on its Web site (http://www.storageperformance.org ). The new information will provide product pricing, as well as configuration details.
When full results become available from more vendors, customers will be able to use the information to help narrow down their product choices, according to Baker. "It's a screen to get to a short list," he said.
The SPC has been testing vendors' storage systems under the new benchmark for the past three months. In the highly fractionalized storage industry, though, the benchmark took four-and-a-half years to get off the ground.
The initiative started with a year of informal discussions back in 1998. "It's been a lot of work," acknowledged SPC Founder Roger Reich, who is also senior technical director at Veritas Software.
Now, though, the SPC has arrived at a "complete package," including a specification, test kit, tools, and benchmark results, according to Reich. "We have industry analysts on board," he added. The SPC has named two "Preferred Analysts" - Ideas International and Evaluator Group - to help customers sort through the benchmark results.
In June, the SPC expects to release a multi-platform SPC-1 test kit in June to users, manufacturers, integrators, and academic institutions.
Also in the third quarter, the SPC will release Goliath, a "superset" of tools that will include the test kit, along with a performance analysis environment.
SPC member Hitachi Data Systems (HDS) also took part in the initial SPC-1 benchmark tests, but hasn't released its results yet. "We supported Sun in completing theirs. We do support the benchmark, though, and we plan to release our results in a very few weeks. We literally moved the earth in two-and-a-half weeks," contended Robert Ward, Hitachi's director of product management.
Corroborated Leah Schoeb, chairman SPC and a staff engineer at Sun: "We are a major reseller of HDS. They were a big help to us."
Other current SPC members have been "in the middle of product cycles," according to Baker. "The date didn't match up with their product lifetime. I would anticipate (other vendors' results), however, over the next four to six weeks," he said.
In previous industry bench-marketing initiatives, there have been "many cases where it would be several weeks before you even had one result," Baker added.
Customers will be able to use the SPC-1 results to arrive at price/performance ratios by dividing the price of a system by IOPs or LRTs. In making buying decisions, though, many other factors also should be weighed, including needs for cache and redundancy, SPC members suggested.
Initial results from IBM, Sun and LSI "cover a good spectrum," said Reich. "Each (storage system) demonstrates a value proposition." Product comparisons, however, "will get a good bit more barnacled" when more vendors enter the SPC-1 ring, according to Reich.
The SPC hasn't yet decided which project to work on next. Possibilities already raised include a sequentially oriented benchmark, for streaming applications; a mass storage benchmark, and extensions to SPC-1 to "make sure it represents SAN performance," Baker said.
Baker applauded SPC members for the "unusual" degree of collaboration shown during the benchmarking process. In a Q&A session, one participant noted that when industry player EMC left the SPC, it claimed the SPC-1 specification doesn't reflect "real world" conditions.
Baker said he encountered similar kinds of industry dissension in his own previous work with the TPC. "No matter what, there was always a feeling (by someone) that (a specification) wasn't representative of the real world," according to Baker.
Aside from IBM, Sun, Hitachi, and Veritas, other members of the SPC include Hewlett-Packard; Compaq; Dell; Adaptec; NEC; and Unisys.
The SPC has also been talking with storage switch specialist Brocade. Brocade, though, probably won't see much reason to join unless the SPC decides to benchmark implementations that include switches, according to Baker.