News

A Question of Scale

• By Scott Bekker
• 07/01/2005

Microsoft went part way toward answering those concerns at Microsoft TechEd 2005 in June. Microsoft worked with Hewlett-Packard and Intel to publish results for SQL Server 2005 on the TPC-C (OLTP) and TPC-H (decision support/business intelligence) benchmarks.

First, the numbers. Running on a 64-way HP Superdome server with Itanium 2 processors, SQL Server 2005 achieved a TPC-C result of 1,082,203 transactions per minute (tpmC) at a cost of $5.38 per tpmC. Both are respectable results and represent Microsoft's first time to pass the aesthetically satisfying threshold of 1 million transactions. Still, the result puts Microsoft fourth on the OLTP scalability list.

A massive storage array supports the SQL Server 2005 benchmark runs.

On the TPC-H benchmark, Microsoft graduated to a new scalability category but came in about halfway down the list of results for that grouping. TPC-H results are segmented into five categories based on the size of the workload. The categories include 100GB, 300GB, 1TB, 3TB and 10TB. SQL Server 2000 competed in the 100GB workload range, but hadn't cracked the Top 10 in even the 300 GB category. With SQL Server 2005, Microsoft is ranked fifth in the 1 TB grouping. SQL Server 2005 recorded 20,231 queries per hour on the TPC-H (QphH) at a cost of $77/QphH.

The results were not the kind of knock-it-out-of-the-park showings that Microsoft engineered with SQL Server 2000. When Windows 2000 launched in April 2000, Microsoft used pre-release code from SQL Server 2000 to produce a jaw-dropping first place result for TPC-C scalability. The benchmark run required a highly controversial clustered configuration that Microsoft later withdrew and resubmitted.

Then at the launch of Windows Server 2003, Microsoft, HP and Intel showcased the scalability of SQL Server 2000 on the new operating system with another first place result on the TPC-C. This time it was a more socially acceptable configuration, using a 64-processor HP Superdome similar to the current configuration.

Although this isn't a scalability home run, it puts to bed any speculation that SQL Server 2005 might not scale better than SQL Server 2000. The new system beat SQL Server 2000 on similar hardware by 38 percent. It also compares well against the Oracle 10g database from a 2003 TPC-C run on a similar HP Superdome configuration--beating Oracle's raw performance by 7 percent.

Microsoft got strong but not record-setting results in benchmarking SQL Server 2005 on the HP Superdome server.

On the hardware side, Microsoft's competitive decision to limit SQL Server to the Windows platform hurts the database's ability to really be tested against database peers from Oracle and IBM. SQL Server has nowhere else to go--the 64-way HP Superdome running Itanium 2 processors has the most headroom of any system available to Windows. On an IBM 64-way pSeries running IBM AIX, however, Oracle 10g scored higher on the TPC-C. Using its own IBM DB2 database on the Unix/RISC box, IBM's results are triple Microsoft's.

And when it comes to decision support, SQL Server 2005 is nowhere near the top of the scale. Microsoft's database still doesn't play in the 3 TB or 10 TB classes.

Microsoft resorted to familiar, but reasonable, defenses. Paul Flessner, senior vice president for Microsoft server applications, pointed out that Microsoft's TPC-C results are several times Visa's daily traffic.

In the end, Microsoft's SQL Server 2005 benchmarks put to rest any doubts that SQL Server 2005 won't scale better than SQL Server 2000. The fact that Microsoft didn't also resort to a scalability stunt to produce a record-breaking benchmark indicates the confidence the database vendor is feeling about scalability.