Benchmarking browsers

- does Opera deliver on speed promise with 10.5 pre alpha?

Introduction

The launch of Opera 10.5 pre alpha prompted me to finally do some browser benchmarking myself. I've often been critical on the way other benchmarks were performed. It wouldn't always be clear which benchmarks were used and under which circumstances. Sometimes I simply wouldn't agree on the environments used, such as browsers running on virtual machines, which could have an unequal influence on some browsers. Besides, as an Opera user, I was keen to immediately find out if the promise of Opera's developers - to deliver speed - was fulfilled.

Author: Rachid Finge

Testing and Methods

The following browsers were tested during this benchmark, listed in alphabetical order:

The reason for testing two versions of Opera is to find out the scope of the performance improvements promised in Opera 10.5 over the older branch of the browser, of which Opera 10.2 alpha is the latest available to the public. While there also is a 64-bit version of Internet Explorer 8, I have not included it in the test, since the 32-bit version is the one Microsoft puts forward as the default version to use in Windows 7.

All tests were conducted on a PC with the following specifications:

All potentially intruding background processes, such as virus scanners and mail clients, were terminated prior to benchmarking. The Windows Task Manager confirmed that CPU usage was 0% for a full five minutes before the first benchmark took place.

All mentioned browsers were benchmarked using Futuremark's Peacekeeper benchmark and the Sunspider JavaScript benchmark, developed by the WebKit team. I chose those two benchmarks as they seem to me the most reliable and most popular and are easy to run multiple times on multiple browsers, making comparisons possible in a reliable sense.

A browser test was conducted as follows. The browser being tested was start from the Windows 7 Superbar. All tabs were closed and data such as history, cookies and cache were cleared. Then, through Google Search, the Sunspider benchmark was loaded and executed. The total execution time was written down on a piece of paper. The benchmark was conducted twice more to get a total of three samples per benchmark per browser. The median value was used in the final results.

After Sunspider, Peacekeeper was loaded in the same tab and executed. The first browser being tested, Google Chrome, performed a system scan as requested by Peacekeeper. Other browsers didn't need to do the test, as a special url was given by Peacekeeper, transferring the system scan data and previous benchmarking results to other browsers for easy comparisons. As with Sunspider, the Peacekeeper benchmark was run three times, taking the median points score to the final results. The points scored on the Complex Graphics part of the benchmark are not taken into account into the final points calculation of Peacekeeper, as Internet Explorer does not support the required canvas HTML element. Scores of the Complex Graphics section were however also noted for seperate interpretation.

Results

Following are graphs that show the results of the benchmarks per benchmark.

Figure 1: results of the Sunspider benchmark in milliseconds, per browser. Shorter bars indicate higher performance.
Figure 1: results of the Sunspider benchmark in milliseconds, per browser. Shorter bars indicate higher performance.

Figure 2: results of the Peacekeeper benchmark in points, per browser. Longer bars indicate higher performance.
Figure 2: results of the Peacekeeper benchmark in points, per browser. Longer bars indicate higher performance.

Figure 3: results of the Peacekeeper benchmark, Complex Graphics section, in points, per browser. Longer bars indicate higher performance.
Figure 3: results of the Peacekeeper benchmark, Complex Graphics section, in points, per browser. Longer bars indicate higher performance.

Discussion

It appears the developers at Opera have more than kept their word: not only is 10.5 pre alpha magnitudes faster than the previous version of Opera, it also edges out all of its competitors in the two benchmarks used for this test. According to the Sunspider benchmark on the test system, Opera 10.5 pre alpha is literally ten times faster than its 10.2 counterpart and 20% faster than Google Chrome, the next fastest browser in JavaScript processing. Looking at the Peacekeeper benchmark, Opera 10.5 pre alpha scores more than twice as much points as Opera 10.2 and about 8% more points than Google Chrome and Apple Safari, who mark second and third in this benchmark respectively.

The final test, the Complex Graphics section of the Peacekeeper benchmark, again puts Opera 10.5 pre alpha in the lead by 35% over number two, which again is Google Chrome. Safari is beaten by Mozilla's Firefox only in this test. Internet Explorer did not produce a result as it doesn't support the canvas HTML element, which is required to pass this test.

It should be noted that the results on Windows cannot be directly translated to other operating systems. It is known that Apple's Safari runs quicker in Mac OS X Snow Leopard due to specific optimizations not available in Windows, while Opera has suggested that its optimizations in Opera 10.5 on Mac are "not as far along" as on Windows. Also, while the versions of Google Chrome and Apple Safari used in this test are already in production state, Opera 10.5 is not. The current build should not be used for day to day browsing, Opera has stated.

Conclusion

Norway's Opera has got a winner on its hand: its latest iteration of the browser could finally retake the claim that it is the fastest browser on earth. While its edge on JavaScript over competitors is not likely to be significant, it shows great gains in Futuremark's Peacekeeper benchmark.Chrome remains the solid number two in this test, strictly being the fastest in-production browser. Mozilla Firefox never really shines in this test, although it still keeps Microsoft's Internet Explorer at a distance in all tests.