Despite the variety of generally accepted tests, recently their results have often added confusion than to clarify the essence of the phenomena.
There are tests that allow you to measure the performance of a computer subsystem, for example, the speed of data exchange between a processor and memory or cache. These tests are necessary in order to find out the impact of individual technical solutions, more precisely, to understand why a particular processor is better or worse than others. But the practical sense in these tests is quite small.
There are tests based on measuring the execution time of individual mathematical problems. These are very good tests. They have good repeatability of results, including on different platforms. Their only drawback is that they correctly reflect the capabilities of the processor only when solving a specific problem and only according to a specific algorithm used in the test. As a result, the results of such tests are uninformative. In fact, think about quotes from real publications on the Internet: “: According to this test, based on the algorithm for constructing the Mandelbrot set, it turns out that the Athlon XP processor SSE instruction block is very good.” Or so: “: in the Primodia test, based on measuring the time it takes to solve the Schrödinger equation for all 61 electrons of the Prometheus element, all Pentium 4 processors (with both the Willamette core and the Northwood core) are ahead of the entire Athlon XP line.” Does the majority buy computers with the intention of solving the Schrödinger equation faster or building this “Mandelbrot set”?
There are tests based on measuring performance in real-life, most commonly used applications. A classic example is Content Creation Winstone. The latest version of this popular test application includes Adobe Photoshop 6.0.1, Adobe Premiere 6.0, Macromedia Director 8.5, Macromedia Dreamweaver UltraDev 4, Netscape Navigator 6 / 6.01, Sonic Foundry Sound Forge 5.0c (build 184) and Microsoft Windows Media Encoder 7.01.00.3055 . The idea is very good, until at least until we try to compare processors with different architectures. Here it turns out that Microsoft Windows Media Encoder 7.01.00.3055, when it determines that it works with the Athlon processor, disables all optimization. At least it doesn’t use the SSE instruction set supported by Athlon processors. Therefore, a comparison of the performance of different processors by the criterion of the execution time of operations in the Microsoft Windows Media Encoder application dooms Athlon processors in advance. In this test, one processor uses the maximum of its capabilities, and the second only basic capabilities. Is it a sport or mayhem?
But this is not the main thing. Performance is important primarily for professional work with demanding applications, for example, Adobe Photoshop, Adobe Premiere. What does Netscape Navigator have to do with it? And vice versa, if surfing on the Internet is most important, then what does Sonic Foundry Sound Forge have to do with it? Especially if the system does not have Netscape installed, but standard IE? What then do the results of the Content Creation Winstone test tell us? It is no coincidence that quite often a “significant” conclusion is made based on the test results (again, a quote): “: In fact, Athlon XP demonstrates its superiority only where the CPU requires high computing power and does not use the SSE2 instruction set.”
Since we believe that it makes sense to measure performance only in cases where “high computational power is required from the CPU”, we prefer to measure performance only on real applications, and at the same time we are talking about processor performance not in general, but about performance on specific applications. And this indicator can differ significantly from the estimates obtained in some kind of artificial environment.
In our opinion, one of the most common tasks in which performance is really important is the non-linear editing of films. Well, do not fight for performance in a Microsoft Word environment?