The Beginning : How it all started
Even yesterday, an allegedly Snapdragon S800 Mi3 leaked out on Antutu and there was a lot of talking surrounding the low score and a few theories quickly attached to it.
Cheating in benchmarking has always been the most popular sport in the PC world. We were just hoping it wouldn't becoming one so fast so soon in Android. The first one caught was the Samsung Galaxy 4 in late June via a post at Beyond3D from @AndreF.
Lately, Samsung, has been caught doing it again with the Note 3. This time, it was Arstechnicawhich spotted it.
Samsung took most of the heat, but what about others manufacturers ?
1) The State of the Art :
Two ways were used in order to cheat :
- the Samsung Exynos Chip 5410 (4x Cortex A15 and 4 Cortex A7) raised thermal limits when a benchmark is detected, so the SoC is allowed to go faster or keeping max speed longer
- Qualcomm chips are boosted to run at max speed with all cores when a benchmark is detected.
Anandtech did a great job and tested devices from end 2012 to this day. They also revealed, through OEM, that this is going on for the last two years at the very least. So Samsung is just the tree hiding the whole forest. This is, of course, not exclusive to ARM based chips, Intel, from all its experience in the PC field, brought all its savoir-faire.
Here is a table done by Anandtech.
It seems only Google devices don't cheat alongside the nVidia console gaming. That is why the Nexus 4 was trailing behind all the Qualcomm Snapdragon S4 pro last year.
2) Why It's Cheating
The latest Exynos chips don't equip so many devices aside from Samsung Galaxy S4, Note 3 and the Meizu MX3.
But Qualcomm is all over the place when it comes to high end devices where the benchmarks war is raging. While some would argue that since the max speed of the S800 chips is 2.3 GHz, it wouldn't be cheating to force the 4 cores to run at max speed all the time, right ?
Well, those people couldn't be more wrong. Chips are designed to be allowed to go as fast as those limits but not all the time because they have to manage their own thermal conditions and the consequences on the rest of the device like the battery which the overall life is being reduced.
So, in normal conditions, even in very heavy games or benchmarks when the SoCs are much needed, there are times when everything goes slower in order to respect thermal conditions and designed safety limits. Those same limits are lifted in those benchmarks to the sole purpose to boost the results because those same SoCs won't behave like this in real conditions.
Here is what we are talking about.
As for Xiaomi, particularly for the Mi3, we will do a video demonstration to know if the Chinese firm boosts its results in our review coming tomorrow.
Conclusion
So far, except for the Exynos series that can raise up the thermal limit, the cheating has been targeted to CPU. GFXBench which is the reference GPU wise, has managed to keep clean sheets through Kishonti Ltd connections (every manufacturer is paying a licensing fee) and work ethics but there's no telling if Kishonti can maintain this balance. We certainly hope so.
But others benchmarks heavily based on GPU like Basemark X has seen different results from the Galaxy Note 3 when it is running the same renamed benchmark.
The 10% boost is a clear sign of GPU tweaking. Same goes for others CPU/GPU benchmarks like 3DMark which is being targeted by HTC, then Samsung.
The trend shows that it is only the beginning and while we were lucky enough to detect CPU tweaking, GPU is shown to be far more challenging to detect.
As PC showed, it is bound to get worse, so please, everyone needs to take a step back when talking about Benchmarks results even in those where no cheating is detected. There is no telling what tomorrow brings for us.
Never underestimate manufacturers ability to come up with new ways to cloud results.
Stay tuned on PhonAsia and discover how we managed to easily show what the Mi3 really holds for us.
0 comments:
Post a Comment