Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headache
by Andrei Frumusanu & Ian Cutress on September 4, 2018 8:59 AM EST- Posted in
- Smartphones
- Huawei
- SoCs
- Benchmarks
- honor
- Kirin 970
The Raw Benchmark Numbers
Section By Andrei Frumusanu
Before we go into more details, we're going to have a look at how much of a difference this behavior contributes to benchmarking scores. The key is in the differences between having Huawei/Honor's benchmark detection mode on and off. We are using our mobile GPU test suite which includes of Futuremark’s 3DMark and Kishonti’s GFXBench.
The analysis right now is being limited to the P20’s and the new Honor Play, as I don’t have yet newer stock firmwares on my Mate 10s. It is likely that the Mate 10 will exhibit similar behaviour - Ian also confirmed that he's seeing cheating behaviour on his Honor 10. This points to most (if not all) Kirin 970 devices released this year as being affected.
Without further ado, here’s some of the differences identified between running the same benchmarks while being detected by the firmware (cheating) and the default performance that applies to any non-whitelisted application (True Performance). The non-whitelisted application is a version provided to us from the benchmark manufacturer which is undetectable, and not publicly available (otherwise it would be easy to spot).
We see a stark difference between the resulting scores – with our internal versions of the benchmark performing significantly worse than the publicly available versions. We can see that all three smartphones perform almost identical in the higher power mode, as they all share the same SoC. This contrasts significantly with the real performance of the phones, which is anything but identical as the three phones have diferent thermal limits as a result of their different chassis/cooling designs. Consequently, the P20 Pro, being the largest and most expensive, has better thermals in the 'regular' benchmarking mode.
Raising Power and Thermal Limits
What is happening here with Huawei is a bit unusual in regards to how we’re used to seeing vendors cheat in benchmarks. In the past we’ve seen vendors actually raise the SoC frequencies, or locking them to their maximum states, raising performance beyond what’s usually available to generic applications.
What Huawei instead is doing is boosting benchmark scores by coming at it from the other direction – the benchmarking applications are the only use-cases where the SoC actually performs to its advertised speeds. Meanwhile every other real-world application is throttled to a significant degree below that state due to the thermal limitations of the hardware. What we end up seeing with unthrottled performance is perhaps the 'true' form of an unconstrained SoC, although this is completely academic when compared to what users actually expereience.
To demonstrate the power behaviour between the two different throttling modes, I measured the power on the newest Honor Play. Here I’m showcasing total device power at fixed screen brightness; for GFXBench the 3D phase of the benchmark is measured for power, while for 3DMark I’m including the totality of the benchmark run from start to finish (because it has different phases).
The differences here are astounding, as we see that in the 'true performance' state, the chip is already reaching 3.5-4.4W. These are the kind of power figures you would want a smartphone to limit itself to in 3D workloads. By contrast, using the 'cheating' variants of the benchmarks completely explodes the power budget. We see power figures above 6W, and T-Rex reaching an insane 8.5W. On a 3D battery test, these figures very quickly trigger an 'overheating' notification on the device, showing that the thermal limits must be beyond what the software is expecting.
This means that the 'true performance' figures aren’t actually stable - they strongly depend on the device’s temperature (this being typical for most phones). Huawei/Honor are not actually blocking the GPU from reaching its peak frequency state: instead, the default behavior is a very harsh thermal throttling mechanism in place that will try to maintain significantly lower SoC temperature levels and overall power consumption.
The net result is that that in the phones' normal mode, peak power consumption during these tests can reach the same figures posted by the unthrottled variants. But the numbers very quickly fall back in a drastic manner. Here the device thottles down to 2.2W in some cases, reducing performance quite a lot.
Facebook
Twitter
RSS
84 Comments
View All Comments
Amandtec - Wednesday, September 5, 2018 - link
Actually, social science style studies reveal that (ballpark) 10% of people always cheat, 10% never cheat no matter what, and the 80% in the middle are a continum who are mostly swayed by perceived risks and incentives. If you are always mistrusting everyone it prevents one from building profitable relations.skoondi - Sunday, September 9, 2018 - link
Not sure about Chinese culture but you'll find more than a few people care about phone GPU performance, I can only wonder why you are on a site basically dedicated to people who are interested in such things Phones are the new computer, with a decent docking set up many people probably don't need a desktop or laptop. I've been gaming since the days of the first civilisation game and currently do a lot of gaming on my phone.techconc - Monday, September 10, 2018 - link
@shogun18 - You said:"But I also have to wonder, are there really more than a 100 basement-dwelling morons who give a flying F about how fast a GPU is on a stupid phone? And furthermore play games on such pointless platforms where frame rates or triangles/sec matters?"
It seems your comments are very much out of touch with what is happening today, even in the world of gaming.
PeachNCream - Tuesday, September 4, 2018 - link
I don't get the point in cheating on phone benchmarks or even the point of phone benchmarks in general. I also don't get the point of sticking an auto playing video about buying the right CPU on EVERY DAMNED PAGE of this article. The same video. Every time! If seeing it on the first page pissed me off and I didn't watch it there (while also being enraged at the auto starting nature of it) what on Earth would make you think it's a good idea to do it to me three more times in the same article? Did the Purch buyout somehow involve a lobotomy that removed logic and reason or was that SOP already in place beforehand?vgray35@hotmail.com - Tuesday, September 4, 2018 - link
I would use stronger words than that to really convey the egregious nature of repetitive onslaughts of their bloody (not a swear word) videos against us. Their blatant "up yours, screw you" attitude turns rage to a frothing mouth frenzy ready to spill blood (especially on Toms Hardware). What the flying F is wrong with allowing users to decide whether to hit the play button or not? Or am I really only talking to a lobotomized miscreant that should be restrained from interacting with the public. Cease and desist auto play videos! A "up yours, screw you" attitude is not a smart way to deal with the public. Toms are you listening, or are sending us all another finger image?shogun18 - Tuesday, September 4, 2018 - link
er it's 2018. How the hell have you been using the internet for the last decade without all browser plugins disabled or at the very least noScript (or similar) with or without AdBlock (or similar)?vgray35@hotmail.com - Tuesday, September 4, 2018 - link
You missed the point. Most of us are clueless.shogun18 - Tuesday, September 4, 2018 - link
No, I totally get your point and I agree (with) you have every right to be incensed with auto-play videos. However a random comment here is unlikely to get the author or site-admin's attention. And said person is doubtless unable to oppose corporate policy. When Anand's site readership takes a major nose dive (further?) and the morons in marketing (but I repeat myself) can be beaten over the head that one of the major reasons they are losing revenue and clicks is because of auto-play, THEN and ONLY then will the practice stop.There haven't been more than a minuscule handful (again, basically zero given the side of the ecosystem) that could be described as "responsible" or "good" website operators for well over 10 years. Therefore if you haven't been black-listing every feature in your browser and toggling the choices conveniently given to you, then your ire is misplaced IMO. I am an equal opportunity banner of site content. Join the club and be amazed at far more enjoyable the experience is.
vgray35@hotmail.com - Tuesday, September 4, 2018 - link
Put that way you may be right.1_rick - Tuesday, September 4, 2018 - link
Well, Purch has been pushing those annoying ads on Tom's Hardware for quite some time now, so I guess it's only to be expected they'd be forcing them on this site as well.TBH I hadn't even noticed the ads because I used my hosts file to block assets.purch.com, because my adblocker wasn't taking care of it. You could try that too.