Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headacheby Andrei Frumusanu & Ian Cutress on September 4, 2018 8:59 AM EST
- Posted in
- Kirin 970
Section By Ian Cutress & Andrei Frumusanu
Does anyone remember our articles regarding unscrupulous benchmark behavior back in 2013? At the time we called the industry out on the fact that most vendors were increasing thermal and power limits to boost their scores in common benchmark software. Fast forward to 2018, and it is happening again.
Benchmarking Bananas: A Recap
cheat: verb, to act dishonestly or unfairly in order to gain an advantage.
AnandTech exposing benchmark cheating on smartphones has a long and rich history. It is quite apt that this story goes full circle, as the one to tip off Brian on Samsung’s cheating behaviour on the Exynos Galaxy S4 a few years back was Andrei, who now writes for us.
When we exposed one vendor, it led to a cascade of discussions and a few more articles investigating more vendor involved in the practice, and then even Futuremark delisting several devices from their benchmark database. Scandal was high on the agenda, and the results were bad for both companies and end users: devices found cheating were tarnishing the brand, and consumers could not take any benchmark data as valid from that company. Even reviewers were misled. It was a deep rabbit hole that should not have been approached – how could a reviewer or customer trust what number was coming out of the phone if it was not in a standard ‘mode’?
So thankfully, ever since then, vendors have backed off quite a bit on the practice. Since 2013, for several years it would appear that a significant proportion of devices on the market are behaving within expected parameters. There are some minor exceptions, mostly from Chinese vendors, although this comes in several flavors. Meizu has a reasonable attitude to this, as when a benchmark is launched the device puts up a prompt to confirm entering a benchmark power mode, so at least they’re open and transparent about it. Some other phones have ‘Game Modes’ as well, which either focus on raw performance, or extended battery life.
Going Full Circle, At Scale
So today we are publishing two front page pieces. This one is a sister article to our piece addressing Huawei’s new GPU Turbo, and while it makes overzealous marketing claims, the technology is sound. Through the testing for that article, we actually stumbled upon this issue, completely orthogonal to GPU turbo, which needs to be published. We also wanted to address something that Andrei has come across while spending more time with this year’s devices, including the newly released Honor Play.
The Short Detail
As part of our phone comparison analysis, we often employ additional power and performance testing on our benchmarks. While testing out the new phones, the Honor Play had some odd results. Compared to the Huawei P20 devices tested earlier in the year, which have the same SoC, the results were also quite a bit worse and equally weird.
Within our P20 review, we had noted that the P20’s performance had regressed compared to the Mate 10. Since we had encountered similar issues on the Mate 10 which were resolved with a firmware update pushed to me, we didn’t dwell too much on the topic and concentrated on other parts of the review.
Looking back at it now after some re-testing, it seems quite blatant as to what Huawei and seemingly Honor had been doing: the newer devices come with a benchmark detection mechanism that enables a much higher power limit for the SoC with far more generous thermal headroom. Ultimately, on certain whitelisted applications, the device performs super high compared to what a user might expect from other similar non-whitelisted titles. This consumes power, pushes the efficiency of the unit down, and reduces battery life.
This has knock-on effects, such as trust, in how the device works. The end result is a single performance number is higher, which is good for marketing, but is unrealistic to any user with the device. The efficiency of the SoC also decreases (depending on the chip), as the chip is pushed well outside its standard operating window. It makes the SoC, one of the differentiating points of the device, look worse, all for the sake of a high benchmark score. Here's the example of benchmark detection mode on and off on the Honor Play:
|GFXBench T-Rex Offscreen Power Efficiency
(Total Device Power)
|AnandTech||Mfc. Process||FPS||Avg. Power
|Honor Play (Kirin 970) BM Detection Off||10FF||66.54||4.39||15.17 fps/W|
|Honor Play (Kirin 970) BM Detection On||10FF||127.36||8.57||14.86 fps/W|
We’ll go more into the benchmark data on the next page.
We did approach Huawei about this during the IFA show last week, and obtained a few comments worth putting here. Another element to the story is that Huawei’s new benchmark behavior very much exceeds anything we’ve seen in the past. We use custom editions of our benchmarks (from their respective developers) so we can test with this ‘detection’ on and off, and the massive differences in performance between the publicly available benchmarks and the internal versions that we’re using for testing is absolutely astonishing.
As usual with investigations like this, we offered Huawei an opportunity to respond. We met with Dr. Wang Chenglu, President of Software at Huawei’s Consumer Business Group, at IFA to discuss this issue, which is purely a software play from Huawei. We covered a number of topics in a non-interview format, which are summarized here.
Dr. Wang asked if these benchmarks are the best way to test smartphones as a whole, as he personally feels that these benchmarks are moving away from real world use. A single benchmark number, stated Huawei’s team, does not show the full experience. We also discussed the validity of the current set of benchmarks, and the need for standardized benchmarks. Dr. Wang expressed his preference for a standardized benchmark that is more like the user experience, and they want to be a part of any movement towards such a benchmark.
I explained that we work with these benchmark companies, such as Kishonti (GFXBench) and Futuremark (3DMark), as well as others, to help steer them in a way that is better represented for benchmarking. We explained employing a benchmarking mode to game test results is not a solution to solving what they see as a misrepresentation of user experience with these benchmarks. This is especially valid when the chip ends up with lower efficiency – but to be honest with test: the only way for it to be better related to user experience is to run it in the standard power envelope that every regular game runs in.
Huawei stated that they have been working with industry partners for over a year to find the best tests closest to the user experience. They like the fact that for items like call quality, there are standardized real-world tests that measure these features that are recognized throughout the industry, and every company works towards a better objective result. But in the same breath, Dr. Wang also expresses that in relation to gaming benchmarking that ‘others do the same testing, get high scores, and Huawei cannot stay silent’.
He states that it is much better than it used to be, and that Huawei ‘wants to come together with others in China to find the best verification benchmark for user experience’. He also states that ‘in the Android ecosystem, other manufacturers also mislead with their numbers’, citing one specific popular smartphone manufacturer in China as the biggest culprit, and that it is becoming ‘common practice in China’. Huawei wants to open up to consumers, but have trouble when competitors continually post unrealistic scores.
Ultimately Huawei states that they are trying to face off against their major Chinese competition, which they say is difficult when other vendors put their best ‘unrealistic’ score first. They feel that the way forward is standardization on benchmarks, that way it can be a level field, and they want the media to help with that. But in the interim, we can see that Huawei has also been putting its unrealistic scores first too.
Our response to this is that Huawei needs to be a leader, not a follower on this issue. I explained that the benchmarks we use (GFXBench) are well understood and are ‘standard’, and as real world as possible, but there are benchmarks we don’t use (AnTuTu) because they don’t mean anything. We also use benchmarks such as SPEC, which are very standard in this space, to evaluate an SoC and device.
The discussion then pivoted towards the decline in trust Huawei’s benchmark numbers in presentations as a result of this. We already take the data with a large grain of salt, but now we have no reason to listen to them as we do not know which values are in this ‘benchmark’ mode.
Huawei’s reaction to this is that they will ensure that future benchmark data in presentations is independently verified by third parties at the time of the announcement. This was the best bit of news.
While not explicitly stated in a clear line, Huawei is admitting to doing what they are doing, citing specific vendors in China as the primary reason for it.
We understand the impact that higher marketing numbers, however this is the worst way to do it – rather than calling out the competition for bad practices, Huawei is trying to beat them at their own game, and it’s a game in which everyone loses. For a company the size of Huawei, brand image is a big part of what the company is, and trying to mislead customers just for a high-score will backfire. It has backfired.
Huawei’s comments about standardized benchmarking are not new – we’ve heard it since time immemorial in the PC space, and several years ago, Arm was similarly discussing it with the media. Since then the situation has gotten better: the canned benchmark companies speak with game developers to develop real-world scenarios, but they also want to push the boundaries.
The only thing that hasn’t happened in the mobile space compared to the PC space on this is proper in-game benchmark modes that output data properly. This is something that is going to have to be vendor driven, as our interactions with big gaming studios on in-game benchmarks typically falls flat. Any frame rate testing on mobile requires additional software, which can require root, however Huawei recently disabled the ability to root their phones. Though we're told that that at some point in the future, Huawei will be re-enabling rooting for registered developers soon.
Overall, while it’s positive that Huawei is essentially admitting to these tactics, we believe the reasons for doing so are flimsy at best. The best way to implement this sort of ‘mode’ is to make it optional, rather than automatic, as some vendors in China already do. But Huawei needs to lead from the front if it ever wants to approach Samsung in unit sales.
Huawei did not go into how the benchmarking detection will be addressed in current and future devices. We will come back to the issue for the Mate 20 launch on October 16th.
Post Your CommentPlease log in or sign up to comment.
View All Comments
Amandtec - Wednesday, September 5, 2018 - linkActually, social science style studies reveal that (ballpark) 10% of people always cheat, 10% never cheat no matter what, and the 80% in the middle are a continum who are mostly swayed by perceived risks and incentives. If you are always mistrusting everyone it prevents one from building profitable relations.
skoondi - Sunday, September 9, 2018 - linkNot sure about Chinese culture but you'll find more than a few people care about phone GPU performance, I can only wonder why you are on a site basically dedicated to people who are interested in such things Phones are the new computer, with a decent docking set up many people probably don't need a desktop or laptop. I've been gaming since the days of the first civilisation game and currently do a lot of gaming on my phone.
techconc - Monday, September 10, 2018 - link@shogun18 - You said:
"But I also have to wonder, are there really more than a 100 basement-dwelling morons who give a flying F about how fast a GPU is on a stupid phone? And furthermore play games on such pointless platforms where frame rates or triangles/sec matters?"
It seems your comments are very much out of touch with what is happening today, even in the world of gaming.
PeachNCream - Tuesday, September 4, 2018 - linkI don't get the point in cheating on phone benchmarks or even the point of phone benchmarks in general. I also don't get the point of sticking an auto playing video about buying the right CPU on EVERY DAMNED PAGE of this article. The same video. Every time! If seeing it on the first page pissed me off and I didn't watch it there (while also being enraged at the auto starting nature of it) what on Earth would make you think it's a good idea to do it to me three more times in the same article? Did the Purch buyout somehow involve a lobotomy that removed logic and reason or was that SOP already in place beforehand?
firstname.lastname@example.org - Tuesday, September 4, 2018 - linkI would use stronger words than that to really convey the egregious nature of repetitive onslaughts of their bloody (not a swear word) videos against us. Their blatant "up yours, screw you" attitude turns rage to a frothing mouth frenzy ready to spill blood (especially on Toms Hardware). What the flying F is wrong with allowing users to decide whether to hit the play button or not? Or am I really only talking to a lobotomized miscreant that should be restrained from interacting with the public. Cease and desist auto play videos! A "up yours, screw you" attitude is not a smart way to deal with the public. Toms are you listening, or are sending us all another finger image?
shogun18 - Tuesday, September 4, 2018 - linker it's 2018. How the hell have you been using the internet for the last decade without all browser plugins disabled or at the very least noScript (or similar) with or without AdBlock (or similar)?
email@example.com - Tuesday, September 4, 2018 - linkYou missed the point. Most of us are clueless.
shogun18 - Tuesday, September 4, 2018 - linkNo, I totally get your point and I agree (with) you have every right to be incensed with auto-play videos. However a random comment here is unlikely to get the author or site-admin's attention. And said person is doubtless unable to oppose corporate policy. When Anand's site readership takes a major nose dive (further?) and the morons in marketing (but I repeat myself) can be beaten over the head that one of the major reasons they are losing revenue and clicks is because of auto-play, THEN and ONLY then will the practice stop.
There haven't been more than a minuscule handful (again, basically zero given the side of the ecosystem) that could be described as "responsible" or "good" website operators for well over 10 years. Therefore if you haven't been black-listing every feature in your browser and toggling the choices conveniently given to you, then your ire is misplaced IMO. I am an equal opportunity banner of site content. Join the club and be amazed at far more enjoyable the experience is.
firstname.lastname@example.org - Tuesday, September 4, 2018 - linkPut that way you may be right.
1_rick - Tuesday, September 4, 2018 - linkWell, Purch has been pushing those annoying ads on Tom's Hardware for quite some time now, so I guess it's only to be expected they'd be forcing them on this site as well.
TBH I hadn't even noticed the ads because I used my hosts file to block assets.purch.com, because my adblocker wasn't taking care of it. You could try that too.