UL Delists Huawei Devices Due To Inaccurate Benchmarks
by Andrei Frumusanu on September 6, 2018 12:55 PM EST- Posted in
- Mobile
- Futuremark
- Smartphones
- 3DMark
- UL
A few days ago we published our article addressing Huawei and Honor’s inaccurate benchmarking behaviour. In a nutshell, we had found out that this year’s devices had introduced a new thermal limiting behaviour that quickly throttled power consumption to ~4-4.5W in 3D workloads. While this in itself wasn’t an issue, the problem is that the firmware did not apply this new behaviour to a specific list of whitelisted 3D benchmarks.
Sidenote: Earlier this year, Futuremark (The 3DMark creators) had been acquired by UL, joining the global safety consulting and certification company as a benchmarking division.
We work closely with all benchmark vendors, and UL isn’t an exception. We had first given UL note of the behaviour two weeks ago and had been sharing our early results with the development team. Today UL published their independent confirmation of our results on their own devices, and have subsequently decided to remove the tested devices from their results database.
UL stresses that the kind of detection and optimization performed by Huawei infringes the company’s rules for manufacturers. Unfortunately this isn’t the first instance of a vendor being delisted, as most famously a slew of phone manufacturer had been delisted in 2013 after a more in-depth investigation of ours resulted in quite embarrassing results for a lot of vendors.
The UL team further explains that they’ve also been in touch with Huawei, and that the Chinese vendor is planning to address the behaviour by introducing a new “performance mode” that disables the new thermal throttling behaviour. In essence this mode would revert back to the behaviour we’ve seen in the past such as the Mate 9 – where the SoC is allowed higher peak performance figures at a cost of high power.
While this is still problematic for representative benchmarking for devices with Kirin 960’s and 970’s (due to the very large peak power at the highest performance states of those SoCs), I expect this to be a non-issue for the new Kirin 980, as it’s projected to have significantly better sustainable peak power figures. We're still looking forward to the new SoC in the upcoming Mate 20, and believe it to be an outstanding performer.
Source: UL News Release
33 Comments
View All Comments
bji - Thursday, September 6, 2018 - link
Actually, you are seeing *fewer* and *fewer* posts without capitals. Less/fewer misuse is also annoying!"Less" is for uncountable things, like water. So you can have "less water", but not "fewer water".
"Fewer" is for countable things, like "cups of water". So you can have "fewer cups of water", but not "less cups of water".
grahad - Monday, September 10, 2018 - link
Philosoraptor: Can you truly count the number of posts on the internet? Kappadamianrobertjones - Thursday, September 13, 2018 - link
When I visit the MS Word thesaurus, to view the word 'less', the first description is 'fewer' (A reduced amount).Good info though. Thanks.
Morawka - Thursday, September 6, 2018 - link
Says the guy who can't even capitalize his own sentences. Also "who the hell do you do this" makes no sense.thexile - Thursday, September 6, 2018 - link
Clearly you are the dumb one.Manch - Friday, September 7, 2018 - link
Zmeul bitches about the title, as he writes a comment that would bleed red if corrected. LOLspartaman64 - Sunday, September 9, 2018 - link
i missed youcasperes1996 - Thursday, September 6, 2018 - link
I love the fact that so far, literally every single one of the 9 comments here is about grammar, and correcting what I presume to be a troll, and nobody talks about the actual articlebubblyboo - Thursday, September 6, 2018 - link
Diversion created by a Huawei employee?Holliday75 - Thursday, September 6, 2018 - link
Seems par for the course in today's social media environment.