3DMark for Android: Performance Preview
by Anand Lal Shimpi & Brian Klug on April 2, 2013 7:00 AM ESTAs I mentioned in our coverage of GL/DXBenchmark 2.7, with the arrival of Windows RT/8 we'd finally see our first truly cross-platform benchmarks. Kishonti was first out of the gate, although Futuremark was first to announce its cross platform benchmark simply called 3DMark.
Currently available for x86 Windows 8 machines, Futuremark has Android, iOS and Windows RT versions of 3DMark nearing release. Today the embargo lifts on the Android version of 3DMark, with iOS and Windows RT to follow shortly.
Similar to the situation with GL/DXBenchmark, 3DMark not only spans OSes but APIs as well. The Windows RT/8 versions use DirectX, while the Android and iOS versions use OpenGL ES 2.0. Of the three major tests in the new 3DMark, only Ice Storm is truly cross platform. Ice Storm uses OpenGL ES 2.0 on Android/iOS and Direct3D feature level 9_1 on Windows RT/8.
The Android UI is very functional and retains a very 3DMark feel. There's an integrated results brower, history of results and some light device information as well:
There are two options for running Ice Storm: the default and extreme presets.
3DMark - Ice Storm Settings | ||||||||
Default | Extreme | |||||||
Rendering Resolution | 1280x720 | 1920x1080 | ||||||
Texture Resolution | Normal | High | ||||||
Post-processing Quality | Normal | High |
Both benchmarks are rendered to an offscreen buffer at 720p/1080p and then scale up to the native resolution of the device being tested. This is a very similar approach we've seen by game developers to avoid rendering at native resolution on some of the ultra high resolution tablets. The beauty of 3DMark's approach here is the fact that all results are comparable, regardless of a device's native resolution. The downside is we don't get a good idea of how some of the ultra high resolution tablets would behave with these workloads running at their native (> 1080p) resolutions.
Ice Storm is divided into two graphics tests and a physics test. The first graphics test is geometry heavy while the second test is more pixel shader intensive. The physics test, as you might guess, is CPU bound and multithreaded.
Before we get to the results, I should note that a number of devices wouldn't complete the tests. The Intel based Motorola RAZR i wouldn't run, the AT&T HTC One X (MSM8960) crashed before the final score was calculated so both of those devices were excluded. Thankfully we got the Galaxy S 3 to complete, giving us a good representative from the MSM8960/Adreno 225 camp. Thermal throttling is a concern when running 3DMark. You have to pay close attention to the thermal conditions of the device you're testing. This is becoming something we're having to pay an increasing amount of attention to in our reviews these days.
Graphics Test 1
Ice Storm Graphics test 1 stresses the hardware’s ability to process lots of vertices while keeping the pixel load relatively light. Hardware on this level may have dedicated capacity for separate vertex and pixel processing. Stressing both capacities individually reveals the hardware’s limitations in both aspects.
In an average frame, 530,000 vertices are processed leading to 180,000 triangles rasterized either to the shadow map or to the screen. At the same time, 4.7 million pixels are processed per frame.
Pixel load is kept low by excluding expensive post processing steps, and by not rendering particle effects.
Although the first graphics test is heavy on geometry, it features roughly 1/4 the number of vertices from GL/DXBenchmark 2.7's T-Rex HD test. In terms of vertex/triangle count, even Egypt HD is more stressful than 3DMark's first graphics test. That's not necessarily a bad thing however, as most Android titles are no where near as stressful as what T-Rex and Egypt HD simulate.
Among Android smartphones, Qualcomm rules the roost here. The Adreno 320 based Nexus 4 and HTC One both do very well, approaching 60 fps in the first graphics test. The Mali 400MP4, used in the Galaxy Note 2 and without a lot of vertex processing power, brings up the rear - being outperformed by even NVIDIA's Tegra 3. ARM's Mali-T604 isn't enough to pull ahead in this test either; the Nexus 10 remains squarely behind the top two Adreno 320 based devices.
Graphics Test 2
Graphics test 2 stresses the hardware’s ability to process lots of pixels. It tests the ability to read textures, do per pixel computations and write to render targets.
On average, 12.6 million pixels are processed per frame. The additional pixel processing compared to Graphics test 1 comes from including particles and post processing effects such as bloom, streaks and motion blur.
In each frame, an average 75,000 vertices are processed. This number is considerably lower than in Graphics test 1 because shadows are not drawn and the processed geometry has a lower number of polygons.
As you'd expect, shifting to a more pixel shader heavy workload shows the Galaxy Note 2 doing a lot better - effectively tying the Tegra 3 based HTC One X+ and outperforming the Nexus 7. The Mali-T604 continues to, at best, tie for third place here. Qualcomm's Adreno 320 just seems to deliver better performance in 3DMark for Android.
The overall score pretty much follows the trends we saw earlier. Qualcomm's Adreno 320 leads things (Nexus 4/HTC One), followed by ARM's Mali-T604 (Nexus 10), Adreno 225 (SGS3), Adreno 305 (One SV), Tegra 3 (One X+/Nexus 7) and finally Mali 400MP4 (Note 2). The only real surprise here is just how much better Adreno 320 does compared to Mali-T604.
Physics Test
The purpose of the Physics test is to benchmark the hardware’s ability to do gameplay physics simulations on CPU. The GPU load is kept as low as possible to ensure that only the CPU’s capabilities are stressed.
The test has four simulated worlds. Each world has two soft bodies and two rigid bodies colliding with each other. One thread per available logical CPU core is used to run simulations. All physics are computed on the CPU with soft body vertex data updated to the GPU each frame. The background is drawn as a static image for the least possible GPU load.
The Physics test uses the Bullet Open Source Physics Library.
The physics results give us an indication of just how heavily threaded this benchmark is. The quad-core devices are able to outperform the dual-core Cortex A15 based Nexus 10, despite the latter having far better single threaded performance. The Droid DNA/Optimus G vs. Nexus 4 results continue to be a bit odd, perhaps due to the newer drivers included in the Nexus 4's use of Android 4.2 vs. 4.1.2 for the other APQ8064 platforms.
The overall Ice Storm results don't paint a vastly different picture from what we've seen thus far:
We will likely have to revisit this as non-Nexus devices get updated to Android 4.2, especially given the strange delta between the Nexus 4 and Optimus G/Droid DNA. When the One gets updated to Android 4.2.x, it'll be interesting to see if it pulls ahead of the Nexus 4 in 3DMark.
It's also interesting to note that 3DMark seems to value the Adreno 320 more than ARM's Mali-T604 in the Nexus 10. I tossed results from the HTC One into the T-Rex HD benchmark data from GL/DXBenchmark 2.7 and found that the two benchmarks seem to agree on this point as well:
The magnitude of difference between the One (Adreno 320) and Nexus 10 (Mali-T604) isn't as large, but we are looking at two very different benchmarks here.
3DMark includes an optional demo run that will generate a score as well (although these results aren't included in the overall Ice Storm score):
Just for kicks I ran the extreme (1080p, high quality textures and post processing) presets on a couple of devices to see how the numbers changed:
3DMark for Android - Default vs. Extreme Settings Comparison | ||||||||||||
HTC One | Google Nexus 10 | Google Nexus 7 | ||||||||||
Default | Extreme | Default | Extreme | Default | Extreme | |||||||
Ice Storm Score | 11309 | 6190 | 7990 | 5499 | 3539 | 1895 | ||||||
Graphics Score | 11250 | 5942 | 7952 | 5030 | 3130 | 1581 | ||||||
Physics Score | 11520 | 7246 | 8127 | 8165 | 6527 | 6196 | ||||||
Graphics Test 1 | 49.2 fps | 29.8 fps | 36.9 fps | 27.3 fps | 12.2 fps | 7.5 fps | ||||||
Graphics Test 2 | 48.7 fps | 22.8 fps | 32.5 fps | 18.2 fps | 15.3 fps | 6.4 fps | ||||||
Physics Test | 36.6 fps | 23.0 fps | 25.8 fps | 25.9 fps | 20.7 fps | 19.7 fps | ||||||
Demo | 52.5 fps | 25.7 fps | 35.2 fps | 21.5 fps | 16.8 fps | 7.0 fps |
Moving up to the Extreme presets definitely increases the workload on these devices. It's important to note that today's flagships are close to running 3DMark at extreme presets at 30 fps in some cases. On average, we're looking at low to mid 20 fps for the best of the best on Android today.
3DMark for Android seems to be another good data point to have. While the holy grail remains a good collection of actual games with integrated benchmarks representative of game play, at least having a good selection of simulated game tests is a lot better than where we were just a year ago.
The fact that all of 3DMark's benchmarks are comparable across devices regardless of display resolution is a nice touch, although I do appreciate GLBenchmark's ability to target the native resolution of any device (especially when dealing with very high res tablets). The inclusion of a dedicated physics test is very nice, however I'd like to see equal emphasis placed on single (or lightly) threaded performance as the real world tends to not be as thread happy as 3DMark's physics test implies.
Overall, the 3DMark Ice Storm workload is clearly less stressful than GL/DXBenchmark, which makes me wonder how it'll age over time. On the bright side, 3DMark will make a nice companion to GL/DXBenchmark - offering a glimpse at what GPU performance would look like in workloads closer to what is presently available, rather than forecasting what may be possible in the future. We plan on using both benchmarks in our reviews going forward.
56 Comments
View All Comments
UpSpin - Tuesday, April 2, 2013 - link
Well written article. You tried to explain each anomalitiy in those charts this time, and give those charts much more descriptive text, much better than the T-Rex article.And as you said, it's really odd that the same hardware (Nexus 4 vs. Optimus G) on different Android versions scores so differently on this benchmark, so let's hope that all get an update to 4.2 soon or Futuremark fixes this glitch.
ChronoReverse - Tuesday, April 2, 2013 - link
It could be something to do with the graphics drivers as well as the Android version.jonup - Tuesday, April 2, 2013 - link
Well there has to be something on the software level. N4 and OG have virtually the same hardware and 6 months down the road Quadrant still scores completely different on the two devices. We kept hearing that Quadrant is not optimized for 4.2. how long does it take them to fix the app. I am starting to doubt that hypothesis.p.s. I wonder how high my N4 would score in 3DMark. It is capable of high 23K in Antatu with some extra juice. :banghead I left the phone home. Any speculations?
psychobriggsy - Tuesday, April 2, 2013 - link
I guess here we can see a progression of the capability of ARM SoC GPUs over time too. The Mali-400 MP4 was available over two years ago in the Samsung Galaxy S2 for example. Might be interesting to see a graph of this progression over time to get an idea of how fast mobile graphics are improving.StormyParis - Tuesday, April 2, 2013 - link
I just hope that more gaming performance tests will not distract the reviewers from the fact that games, thus graphics performance, are a very minor concern, as opposed to ... everything else.I'd hate finding myself in the same situation as for PCs, with realms and realms of graphics performance data, at the cost of more relevant stuff (reliability, sound quality, ...)
LetsGo - Tuesday, April 2, 2013 - link
Its going to be very interesting to see how well the IOS machines do in the multithreaded physic tests.Should shut up many a Apple Fanbois.
Diagrafeas - Tuesday, April 2, 2013 - link
As far as i can calculate Krait 300 @ 1.7GHz has the same performance in physics as my 8 year old Athlon 64 X2 4400+ @2.2GHz.It would be interesting to see how it does against NVidia 7800GTX of that era...
I have one sitting around and if i find some time i will try to run a test...
JPForums - Tuesday, April 2, 2013 - link
Do you have some scores you can post?Given the multithreaded nature of this benchmark, they may be able to pull off an upset due to higher core count. I would love to get confirmation on it. The highend 7800/X1900 series graphics cards would be an excellent point of comparison as well given their console brethren. Perhaps the Anandtech crew would be so kind as to run 3DMark on a 2003, 2006, and 2009 era gaming rig as points of comparison to current SoC technology. The venerable Radeon 9700 Pro is just asking for one more round.
I'm admittedly skeptical, though, given the massive difference between my system's score in ice storm vs the SoCs in the article (we're talking an order of magnitude or more). I'm not entirely convinced ARM SoCs have progressed quite as far as people tend to say. I see benchmarks here and there that are supposed to demonstrate the ARM SoCs prowess relative to the x86 PC realm. Then I see benchmarks that leave the two in different ballparks. I tend to fall back on the fact that the 5 year old Atom architecture is competitive with most SoCs out there (CPU performance), yet was relative trash compared to mainstream x86 processors even when it was released. I can see the progress ARM has made and would love to see them put more pressure on Intel outside of the mobile industry, but I think their single threaded performance holds them back at the moment.
theduckofdeath - Tuesday, April 2, 2013 - link
Well, the score comparison between the Razer Edge and the ARM's shows that it is a bit pointless to compare x86 with a GPU with ARM. The margin of error on the Razer Edge is probably somewhere around the total score on an ARM device. :)Diagrafeas - Tuesday, April 2, 2013 - link
I can tell you the physics score of my Athlon 64 X2 4400+ overclocked @2.5GHz.It's 12600-13000 and 40-42 fps.
Currently i have a ATI Radeon 4890. By old card 7800GTX is in a box at the basement...
I was thinking about consoles too.