With the new 3rd-gen Fire TV Cube being released tomorrow, I decided this was a good time to refresh all the benchmark scores for all competing devices so that everything was on an even playing field, since scores can change over time due to software updates. While no device ever scores exactly the same as before when rerunning benchmarks, especially when years have passed between tests, the two scores are usually close enough to not make much of a difference. As I retested the streaming devices in my arsenal, it became clear that the scores that dropped the most were the ones from devices that updated from Android 9 and 10 to Android 11 and 12, like the Nvidia Shield TV and the Google Chromecast 4K. The performance hit, from software updates alone, was as much as 12% in one case.
It’s pretty common knowledge among enthusiasts that devices, especially those running Android for some reason, tend to slow down over time. This is usually due to new features being added that push the hardware closer to its limits. While that slowdown is easily felt in real-world use, software changing over time tends to not affect benchmark scores much because benchmarks, by design, already push the hardware to its limits from the beginning. Major OS updates, like Android version changes, have the most potential to move a benchmark score up or down due to how much of the OS has changed.
| CPU Benchmark (Geekbench 4: Multi-Core) | GPU Benchmark (GFXBench: Trex Offscren) | |||||
| Old | New | Change | Old | New | Change | |
| 2017 Nvidia Shield TV (2nd Gen) | 4,126 OS 8.2.0 in 2020 (Android 9) | 3,934 OS 9.1.0 in 2022 (Android 11) | -4.65% | 6,600 OS 8.2.0 in 2020 (Android 9) | 6,269 OS 9.1.0 in 2022 (Android 11) | -5.02% |
| 2019 Nvidia Shield TV (3rd Gen) | 3,626 OS 8.2.0 in 2020 (Android 9) | 3,187 OS 9.1.0 in 2022 (Android 11) | -12.11% | 7,096 OS 8.2.0 in 2020 (Android 9) | 7,006 OS 9.1.0 in 2022 (Android 11) | -1.27% |
| 2020 Google Chromecast 4K | 2,420 OS QTS1.200816 in 2020 (Android 10) | 2,401 OS STTE.220621 in 2022 (Android 12) | -0.79% | 826 OS QTS1.200816 in 2020 (Android 10) | 787 OS STTE.220621 in 2022 (Android 12) | -4.72% |
In the table above are my old and new benchmark scores for the streaming devices that dropped the most. In the case of the Nvidia Shield TV models, the OS changed from Android 9 to Android 11 between tests. For the Google Chromecast 4K, the OS changed from Android 10 to Android 12 between tests. The devices themselves are the identical units that were previously tested, so hardware revisions, if they exist, aren’t a factor. For the CPU benchmark using Geekbench 4, the exact same app version was used for both old and new tests, down to the same APK file. For the GPU benchmark using GFXBench, the old test used v5.0.0 of the app and the new test used v5.0.5. This is because GFXBench prevents old versions of the app from running when a new version has been released, however, the app changes seem to be minor and shouldn’t affect the score achieved between app versions.
In both old and new cases, I used the same testing methodology I always use, which is to run each test 3 times and average the top 2 runs for the final score. I also factory reset all devices and make sure the OS and all system apps are fully updated before running my benchmarks. With the hardware, the benchmarking apps, and the testing methods all being the same, the only factor that should be affecting the score is the operating system.
It’s clear to me that increasing the base Android version has its toll on performance. The average performance drop across all six scores listed above is 5%. The biggest CPU performance drop is with the newest Nvidia Shield TV which saw a decrease of 12.11%. The large drop could be due to changes in cooling thresholds, implemented by either the new version of Android or Nvidia itself, since the cylindrical compact shape of the 3rd-gen Shield TV is more difficult to cool than the box shape of other models. That’s just a theory, but I can’t think of any other reason why the score would drop so much for that one device.
| CPU Benchmark (Geekbench 4: Multi-Core) | GPU Benchmark (GFXBench: Trex Offscren) | |||||
| Old | New | Change | Old | New | Change | |
| 2019 Fire TV Cube (2nd Gen) | 4,024 Fire OS 7.2.1.6 in 2020 (Android 9) | 4,180 Fire OS 7.6.0.3 in 2022 (Android 9) | +3.88% | 1,781 Fire OS 7.2.1.6 in 2020 (Android 9) | 1,894 Fire OS 7.6.0.3 in 2022 (Android 9) | +6.34% |
| 2021 Fire TV Stick 4K Max | 2,506 Fire OS 7.2.5.5 in 2021 (Android 9) | 2,481 Fire OS 7.6.0.3 in 2022 (Android 9) | -1.00% | 1,201 Fire OS 7.2.5.5 in 2021 (Android 9) | 1,211 Fire OS 7.6.0.3 in 2022 (Android 9) | +0.83% |
| 2018 Fire TV Stick 4K | 2,012 Fire OS 6.2.7.3 in 2020 (Android 7.1) | 2,019 Fire OS 6.2.9.3 in 2022 (Android 7.1) | +0.35% | 803 Fire OS 6.2.7.3 in 2020 (Android 7.1) | 801 Fire OS 6.2.9.3 in 2022 (Android 7.1) | -0.25% |
For comparison, above are score changes for the top Fire TV models currently being sold. The Fire TV Stick 4K Max and the Fire TV Stick 4K have insignificant changes of 1% or less, which is what you’d expect to see, despite there being one and two years worth of software updates, respectively, between tests. The 2nd-gen Fire TV Cube, on the other hand, has had an unusually high increase in performance.
It appears as though Amazon has put effort into improving the performance of the Fire TV Cube recently while keeping the performance of other models relatively level. No Fire TV model in recent years has been upleveled from one version of Fire OS to another, most notably being the Fire TV Stick 4K which remains the only model in the current lineup still running Fire OS 6, while all other models are running Fire OS 7. You can see the potential performance loss for upleveling the Fire TV Stick 4K in action with the Android TV devices that have moved up in OS versions over the years.
For Android TV devices, like the Shield TV and Chromecast, moving to a newer version of Android is often necessary to gain new features. For those devices, it’s usually worth the performance hit of a new OS to keep the device competitive with newer devices since gaining new features is usually dependent on updating the core OS. On the contrary, with Fire TV devices, new features are independent of the underlying Android version because Fire OS is so heavily modified relative to the version of Android that it is forked from.
For example, Fire TV devices gained Frame Rate Matching in 2018 when Fire OS 6 (based on Android 7.1) was the newest OS available, despite it not existing in any version of Android TV. It wasn’t until a few days ago that the Chromecast 4K (sort of) gained Frame Rate Matching because it was finally included in Android TV 12. Tired of waiting for Google to add it, Nvidia tried to add it itself to the Shield TV in 2019 but the feature is still in beta and not fully supported by many key apps, like Netflix.
With its streaming devices, tablets, and even Echos running on Fire OS, Amazon can devote resources and push development beyond the underlying Android version when needed. Other Android device manufacturers mostly rely on Google to devote resources towards the development of new features, so the version of Android running on the device becomes more important and, therefore, worth the often unavoidable performance hit that comes with a new version. Ultimately, the roughly 5% performance hit on the Shield TV and Chromecast isn’t too noticeable in everyday use and well worth it to gain new features, since there’s otherwise little to no avenue for gaining those features.
@AFTVnews
AFTVnews
Thanks, Elias for writing about this topic.
I’m glad to know that there is an ongoing effort to mitigate the performance decrease hit on Amazon’s behalf.
I, unfortunately, own a Fire TV Cube 1st Gen that I believe won’t be usable in three years, max, because of exactly this.
Looking forward to your in-depth hands-on review of the 3rd Gen Cube.
Keep up the good work.
Thanks
Wait, A whole lot of words when you only compared the base “tube” model, which isn’t exactly very popular, and only has 2GB of RAM, without testing or even mentioning the Pro model, which is far more popular and has 3GB of RAM. What a very strange oversight on what is on the surface a fairly in-depth piece.
I only this week purchased the Shield TV Pro to include in comparisons, as I did for yesterday’s benchmarking article. Since I didn’t own the Shield TV Pro in 2020, I don’t have old benchmark scores to compare to, so I couldn’t include it in this article.
Good luck getting is and security updates on your Amazon devices for another 3 years , nvidia on the other hand has a long track record of updating the sheildTV. Some units going all the way back to 2015 and while the the Os may be causing the unit to finally show some age ….it has taken almost a decade for other streamers (fire, cast, etc.)to catch up to the sheild in terms of raw performance.
I think also that Fire TV OS is better optimized than Android TV. I have a HD TV with Android TV 9 and 2 GB of RAM and two Fire TV Stick (3° generation) with Fire TV OS 7 and 1 GB of RAM. Also the Android one have better CPU and GPU than Fire TV Stick. Somehow the Fire Sticks are much much faster than the Android one. The exact same apps, the same resolution. The only explanation I have, is that Amazon have deeply optimized Android TV in Fire TV OS.
I agree. I switched to the shield TV years ago because I hated the forced ads on the home screen, but I bought a fire TV max when it was on sale just to play with and use it on trips. I can’t help but think that it is snapper, smoother and Alexa is much more usable then android assistant.
If the Android TV you’re talking about is built in to your TV, it makes sense that your Fire Stick is better as it’s a device singularly designed to send a video signal to your TV. Your TV needs to decode and figure what to do with that signal, sometimes it needs to upscale/downscale or do other post processing to the video it’s receiving. This depends on the specific model of TV, but if you’re using the built-in Android TV you’re also sharing your resources with your TV’s built-in post processing. So while the specs might look the same on paper they actually aren’t.
Yes, it’s a built in Android TV. But it’s the OS of the TV, I don’t think it needs to process too much. There’s nothing connected to it. But also it has a better CPU and GPU, and twice RAM than Fire Stick. My theory is that Google Play Services and other background Google apps and services sucks.
This is what I like to see. Under the hood differences of the nuanced features of android based streaming devices. Could you please give more of these feature differences? Like are the radios different and which are best and well optimized?
My Google TV/Chromecast was slow already, now its even worse. Apps randomly close too. Wish i never upgraded. I’m really waiting for a new, affordable device. Been using the Walmart Onn Adroid TV box as my main driver since its cleaner than Google’s and has a better remote. Is there any Google hardware that doesnt have performance issues? So dissapointing such a giant company contiues to be outdone by its competitors accross the board. Imagine if Samsung wasnt there to push Google to be better.
Yet another in the long list of reasons updating the Shield TV to the newer version of Android has been a debacle. People have been having issues galore since the update, and a performance hit on top of that is just crappy icing on the crap cake.
I’m one of those having problems with my Shield. I called Nvidia and the tech acted like this was the first time he had ever heard of these problems.
Chances are he was telling the truth. I’ve been dealing with the Shield TV for years, and have had no problems. This is most likely the case for the majority of owners.
Not only these devices, I noted a decrease in performance in my smartphone as well…
You compared only the tube and not the flagship Nvadia Shield pro.
I qould never use any other device than a Nvadia Shield Pro.
My experience in actually using a Chromecast 4k with Google TV is improved. I usually didn’t bother watching Fox YouTubeTV sports presentations in 4k because the frame rate had odd jitters and weird pauses in the UI. After updating to 12, it now works just as well as a 1080 stream. I don’t care about what synthetic benchmarks do, instead I care about how it streams content.
Crap is what I have now. 2 new Google tv 4K we’re fine, not now. Both are slow running on Netgear MR60 Nighthawk. oh well, I guess I’ll wait for 13 and hope for the best.
It’s Google so you should expect the worst and not update any further
at least until it has been out a few months
I have 2 pixel 3a XL’s one on 11 and one on 19 ,the one 10 is better the one on 1 has issues with Android auto and other things as well
I will never update to 12 unless it forces me, and it has already downloaded and is waiting to install on the one on 11
all Google does is break things and remove useful features (such as smart home control from power menu from 11 to 12)
This is super interesting.. I use my Nvidia shield pros specifically to watch TV and I have noticed a decrease in performance since I bought them. To the point where I actually have reset them a few times. This is probably done by design to get us to buy new hardware in the future which is bs.
I have the Nvidia Shield TV 32 bit (2019) device. I have so many problems with this device now. It slows down to a crawl every 1 to 2 days to the point where I need to reboot it often. It does not feel as snappy transitioning between apps. I think there must be a serious memory leak or something effecting the this device – since the performance degrades over time between reboots. Needless to say – I have a much better experience with the Roku Ultra(s) in my other TVs. I am hoping Nvidia will release a software patch for this soon – or update the hardware to use a more powerful CPU. I like the Android platform so much better for the Android apps – but this device has turned into a problem now and becoming unpleasant to use.
For a valid test, you need to have done the test right before the update. This is especially true for any app that might load a resource from storage. Solid state storage devices use a garbage collection mechanism that causes the speed to change dramatically depending on a number of factors.
Also, after the update, you have to let the system settle awhile. Android doesn’t use native machine code for its APKs. This means it has to translate, using combinations of JIT and AOT optimizations. A major update can mean things are slower for a bit until all the optimizations complete.
Using a benchmark available on play store would have been better as you could get input from more people as to what an average result actually is. A single benchmark from 1 person is not conclusive.
Also, Android always has a governor and these can be tweaked from version to version. The benchmark app has no control over how the governor scales up and down. Benchmarking on a non-rooted device, between different android versions (which means the kernel and governor can be wildly different) without the ability to take the governor out of the equation is pointless!
Really, this post just looks like someone trying to get attention and I’m sorry I did so. The main thing these devices do is play back encoded video and that is all in hardware.
I’m still using the original Nvidia shield TV from 2015 and I have no issues with it at all.
I have to the shield pro 2019 and have issues with performance, it is no longer the same device I bought. Lag in apps, sound desync, stuttering, the latest update is awful.
dang I have 3 shields total, 1 tube, 2 pros.. have only used the tube the pros are still boxed from maybe a year or 2 ago.. so it would be best to not even bother to update the pros if and when I use them? because the tube has been so so in performance since the updates
amateurs.
the biggest problem is not the CPU, but the slow transfer rate of the local & external storage which make the shield run so slow and burp even when turning the volume up/down when watching a 4K video, like on all those Chinese boxes. it never happened on my Shield pro, even when running those 100GB 4K Remux.
I wonder why they don’t give us an easy rollback option.