The Enterpise TLC Storage Era Begins: Rounding Up 13 SSDs With Samsung, Intel, and Memblaze
by Billy Tallis on January 3, 2019 9:45 AM EST2018 was a busy year for consumer SSDs. With all the NAND flash manufacturers now shipping high-quality 3D NAND in volume, we've seen more competition than ever, and huge price drops. NVMe is starting to go mainstream and Samsung is no longer sitting atop that market segment unchallenged. But not all of the interesting SSD advancements have been in the consumer realm. We've reported on new datacenter SSD form factors, the introduction of QLC NAND and enterprise SSDs with staggering capacities, but we haven't been publishing much in the way of traditional product reviews for enterprise/datacenter SSDs.
Tackling the Enterprise SSD Market
Today we're looking at several recent models that cover the wide range of enterprise SSDs, from entry-level SATA drives based on the same hardware as mainstream consumer SSDs, up to giants that deliver 1M IOPS over a PCIe x8 connection while pulling more than 20W.
No two models in this review are direct competitors against each other, but together these drives represent almost every important product segment for enterprise/datacenter SSDs. The selection of drives reflects recent trends: the fact that MLC NAND is basically gone from the market, and now replaced by mature 3D TLC NAND. The state of 3D TLC is that it now offers plenty of endurance and performance when aggregated into the multi-TB drives that are most cost-effective.
For the few niches that still require the highest endurance and lowest latency that money can buy, alternative memories like Intel 3D XPoint and Samsung Z-NAND are filling the gap left by the disappearance of traditional MLC and SLC NAND flash. SAS SSDs still exist, but their relevance is waning now that NVMe SSDs have matured to the point of offering not just higher performance, but all the same reliability and management features that are standard in the SAS ecosystem.
In the consumer SSD market, there's not a big price between the cheapest NVMe SSDs and the fastest (NAND-based) NVMe SSDs, so it's difficult to divide the consumer NVMe market into multiple product segments that each contain reasonable products. The enterprise SSD market doesn't have this problem. There's a clear distinction between the lower-power NVMe drives that usually share a controller platform with a consumer product, and the high-end drives that use enterprise-only 16 to 32-channel controllers.
Probing Samsung, Intel, and Memblaze
The table below lists the drives that are the subject of this review. Three of the four new models Samsung announced during Q3 2018 are included: two SATA and one NVME, and a fourth (983 ZET Z-SSD) will be reviewed later. The Intel DC P4510 mid-range NVMe SSD and Optane DC P4800X have already been reviewed on AnandTech, but are being revisited now that we have more relevant drives to compare against. The Memblaze PBlaze5 is the most powerful flash-based SSD we have tested, and is a good representative of the top-tier enterprise SSDs that don't get sampled for review very often.
Not shown in the table but included in the review is the older Samsung PM863 SATA drive, which helps illustrate how things have progressed since the days when Samsung had a monopoly on 3D NAND. We do have several other enterprise SSDs on hand, but they're all either in service as boot drives for various testbeds, or are old planar NAND drives, so they're not included in this review.
Reviewed Models Overview | |||||||
Model | SATA/PCIe | Form Factor | Capacities | Memory | Write Endurance (DWPD) |
||
Samsung 860 DCT | SATA | 2.5" 7mm | 960 GB 1920 GB 3840 GB |
64L 3D TLC | 0.2 DWPD | ||
Samsung 883 DCT | SATA | 2.5" 7mm | 240 GB 480 GB 960 GB 1.92 TB 3.84 TB |
64L 3D TLC | 0.8 DWPD | ||
Samsung 983 DCT | PCIe 3.0 x4 | 2.5" 7mm U.2 | 960 GB 1.92 TB |
64L 3D TLC | 0.8 DWPD | ||
PCIe 3.0 x4 | M.2 22110 | 960 GB 1.92 TB |
64L 3D TLC | 0.8 DWPD | |||
Intel DC P4510 | PCIe 3.0 x4 | 2.5" 15mm U.2 | 1 TB 2 TB 4 TB 8 TB |
64L 3D TLC | 0.7–1.1 DWPD | ||
Intel Optane DC P4800X |
PCIe 3.0 x4 | HHHL AIC | 375 GB 750 GB 1.5 TB |
3D XPoint | 60 DWPD | ||
Memblaze PBlaze5 D900 |
PCIe 3.0 x4 | 2.5" 15mm U.2 | 2 TB 3.2 TB 4 TB 8 TB |
32L 3D TLC | 3 DWPD | ||
Memblaze PBlaze5 C900 |
PCIe 3.0 x8 | HHHL AIC | 2 TB 3.2 TB 4 TB 8 TB |
32L 3D TLC | 3 DWPD | ||
Note: Tested capacities are shown in bold |
To determine where an enterprise SSD fits into the competitive landscape, the first specifications to look at are the form factor, interface, and write endurance rating.
- Interface: As with consumer SSDs, in the enterprise storage market SATA drives are generally cheaper and much slower than NVMe drives. But in a server, the performance differences between SATA and NVMe can have a much larger impact on overall system performance than with an interactive desktop workload. SATA product lines usually top out at 4TB or 8TB; beyond that, the SATA interface bottleneck leads to unacceptably low performance-per-TB ratios. NVMe product lines usually start around 1TB and go up from there, because smaller capacities would not be able to make good use of PCIe bandwidth.
- Form Factor: Smaller form factors like M.2 and 7mm thick 2.5" drives cannot draw as much power as a PCIe add-in card, and this often leads to the physically smaller drives offering lower performance. The largest 16 and 32-channel SSD controllers cannot even fit on a 22mm wide M.2 card. This is why newer standards are coming into the market, like Samsung's NF1 and Intel's Ruler.
- Write Endurance: Endurance ratings indicate what kind of workload a drive is intended for. Drives with higher endurance ratings also have higher write performance, especially random write performance. Ratings of less than one drive write per day usually indicate a drive that is intended for a relatively read-heavy workload. As little as 0.1 or 0.2 DWPD can be plenty for a use case like a streaming video CDN node, which experiences little write volume and what writes it does have to contend with are mostly sequential. Drives with ratings of 5 DWPD or more are becoming rare, except for specialty models like Intel's Optane SSDs and Samsung's Z-SSDs.
Based on these guidelines, we can expect the Intel SSD DC P4510 to be a small step up from the Samsung 983 DCT, in price and power consumption if not performance. The Memblaze PBlaze5 D900 is another step up with its higher write endurance rating, and the PBlaze5 C900 is clearly the beefiest NAND-based drive in this bunch with its PCIe 3 x8 interface.
Test System
Intel provided our enterprise SSD test system, one of their 2U servers based on the Xeon Scalable platform (codenamed Purley). The system includes two Xeon Gold 6154 18-core Skylake-SP processors, and 16GB DDR4-2666 DIMMs on all twelve memory channels for a total of 192GB of DRAM. Each of the two processors provides 48 PCI Express lanes plus a four-lane DMI link. The allocation of these lanes is complicated. Most of the PCIe lanes from CPU1 are dedicated to specific purposes: the x4 DMI plus another x16 link go to the C624 chipset, and there's an x8 link to a connector for an optional SAS controller. This leaves CPU2 providing the PCIe lanes for most of the expansion slots, including most of the U.2 ports.
Enterprise SSD Test System | |
System Model | Intel Server R2208WFTZS |
CPU | 2x Intel Xeon Gold 6154 (18C, 3.0GHz) |
Motherboard | Intel S2600WFT |
Chipset | Intel C624 |
Memory | 192GB total, Micron DDR4-2666 16GB modules |
Software | Linux kernel 4.19.8 fio version 3.12 |
Thanks to StarTech for providing a RK2236BKF 22U rack cabinet. |
The enterprise SSD test system and most of our consumer SSD test equipment are housed in a StarTech RK2236BKF 22U fully-enclosed rack cabinet. During testing for this review, the front door on this rack was generally left open to allow better airflow, since the rack doesn't include exhaust fans of its own. The rack is currently installed in an unheated attic and it's the middle of winter, so this setup provided a reasonable approximation of a well-cooled datacenter.
The test system is running a Linux kernel from the most recent long-term support branch. This brings in the latest Meltdown/Spectre mitigations, though strategies for dealing with Spectre-style attacks are still evolving. The benchmarks in this review are all synthetic benchmarks, with most of the IO workloads generated using FIO. Server workloads are too widely varied for it to be practical to implement a comprehensive suite of application-level benchmarks, so we instead try to analyze performance on a broad variety of IO patterns.
Enterprise SSDs are specified for steady-state performance and don't include features like SLC caching, so the duration of benchmark runs doesn't have much effect on the score, so long as the drive was thoroughly preconditioned. Except where otherwise specified, for our tests that include random writes, the drives were prepared with at least two full drive writes of 4kB random writes. For all the other tests, the drives were prepared with at least two full sequential write passes.
Our drive power measurements are conducted with a Quarch XLC Programmable Power Module. This device supplies power to drives and logs both current and voltage simultaneously. With a 250kHz sample rate and precision down to a few mV and mA, it provides a very high resolution view into drive power consumption. For most of our automated benchmarks, we are only interested in averages over time spans on the order of at least a minute, so we configure the power module to average together its measurements and only provide about eight samples per second, but internally it is still measuring at 4µs intervals so it doesn't miss out on short-term power spikes.
Quarch provides a variety of power injection adapters for use with different drive form factors and connectors. For this review, we used two different adapters: one for U.2/SAS/SATA drives that delivers power over a thin ribbon cable that can fit between drives in a hot-swap bay, and a PCIe add-in card riser. For the Samsung 983 DCT M.2, we used a PCIe to M.2 adapter board that includes a 12V to 3.3V converter, and our measurements have been adjusted to account for the overhead of that converter. This is necessary because the 983 DCT M.2 is rated to draw a peak of 3.8A, and it is very difficult to deliver that much current at only 3.3V without serious voltage drop along the cable. Newer Quarch power modules include upgrades to improve high-current delivery and add automatic compensation for voltage drop along the cable, but we haven't had the chance to upgrade yet.
36 Comments
View All Comments
ZeDestructor - Friday, January 4, 2019 - link
Could you do the MemBlaze drives too? I'm really curious how those behave under consumer workloads.mode_13h - Thursday, January 3, 2019 - link
At 13 ms, the Peak 4k Random Read (Latency) chart is likely showing the overhead of a pair of context switches for 3 of those drives. I'd be surprised if that result were reproducible.Billy Tallis - Thursday, January 3, 2019 - link
Those tail latencies are the result of far more than just a pair of context switches. The problem with those three drives is that they need really high queue depths to reach full throughput. Since that test used many threads each issuing one IO at a time, tail latencies get much worse once the number of threads outnumbers the number of (virtual) cores. The 64-thread latencies are reasonable, but the 99.9th and higher percentiles are many times worse for the 96+ thread iterations of the test. (The machine has 72 virtual cores.)The only way to max out those drive's throughput while avoiding the thrashing of too many threads is to re-write an application to use fewer threads that are issuing IO requests in batches with asynchronous APIs. That's not always an easy change to make in the real world, and for benchmarking purposes it's an extra variable that I didn't really want to dig into for this review (especially given how it complicates measuring latency).
I'm comfortable with some of the results being less than ideal as a reflection of how the CPU can sometimes bottleneck the fastest SSDs. Optimizing the benchmarks to reduce CPU usage doesn't necessarily make them more realistic.
CheapSushi - Friday, January 4, 2019 - link
Hey Billy. this is a bit of a tangent but do you think SSHDs will have any kind of resurgence? There hasn't been a refresh at all. The 2.5" SSHDs max out at about 2TB I believe with 8GB of MLC(?) NAND. Now that QLC is being pushed out and with fairly good SLC schemes, do you think SSHDs could still fill a gap in price + capacity + performance? Say, at least a modest bump to 6TB of platter with 128GB of QLC/SLC-turbo NAND? Or some kind of increase along those lines? I know most folks don't care about them anymore. But there's still something appealing to me about the combination.leexgx - Friday, January 4, 2019 - link
Sshd tend to use MLC, Only ones been interesting has been the Toshiba second gen sshds as they use some of the 8gb for write caching (from some Basic tests I have seen)where as seagate only caches commonly read locations
leexgx - Friday, January 4, 2019 - link
Very annoying the page reloadingWant to test second gen Toshiba but finding the right part number as they are using creptic part numbers
CheapSushi - Friday, January 4, 2019 - link
Ah, I was not aware of the ones from Toshiba, thanks for the heads up. Write caching seems the way to go for such a setup. Did the WD SSHD's do the same as Seagates?leexgx - Friday, January 11, 2019 - link
I have obtained the Toshiba mq01, mq02 and there h200 sshd all 500gb to test to see if write caching works (limit testing to 500mb writing at start see how it goes from Therethiagotech - Friday, January 4, 2019 - link
Can someone help me understanding which scenarios is considered as QD1 and higher? Does anyone have a guide for dummies what is queue depth? Lets suppose i'll start Windows and there is 200 files of 4k, is it a QD1 or QD64? Because i was copying a folder with a large number of tiny files and my Samsung 960 Pro reached like 70MBPS of copy speed, is really bad number...Greg100 - Saturday, January 5, 2019 - link
thiagotech,About queue depth during boot up a Windows check last post: https://forums.anandtech.com/threads/qd-1-workload...
About optimization Samsung 960 Pro performance check: "The SSD Reviewers Guide to SSD Optimization 2018" on thessdreview