Nimbus Data’s New ExaDrive NL: 64 TB of Enterprise Grade QLC in 3.5-inch

Today Nimbus Data, one of the first companies to venture into enterprise flash storage in 2003, is announcing its latest generation ExaDrive product. Following on from the success of the 100 TB ExaDrive DC, the company is releasing a new ExaDrive NL series, from 16 TB to 64 TB. This new series uses enterprise grade QLC, compared to TLC in the DC variant, to provide a more cost effective solution with endurance designed to eclipses any other QLC drive on the market.

Enterprise Storage Comes in Layers

For most enterprise storage needs, tiered storage has become the norm. For random access workloads with high bandwidth requirements, NVMe storage or DRAM-like alternatives are preferred, while on the other end where capacity is king, large mechanical hard drives based on SATA or SAS take center stage, with up to ninety 18 TB drives in a single 4U operating as ‘Nearline’ storage as a data retrieval tier or cold storage. There are a number of companies designing controllers, flash, new platter technology, and all between in order to offers something either at the extremes of these storage needs, or to create new caching tiers in the middle. The idea is to trade off latency, bandwidth and price, for capacity as we move up from those bulky spinning rust things.

This leads us to Nimbus Data. The company started offering flash storage solutions for the enterprise as early as 2003, before flash storage was really a thing – at the time the products were small but offered amazing performance over hard drives, and it was several years before other companies started producing similar products for the market. The company has been profitable for years, never relying on any venture capital or seed funding, and has over 200+ customers, including big names over the years such as eBay, Paypal, and Apple. There are also two big CDNs in the mix, which unfortunately they can’t disclose, but these are the sorts of customers that Nimbus Data is courting with its ExaDrive product range.

Two years ago, Nimbus Data showcased the ExaDrive DC, a 100 TB 3.5-inch drive designed for like-for-like swap for the 14-18 TB drives that populated most of the nearline storage options on the market. The problem as always with NAND flash is that cost per TB is higher than mechanical storage, but the DC was aimed to offer much greater density and better power consumption in the same chassis. The ExaDrive DC internally has four separate PCBs, each with a custom flash controller with firmware written by Nimbus Data to manage the status of the flash cells. These are grouped into a backplane inside the 3.5-inch chassis, where an FPGA does overall management and monitoring of the drive. The drive comes with SAS 2.0 and SATA 3.0 6 Gbps interfaces, and the size of the drive combined with the speed of the interface means that Nimbus Data has been offering the drive with an unlimited endurance warranty for five years. Writing sequential data to this drive, at around 400 MBps, would take 70 hours for context.

From LinusTechTips' ExaDrive DC 100TB Teardown

These drives were priced in the region of $40,000 a piece, however the goal was to be able to replace what could be 20U of 540+ mechanical hard drives, around 8 PB of storage, with one 4U with 9 PB. The metrics that Nimbus Data’s customers cared about were centered around density, better latency, endurance (mechanical hard drives are often rated at 550 TB/year, which is actually a limit on the motor, not the platter), power per TB, and ultimately operational expenses over the lifetime of the server (OpEx), including rack space, power, cooling, and maintenance. The drives were expensive for not only being new on the market, but they also used TLC NAND, as it was easy to come by at the time. Some of Nimbus Data’s customers loved the product, but requested if a cheaper version was possible. Enter the ExaDrive NL.

The ExaDrive NL is almost an identical product to the DC variants, except instead of using TLC NAND, this product uses QLC NAND. This allows the amount of NAND to be used to drop, saving some expense.

The main criticism for QLC NAND is that it is rated at a lower endurance than TLC NAND, at users who deploy QLC at an enterprise level at scale might be put off by this property. Nimbus Data tells us that going into this project, they understood why this was a concern for its customers. As a result, they are using some of the best enterprise-grade QLC available on the market, which I’m told is rated for 1500 P/E cycles minimum.

Also to that, Nimbus Data says that competing products in this space that use QLC often come up across controller limitations when growing in size. Modern NAND controllers can only reliably address 4-16 TB of NAND, and so going beyond that requires adjusting the block size which increases write amplification, reducing the effective endurance of the NAND and lifetime of the drive. It also increases the load on the controller, causing extra power consumption. Due to the ExaDrive’s design, using four PCBs of NAND each with its own controller and an FPGA intermediary before the SAS/SATA interface, the ExaDrive NL maintains high endurance and low write amplification as well as a standard 4K block size. The drive also has overprovisioning, around the 8% level as is industry standard.

The new ExaDrive NL versions will be offered in capacities including 16 TB, 32 TB, and 64 TB, with SATA and dual-port SAS interfaces. These drives don’t have an unlimited endurance 5-year warranty like the DC 100 TB model, but are rated up to 75 PB within the 5 years, which is good for 0.6 DWPD. However, writing 60% of the 64 TB drive at SATA speeds will take 1.7 days anyway, indicating that this drive should be good for a long time.

Performance is rated at 500 MB/s read and 460 MB/s write with sequential 128 KB data, and up to 100K read/write IOPs with 4K data. The drives do operate an SLC cache, and Nimbus Data was keen to point out that these drives ‘in typical scenarios’ will never go outside that SLC cache window. Also, the fact that the design is four PCBs of NAND each with its own controller means that write performance is almost identical to read performance, which negates one of the common criticisms with QLC. Latency is in the 50-100 microsecond range. Power consumption is rated at 10W idle and 14W typical for the 64 TB model, and the 3.5-inch enclosure is ventilated to assist this.

One of the comments Nimbus Data made is that in these big 3.5-inch JBOD designs that can hold 90 drives in a 4U, as hard drives spin up and get warm, the drives furthest from the front of the chassis end up with warm air on them, and then the CPU behind that also gets even warmer air. The lower power consumption of NAND helps with that, and the additional ventilation should also keep the drives cooler as required. Also, those vertical drop design chassis are apparently more suited to NAND – Nimbus Data’s CEO in our briefing call stated that in speaking to customers, he learned that vertically mounted HDD reliability is reduced ~5-10%, something which a NAND SSD wouldn’t have a problem with. Also, with regards array rebuild time if these drives were in an array, we were told that due to using NAND that despite the large drive size, rebuilding a 64 TB SSD is quicker than any of the 18 TB HDDs.

The key targets Nimbus Data is hoping to hit with the new ExaDrive NL series is that nearline / CDN type storage, but at a better latency, better density per rack unit, better power per TB, and better OpEx per TB over the lifetime of the hardware. Nimbus Data predicts an 80% reduction in power and cooling costs per TB with its product, as well as potential savings in software licensing (fewer servers needed), reduced admin costs (fewer servers to manage), and a better blending of SSD/HDD as needed with respect to performance and cost. The other metric Nimbus Data is proud of is weight per TB, with an 18 TB HDD weighing ~1.5 lbs, whereas the 64 TB NL drive weighs ~0.87 lbs, an 87% reduction per TB.

In order to assuage concerns about compatibility, Nimbus Data is quoting that the drives have already been tested in a number of high-profile and popular server and storage designs, including PowerEdge Rx40 devices and PERC HBAs, HPE DL300 Gen10 with SmartArray Gen10 storage adaptors, Cisco UCS designs and also Lenovo ThinkSystem and Supermicro builds. A full list is in the slide deck we’ve republished here.

We were told that Nimbus Data currently sells its large 3.5-inch SSD drives mostly into edge deployments, where data density is critical, and these new NL drives will help deliver to customers currently on spinning HDDs that required a better price per TB ratio before jumping to NAND. Pricing will be customer-specific, however the 16 TB model will start through authorized resellers and system manufacturers for under $3000.

Update: Pricing has been revealed as:

• 64 TB: $10900
• 32 TB: $5600
• 16 TB: $2900

Pricing will be updated quarterly as NAND pricing changes.

We’re hoping to get the new ExaDrive NL and perhaps the DC in to test. Stay tuned for that.

Related Reading

• Unlimited 5 Year Endurance: The 100TB SSD from Nimbus Data
• Enterprise NVMe Round-Up 2: SK Hynix, Samsung, DapuStor and DERA
• The Kingston DC500 Series Enterprise SATA SSDs Review: Making a Name In a Commodity Market
• The Enterpise TLC Storage Era Begins: Rounding Up 13 SSDs With Samsung, Intel, and Memblaze
• Memblaze Releases New PBlaze5 520 Series Low-Power Enterprise SSDs
• Western Digital Introduces WD Gold Enterprise SSDs
• Kioxia Launches PM6: First 24G SAS SSD, up to 30.72 TB