NVM Express


NVM Express or Non-Volatile Memory Host Controller Interface Specification is an open logical device interface specification for accessing non-volatile storage media attached via PCI Express bus. The acronym NVM stands for non-volatile memory, which is often NAND flash memory that comes in several physical form factors, including solid-state drives, PCI Express add-in cards, M.2 cards, and other forms. NVM Express, as a logical device interface, has been designed to capitalize on the low latency and internal parallelism of solid-state storage devices.
By its design, NVM Express allows host hardware and software to fully exploit the levels of parallelism possible in modern SSDs. As a result, NVM Express reduces I/O overhead and brings various performance improvements relative to previous logical-device interfaces, including multiple long command queues, and reduced latency. where a very lengthy delay
NVM Express devices are chiefly available in the form of standard-sized PCI Express expansion cards and as 2.5-inch form-factor devices that provide a four-lane PCI Express interface through the U.2 connector. Storage devices using SATA Express and the M.2 specification which support NVM Express as the logical device interface are a popular use-case for NVMe and have become the dominant form of solid-state storage for servers, desktops and laptops alike.

Specifications

Specifications for NVMe released to date include:
Historically, most SSDs used buses such as SATA, SAS or Fibre Channel for interfacing with the rest of a computer system. Since SSDs became available in mass markets, SATA has become the most typical way for connecting SSDs in personal computers; however, SATA was designed primarily for interfacing with mechanical hard disk drives, and it became increasingly inadequate for SSDs, which improved in speed over time. For example, within about 5 years of mass market mainstream adoption many SSDs were already held back by the comparatively slow data rates available for hard drives—unlike hard disk drives, some SSDs are limited by the maximum throughput of SATA.
High-end SSDs had been made using the PCI Express bus before NVMe, but using non-standard specification interfaces. By standardizing the interface of SSDs, operating systems only need one driver to work with all SSDs adhering to the specification. It also means that each SSD manufacturer does not have to use additional resources to design specific interface drivers. This is similar to how USB mass storage devices are built to follow the USB mass-storage device class specification and work with all computers, with no per-device drivers needed.
NVM Express devices are also used as the building block of the burst buffer storage in many leading supercomputers, such as Fugaku Supercomputer, Summit Supercomputer and Sierra Supercomputer, etc.

History

The first details of a new standard for accessing non-volatile memory emerged at the Intel Developer Forum 2007, when NVMHCI was shown as the host-side protocol of a proposed architectural design that had Open NAND Flash Interface Working Group on the memory chips side. A NVMHCI working group led by Intel was formed that year. The NVMHCI 1.0 specification was completed in April 2008 and released on Intel's web site.
Technical work on NVMe began in the second half of 2009. The NVMe specifications were developed by the NVM Express Workgroup, which consists of more than 90 companies; Amber Huffman of Intel was the working group's chair. Version 1.0 of the specification was released on, while version 1.1 of the specification was released on. Major features added in version 1.1 are multi-path I/O and arbitrary-length scatter-gather I/O. It is expected that future revisions will significantly enhance namespace management. Because of its feature focus, NVMe 1.1 was initially called "Enterprise NVMHCI". An update for the base NVMe specification, called version 1.0e, was released in January 2013. In June 2011, a Promoter Group led by seven companies was formed.
The first commercially available NVMe chipsets were released by Integrated Device Technology in August 2012. The first NVMe drive, Samsung's XS1715 enterprise drive, was announced in July 2013; according to Samsung, this drive supported 3 GB/s read speeds, six times faster than their previous enterprise offerings. The LSI SandForce SF3700 controller family, released in November 2013, also supports NVMe. A Kingston HyperX "prosumer" product using this controller was showcased at the Consumer Electronics Show 2014 and promised similar performance. In June 2014, Intel announced their first NVM Express products, the Intel SSD data center family that interfaces with the host through PCI Express bus, which includes the DC P3700 series, the DC P3600 series, and the DC P3500 series., NVMe drives are commercially available.
In March 2014, the group incorporated to become NVM Express, Inc., which as of 2014 consists of more than 65 companies from across the industry. NVM Express specifications are owned and maintained by NVM Express, Inc., which also promotes industry awareness of NVM Express as an industry-wide standard. NVM Express, Inc. is directed by a thirteen-member board of directors selected from the Promoter Group, which includes Cisco, Dell, EMC, HGST, Intel, Micron, Microsoft, NetApp, Oracle, PMC, Samsung, SanDisk and Seagate.
In September 2016, the CompactFlash Association announced that it would be releasing a new memory card specification, CFexpress, which uses NVMe.

Form factors

There are many form factors of NVMe solid-state drive, such as AIC, U.2, U.3, M.2 etc.

AIC (add-in card)

Almost all early NVMe solid-state drives are HHHL or FHHL AIC, with a PCIe 2.0 or 3.0 interface. A HHHL AIC NVMe solid-state drive is easy to adapt to insert in to the PCIe socket of a server.

U.2 (SFF-8639)

, formerly known as SFF-8639, is a computer interface for connecting solid-state drives to a computer. It uses up to four PCI Express lanes. Available servers can combine up to 4 U.2 NVMe solid-state drives.

U.3 (SFF-8639)

is built on the U.2 spec and uses the same SFF-8639 connector. It is a 'tri-mode' standard, combining SAS, SATA and NVMe support into a single controller. U.3 can also support hot-swap between the different drives where firmware support is available. U.3 drives are still backward compatible with U.2, but U.2 drives are not compatible with U.3 hosts.

M.2

, formerly known as the Next Generation Form Factor, uses a M.2 NVMe solid-state drive computer bus. Interfaces provided through the M.2 connector are PCI Express 3.0 or PCI Express 4.0.

NVMe-oF

NVM Express over Fabrics is the concept of using a transport protocol over a network to connect remote devices, contrary to NVMe where devices are connected directly to PCIe bus In September 2014, a standard for using NVMe over Fibre Channel was proposed and this combination is often referred to as FC-NVMe. NVMe-oF is a communication protocol that allows one computer to access block-level storage devices attached to another computer via remote direct memory access over a number of transport protocols:
The standard for NVMe over Fabrics was published by NVM Express, Inc. in 2016.
The following drivers implement the NVMe-oF protocol:
Starting with Linux Kernel 5.0 native support has been added for NVMe/TCP.

Comparison with AHCI

The Advanced Host Controller Interface has the benefit of wide software compatibility, but has the downside of not delivering optimal performance when used with SSDs connected via the PCI Express bus. As a logical interface, AHCI was developed when the purpose of a host bus adapter in a system was to connect the CPU/memory subsystem with a much slower storage subsystem based on rotating magnetic media. As a result, AHCI introduces certain inefficiencies when used with SSD devices, which behave much more like DRAM than like spinning media.
The NVMe device interface has been designed from the ground up, capitalizing on the low latency and parallelism of PCI Express SSDs, and complementing the parallelism of contemporary CPUs, platforms and applications. At a high level, the basic advantages of NVMe over AHCI relate to its ability to exploit parallelism in host hardware and software, manifested by the differences in command queue depths, efficiency of interrupt processing, the number of uncacheable register accesses, etc., resulting in various performance improvements.
The table below summarizes high-level differences between the NVMe and AHCI logical device interfaces.
AHCINVMe
Maximum queue depthOne command queue;
32 commands per queue		65535 queues;
65536 commands per queue
Uncacheable register accesses
Six per non-queued command;
nine per queued command		Two per command
MSI-X
and interrupt steering		A single interrupt;
no steering		2048 MSI-X interrupts
Parallelism
and multiple threads		Requires synchronization lock
to issue a command		No locking
Efficiency
for 4 KB commands		Command parameters require
two serialized host DRAM fetches		Gets command parameters
in one 64-byte fetch

Operating system support

; Chrome OS
; DragonFly BSD
; FreeBSD
; Genode
; Haiku
; illumos
; iOS
; Linux
; NetBSD
; OpenBSD
; OS X/macOS
;Solaris
; VMware
; Windows

Software support

; QEMU
; UEFI

Management tools

nvmecontrol

The nvmecontrol tool is used to control an NVMe disk from the command line on FreeBSD. It was added in FreeBSD 9.2.

nvme-cli

NVM-Express user space tooling for Linux.