Apache Druid


Druid is a column-oriented, open-source, distributed data store written in Java. Druid is designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data. The name Druid comes from the shapeshifting Druid class in many role-playing games, to reflect the fact that the architecture of the system can shift to solve different types of data problems.
Druid is commonly used in business intelligence/OLAP applications to analyze high volumes of real-time and historical data. Druid is used in production by technology companies such as Alibaba, Airbnb, Cisco,, , eBay, Lyft, Netflix, PayPal, Pinterest, Twitter, Walmart,Wikimedia Foundation and Yahoo.

History

Druid was started in 2011 to power the analytics product of Metamarkets. The project was open-sourced under the GPL license in October 2012, and moved to an Apache License in February 2015.
Over time, a number of organizations and companies have integrated Druid into their backend technology, and committers have been added from numerous different organizations.

Architecture

Fully deployed, Druid runs as a cluster of specialized processes to support a fault-tolerant architecture where data is stored redundantly, and there is no single point of failure. The cluster includes external dependencies for coordination, metadata storage, and a deep storage facility for permanent data backup.

Query management

Client queries first hit broker nodes, which forward them to the appropriate data nodes. Since Druid segments may be partitioned, an incoming query can require data from multiple segments and partitions stored on different nodes in the cluster. Brokers are able to learn which nodes have the required data, and also merge partial results before returning the aggregated result.

Cluster management

Operations relating to data management in historical nodes are overseen by coordinator nodes. Apache ZooKeeper is used to register all nodes, manage certain aspects of internode communications, and provide for leader elections.

Features