Everything is a key-value!
Ok, so I could store a table-like data in a key-value?
Yes, this is exactly the idea. It may seem weird at first, but it is a common pattern among new databases. In this section, we will briefly go over some examples.
Yugabyte
From the official documentation:
YugabyteDB architecture follows a layered design. It is comprised of 2 logical layers as shown in the diagram below:
Yugabyte Query Layer
DocDB distributed document store
Information about key encoding format can be found here and here.
F1 and Spanner from Google
There is a lot of contents about Google's datastores and their evolution.
At first, they built Megastore on top of BigTable:
In brief, it provides fully serializable ACID semantics over distant replicas with low enough latencies to support interactive applications.
We use Google’s Bigtable for scalable fault-tolerant storage within a single datacenter, allowing us to support arbitrary read and write throughput by spreading operations across multiple rows.
Then, the first version of Spanner appeared:
Spanner is a scalable, globally-distributed database designed, built, and deployed at Google.
Spanner has evolved from a Bigtable-like versioned key-value store into a temporal multi-version database. Data is stored in schematized semi-relational tables; data is versioned, and each version is automatically timestamped with its commit time
Then they added F1 on top of Spanner:
F1 is a fault-tolerant globally-distributed OLTP and OLAP database built at Google as the new storage system for Google’s AdWords system. It was designed to replace a sharded MySQL implementation that was not able to meet our growing scalability and reliability requirements.
F1 is built on top of Spanner
And finally, Spanner itself became an SQL system:
Google’s Spanner started out as a key-value store offering multi-row transactions, external consistency, and transparent failover across datacenters. Over the past 7 years it has evolved into a relational database system
It seems it is still organized as layers:
Like the lower storage and transactional layers,
TiDB / TiKV
In the titanium stack, TiKV is the storage layer for TiDB:
It has three core components: a distributed storage layer, a Placement Driver(PD), and a computation engine layer.
The distributed storage layer consists of a row store (TiKV) and a columnar store (TiFlash). Logically, the data stored in TiKV is an ordered key-value map. Each tuple is mapped into a key-value pair. The key is composed of its table ID and row ID, and the value is the actual row data, where the table ID and row ID are unique integers
The table codec can be found here.
CosmosDB
More details can be found here:
Cosmos DB supports and projects multiple data models (documents, graphs, key-value, table, etc.) over a minimalist type system and core data model: the atom-record-sequence (ARS) model.
Container and item resources are further projected as reified resource types for a specific type of API interface. For example, while using document-oriented APIs, container and item resources are projected as collection and document resources respectively.
The automatic indexing strategy is described here.
CockroachDB
This old blogpost describes the idea:
The CockroachDB SQL system is built on top of the internal CockroachDB key-value store and leverages the monolithic sorted key-value map to store all of the SQL table data and indexes.
The encoding notes can be found here.
Dgraph
From the official documentation:
Dgraph is a single layer in your tech stack, but inside the inner workings of a Dgraph database instance, there are three distinct entities:
- Badger - Dgraph’s custom-built key-value store
- Ristretto - Dgraph’s custom-built cache
- Dgraph - the methods and algorithms used to parse DQL (and now GraphQL) and act accordingly
Examples from layers
Record-Layer
The Record-layer is using Protobuf to store data. A single Protobuf message is written across multiple rows. It can be encrypted and compressed.
Document-layer
The Document-layer is writing a document across multiple keys