Databases
How data is stored, queried, and scaled. From decades-old relational engines to the vector stores now powering AI applications. Click any topic to explore how it works, where it fits, and the systems that define it.
Database Landscape
The scale of modern data infrastructure
SQL vs NoSQL vs NewSQL
Three paradigms, each optimized for different trade-offs
Structured, consistent, proven
ACID transactions, schema enforcement, powerful joins, decades of optimization. The default for transactional workloads and any application that values data integrity.
Flexible, scalable, fast
Schema-free documents, horizontal scaling, tunable consistency. Best for rapidly evolving data models, caching, and workloads that prioritize availability over strict consistency.
Scale + consistency
The best of both worlds — SQL interface and ACID guarantees with horizontal scalability across nodes. Higher operational complexity but eliminates the SQL-vs-NoSQL trade-off.
Semantic search at scale
Purpose-built for embedding similarity search, powering RAG pipelines and recommendation engines. Either standalone or as extensions to existing databases (pgvector).
Popularity Index
Relative adoption based on community surveys, job postings, and usage data
Deep Dives
Click any topic to expand. Each includes how it works, key technologies, real-world examples, and challenges
Relational Databases
Data organized into tables with rows and columns, related via foreign keys and queried with SQL. ACID transactions guarantee consistency.
Mainstream
Relational Databases
Data organized into tables with rows and columns, related via foreign keys and queried with SQL. ACID transactions guarantee consistency.
How It Works
Relational databases store data in normalized tables connected through foreign key relationships. SQL (Structured Query Language) provides a declarative interface for querying, joining, and aggregating data. ACID properties (Atomicity, Consistency, Isolation, Durability) ensure that transactions either complete fully or not at all, maintaining data integrity even during crashes or concurrent access. Query optimizers use cost-based planning to choose efficient execution strategies.
Key Technologies
- PostgreSQL (most advanced open-source RDBMS)
- MySQL (most deployed, powers much of the web)
- SQLite (embedded, zero-configuration)
- SQL Server (Microsoft enterprise)
- Oracle Database (enterprise legacy)
Real-World Examples
PostgreSQL is ranked #1 on DB-Engines as of 2024. MySQL powers WordPress, which runs about 43% of all websites according to W3Techs. SQLite is the most widely deployed database engine in the world, embedded in every smartphone, browser, and countless applications.
Challenges & Considerations
Horizontal scaling requires sharding or distributed extensions (Citus, Vitess). Schema changes on large tables can cause downtime without careful migration strategies. Connection management becomes a bottleneck at scale without pooling solutions like PgBouncer.
Vector Databases
Specialized stores optimized for approximate nearest-neighbor search over high-dimensional embeddings. They power semantic search and RAG pipelines.
Mainstream
Vector Databases
Specialized stores optimized for approximate nearest-neighbor search over high-dimensional embeddings. They power semantic search and RAG pipelines.
How It Works
Vector databases index dense vector embeddings (typically 768 to 3072 dimensions) using specialized data structures like HNSW (Hierarchical Navigable Small World) graphs or IVF (Inverted File) indexes. At query time, they find the k most similar vectors to a query vector using distance metrics like cosine similarity or dot product. This enables semantic search where results are ranked by meaning rather than keyword match.
Key Technologies
Real-World Examples
Nearly every RAG-based AI application uses a vector database. Notion, Shopify, and Brex use vector search for their AI features. pgvector adoption grew rapidly because teams could add vector search to existing PostgreSQL instances without adding a new database to their stack.
Challenges & Considerations
Recall vs. latency tradeoff, approximate search may miss relevant results. Embedding model updates require full re-indexing. Storage costs scale linearly with vector count and dimensionality. The market is crowded with many competing solutions.
Document Stores
Schema-flexible databases storing semi-structured data as JSON-like documents. Well suited to evolving data models and content-heavy applications.
Mainstream
Document Stores
Schema-flexible databases storing semi-structured data as JSON-like documents. Well suited to evolving data models and content-heavy applications.
How It Works
Document databases store records as self-describing JSON/BSON documents rather than fixed-schema rows. Each document can have a different structure, enabling rapid schema evolution. Documents are typically organized into collections. Queries use document-specific query languages (MQL for MongoDB) rather than SQL. Indexes can be created on any field, including nested fields within documents.
Key Technologies
- MongoDB (market leader, BSON format)
- CouchDB (HTTP/JSON API, replication)
- Amazon DocumentDB (MongoDB-compatible managed service)
- Firebase Firestore (Google, real-time sync)
- RavenDB (.NET ecosystem)
Real-World Examples
MongoDB has 46,000+ customers globally according to their public filings. It is widely used in content management systems, product catalogs, and mobile app backends where schema flexibility accelerates development. Forbes, Toyota, and Bosch use MongoDB in production.
Challenges & Considerations
Lack of joins means denormalization is common, leading to data duplication. Multi-document transactions were only added in MongoDB 4.0 (2018), and still have performance overhead. The SSPL license limits cloud hosting options. Data modeling requires different thinking than relational design.
Key-Value Stores
The simplest data model, mapping keys to values. Optimized for extremely low-latency reads and writes. Commonly used for caching and session storage.
Mainstream
Key-Value Stores
The simplest data model, mapping keys to values. Optimized for extremely low-latency reads and writes. Commonly used for caching and session storage.
How It Works
Key-value stores provide a dictionary-like interface: set a value for a key, get the value back by key. The simplicity of this model enables extremely fast operations, often sub-millisecond. Data is typically stored in memory (Redis) for speed, with optional persistence to disk. Advanced key-value stores add data structures on top of the basic model, such as lists, sets, sorted sets, and hashes.
Key Technologies
- Redis (in-memory, rich data structures)
- Memcached (simple, distributed caching)
- Amazon DynamoDB (managed, serverless)
- etcd (distributed, Kubernetes config store)
- Valkey (Redis fork, Linux Foundation)
Real-World Examples
Redis is used by Twitter for timeline caching, GitHub for job queuing, and Snapchat for session management. DynamoDB powers Amazon.com itself. etcd stores all cluster state for every Kubernetes cluster in the world. After Redis changed its license in 2024, the Linux Foundation forked it as Valkey.
Challenges & Considerations
Limited query capabilities. No joins, no complex filtering without additional indexes. In-memory stores require enough RAM for the entire dataset. Redis licensing changes (from BSD to dual SSPL/RSAL in 2024) caused a community split.
Graph Databases
Store data as nodes and edges, making relationship-heavy queries far more efficient than SQL joins.
Growing
Graph Databases
Store data as nodes and edges, making relationship-heavy queries far more efficient than SQL joins.
How It Works
Graph databases model data as vertices (nodes) connected by edges (relationships). Both nodes and edges can have properties. Queries traverse the graph following edges, which is far more efficient than multi-table joins in relational databases for relationship-heavy queries. Most graph databases use property graph models with query languages like Cypher (Neo4j) or Gremlin (Apache TinkerPop).
Key Technologies
- Neo4j (market leader, Cypher query language)
- Amazon Neptune (managed, multi-model)
- ArangoDB (multi-model: document + graph)
- Apache TinkerPop/Gremlin (standard traversal API)
- Memgraph (in-memory, real-time analytics)
Real-World Examples
Neo4j is used for fraud detection at major banks, recommendation engines at eBay, and knowledge graphs at NASA. Panama Papers investigation used Neo4j to map networks of offshore entities. LinkedIn uses a graph database for its social network features.
Challenges & Considerations
Horizontal scaling is harder than document or key-value stores. The graph query language ecosystem is fragmented (Cypher vs. Gremlin vs. SPARQL). Not suited for bulk analytical queries over large datasets. Smaller community and fewer tools compared to relational databases.
NewSQL
Distributed databases combining horizontal scalability with strong consistency and full SQL support.
Growing
NewSQL
Distributed databases combining horizontal scalability with strong consistency and full SQL support.
How It Works
NewSQL databases distribute data across multiple nodes using range or hash-based sharding, while maintaining ACID transactions across shards using distributed consensus protocols like Raft or Paxos. They present a standard SQL interface, so applications written for PostgreSQL or MySQL can often migrate with minimal changes. Reads and writes are automatically routed to the correct shard.
Key Technologies
- CockroachDB (PostgreSQL-compatible, multi-region)
- TiDB (MySQL-compatible, PingCAP)
- YugabyteDB (PostgreSQL-compatible)
- Google Spanner (globally distributed, TrueTime)
- PlanetScale (MySQL-compatible, Vitess-based)
Real-World Examples
Google Spanner runs Google Ads and Google Play, serving millions of transactions per second globally. CockroachDB is used by DoorDash, Netflix, and Bose. TiDB handles financial workloads at banks in Asia. PlanetScale (based on Vitess) was used by YouTube before becoming a standalone product.
Challenges & Considerations
Higher operational complexity than single-node databases. Cross-region transactions have higher latency due to consensus overhead. Cost is significantly higher than traditional databases. Many applications never actually need distributed SQL.
Analytical / OLAP
Columnar engines optimized for scanning and aggregating large volumes of data for analytics and business intelligence.
Mainstream
Analytical / OLAP
Columnar engines optimized for scanning and aggregating large volumes of data for analytics and business intelligence.
How It Works
OLAP databases store data in columnar format rather than row-based format. This means all values for a single column are stored together on disk, enabling highly efficient compression (similar values compress well) and fast aggregation queries (sum, count, average) that only read the columns they need. These engines are optimized for read-heavy analytical workloads, not transactional writes.
Key Technologies
- ClickHouse (open-source, real-time analytics)
- DuckDB (in-process, embedded analytics)
- Snowflake (cloud data warehouse)
- Google BigQuery (serverless analytics)
- Apache Druid (real-time OLAP)
Real-World Examples
ClickHouse was originally built at Yandex for web analytics and now powers analytics at Cloudflare, Uber, and eBay. DuckDB is used by data scientists and analysts for local-first analytics on CSV, Parquet, and JSON files without a server. Snowflake has over 9,000 customers and processes exabytes of data.
Challenges & Considerations
Not designed for transactional workloads (single-row updates are expensive). Real-time ingestion adds complexity. Cloud data warehouse costs can grow quickly with data volume. Schema changes in columnar stores can be more disruptive than in row-based systems.
Time-Series Databases
Purpose-built for timestamped, append-heavy data like metrics, sensor readings, and logs.
Growing
Time-Series Databases
Purpose-built for timestamped, append-heavy data like metrics, sensor readings, and logs.
How It Works
Time-series databases optimize for the unique access patterns of timestamped data: high-rate appends, range scans over time windows, and downsampling/aggregation queries. They use time-based partitioning, automatic compression (often 10-20x), and retention policies to manage data lifecycle. Queries typically filter by time range and aggregate across metrics or tags.
Key Technologies
- InfluxDB (purpose-built, Flux query language)
- TimescaleDB (PostgreSQL extension)
- Prometheus (metrics monitoring, pull-based)
- QuestDB (high-performance, SQL interface)
- Apache IoTDB (IoT-focused)
Real-World Examples
Prometheus is the standard for Kubernetes monitoring, used by almost every cloud-native deployment. TimescaleDB extends PostgreSQL for IoT and metrics workloads. InfluxDB is used in manufacturing, energy, and telecommunications for sensor data. Tesla collects time-series telemetry from its vehicle fleet.
Challenges & Considerations
High cardinality (many unique tag combinations) degrades query performance. Storage costs grow quickly with high-frequency data. Choosing between a specialized time-series DB and a general-purpose database with time-series features (PostgreSQL + TimescaleDB) is not always clear.
Streaming Databases
Databases that continuously process incoming event streams, maintaining materialized views updated in real time.
Emerging
Streaming Databases
Databases that continuously process incoming event streams, maintaining materialized views updated in real time.
How It Works
Streaming databases consume events from message brokers (Kafka, Redpanda) and continuously update materialized views as new events arrive, rather than requiring batch ETL jobs. They combine the semantics of stream processing with the query interface of a database. Users define queries in SQL that run continuously over the stream, producing always-up-to-date results.
Key Technologies
- RisingWave (PostgreSQL-compatible streaming DB)
- Materialize (streaming materialized views)
- ksqlDB (SQL interface for Kafka Streams)
- Apache Flink SQL (distributed stream processing)
- Decodable (managed Flink)
Real-World Examples
Materialize was used by financial firms for real-time risk calculations. ksqlDB enables real-time analytics on Kafka event streams. Companies like Uber, Netflix, and LinkedIn use Apache Flink for real-time data processing at scale. RisingWave targets developers who want PostgreSQL-compatible streaming.
Challenges & Considerations
Relatively new category with rapidly evolving products. Exactly-once semantics across stream and state are complex to guarantee. Debugging streaming queries is harder than batch queries. Integration with existing data infrastructure adds complexity.