Catalog Backends: Choosing Your Metadata Store

This vignette provides an in-depth look at DuckLake catalog backends - the database that stores metadata about your tables, schemas, snapshots, and file references. Choosing the right backend is crucial for performance, concurrency, and operational simplicity.

What is the Catalog?

DuckLake separates data (Parquet files) from metadata (table definitions, snapshots, file tracking). The metadata lives in a catalog database.

┌─────────────────────────────────────────────────────────┐
│                    DuckLake Architecture                 │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  ┌──────────────────┐      ┌──────────────────────────┐ │
│  │  Catalog (Meta)  │      │    Data (Parquet)        │ │
│  │                  │      │                          │ │
│  │  • Table schemas │      │  📁 schema/              │ │
│  │  • Snapshots     │◄────►│    📁 table/            │ │
│  │  • File refs     │      │      📄 data-001.parquet│ │
│  │  • Partitions    │      │      📄 data-002.parquet│ │
│  │  • Lineage       │      │                          │ │
│  └──────────────────┘      └──────────────────────────┘ │
│         ▲                                                │
│         │                                                │
│    Which database?                                       │
│    • DuckDB                                              │
│    • SQLite                                              │
│    • PostgreSQL                                          │
│    • Quack (EXPERIMENTAL)                                │
│    • (Future: SQL Server)                                │
│                                                          │
└─────────────────────────────────────────────────────────┘

The catalog database choice affects:

Concurrency - How many users can read/write simultaneously
Infrastructure - Server required vs. just a file
Performance - Metadata query speed
Operational complexity - Backup, monitoring, maintenance

Current Backends

DuckDB (Native)

db_connect(
 catalog_type = "duckdb",
  metadata_path = "catalog.ducklake",
  data_path = "/data/lake"
)

Aspect	Details
Storage	Single `.ducklake` file
Concurrency	Single client only
Infrastructure	None - just a file
Best for	Personal use, development, testing

Advantages:

Simplest setup - just a file path
Fastest metadata operations (native DuckDB)
No external dependencies
Perfect for local development and prototyping

Disadvantages:

Single client limitation - only one connection at a time
File locking prevents concurrent access
Not suitable for shared environments

When to use:

Local development and testing
Single-user analytical workflows
CI/CD pipelines (isolated runs)
Prototyping before moving to production

SQLite

db_connect(
  catalog_type = "sqlite",
  metadata_path = "//network/share/catalog.sqlite",
  data_path = "//network/share/data"
)

Aspect	Details
Storage	Single `.sqlite` file
Concurrency	Multiple readers, single writer (with retry)
Infrastructure	None - just a file
Best for	Shared network drives, small teams

Advantages:

Still just a file - no server needed
Multiple users can read simultaneously
Write conflicts handled with automatic retry
Works with existing network share permissions
Familiar to IT teams (standard SQLite)

Disadvantages:

Single writer at a time (queued with retry)
Performance degrades over very slow network connections
No true concurrent writes

When to use:

Recommended default for shared environments
Network drive deployments
Teams where writes are infrequent
Environments where PostgreSQL isn’t available

Concurrency model:

User A reads  ───────────────────────────► OK (instant)
User B reads  ───────────────────────────► OK (instant)
User C writes ─────────────────┐
                               ├──────────► OK (gets lock)
User D writes ─────────────────┘
              waits... retries... ────────► OK (after C finishes)

PostgreSQL

db_connect(
  catalog_type = "postgres",
  metadata_path = "host=db.example.com dbname=ducklake user=app",
  data_path = "/data/lake"
)

Aspect	Details
Storage	PostgreSQL database
Concurrency	Full multi-reader, multi-writer
Infrastructure	PostgreSQL server required
Best for	Large teams, high concurrency, enterprise

Advantages:

True concurrent read and write access
Row-level locking - writes don’t block each other
Robust transaction support
Enterprise features: replication, backup, monitoring
Works with managed services (AWS RDS, Azure, GCP Cloud SQL)

Disadvantages:

Requires PostgreSQL server infrastructure
Additional operational overhead (backups, updates, monitoring)
Network latency for metadata operations
More complex setup

When to use:

Multiple users writing frequently
Enterprise environments with existing PostgreSQL
When you need high availability
Remote/distributed teams

Connection string formats:

# Key-value format
db_connect(
  catalog_type = "postgres",
  metadata_path = "host=localhost port=5432 dbname=ducklake user=myuser password=secret"
)

# URI format
db_connect(
  catalog_type = "postgres",
  metadata_path = "postgresql://myuser:secret@localhost:5432/ducklake"
)

Comparison Matrix

Feature	DuckDB	SQLite	PostgreSQL	Quack
Setup complexity	Trivial	Simple	Moderate	Moderate
Infrastructure	None	None	Server required	DuckDB server
Concurrent readers	1	Unlimited	Unlimited	Unlimited
Concurrent writers	1	1 (queued)	Unlimited	Unlimited
Network drive friendly	No	Yes	N/A	N/A
Cloud managed option	No	No	Yes	No
Metadata query speed	Fastest	Fast	Network dependent	Fast
Backup strategy	Copy file	Copy file	pg_dump / replication	DuckDB backup
Best team size	1	2-20	20+	2-50
Status	Stable	Stable	Stable	EXPERIMENTAL

Security & Access Control

DuckLake’s security model has two distinct layers:

Catalog security - Who can read/modify table definitions and metadata
Data security - Who can read/write the actual Parquet files

Current Model: File System Permissions

Currently, DuckLake relies on file system permissions for data access control:

┌─────────────────────────────────────────────────────────────┐
│                   Security Architecture                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Catalog (Metadata)              Data (Parquet Files)        │
│  ┌──────────────────┐           ┌──────────────────────┐    │
│  │ catalog.sqlite   │           │ /data/trade/         │    │
│  │                  │           │   imports/*.parquet  │    │
│  │ - Table schemas  │           │   exports/*.parquet  │    │
│  │ - Snapshots      │           │                      │    │
│  │ - File refs      │           │ /data/labour/        │    │
│  └────────┬─────────┘           │   employment/*.parquet│   │
│           │                     └──────────┬───────────┘    │
│           ▼                                ▼                │
│    File/DB permissions            Folder ACLs (Windows/NFS) │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Important limitation: If a user can read the Parquet files directly (bypassing DuckLake), they have full access to that data. DuckLake does not encrypt data at rest or enforce row/column-level security.

Security by Backend

Backend	Catalog Security	Data Security
DuckDB	File permissions only	File/folder ACLs
SQLite	File permissions only	File/folder ACLs
PostgreSQL	Full RBAC, SSL, row-level security	File/folder ACLs

PostgreSQL: Enhanced Security Options

PostgreSQL offers significantly more security features for the catalog:

# SSL/TLS encrypted connection
db_connect(
  catalog_type = "postgres",
  metadata_path = "host=db.example.com dbname=ducklake user=app sslmode=require"
)

PostgreSQL security features:

Feature	Description
Role-based access (RBAC)	Grant/revoke permissions per user/role
SSL/TLS encryption	Encrypt metadata in transit
Row-level security (RLS)	Restrict which snapshots/tables users can see
Audit logging	Track who accessed/modified metadata
LDAP/Kerberos/SCRAM	Enterprise authentication integration
Connection pooling	PgBouncer for secure connection management

Example: Restricting schema access in PostgreSQL:

-- Create roles for different teams
CREATE ROLE trade_team;
CREATE ROLE labour_team;

-- Grant access to specific schema metadata only
GRANT SELECT ON ducklake_schema TO trade_team
  WHERE schema_name = 'trade';

-- Users inherit permissions from their role
GRANT trade_team TO alice, bob;

Future: Native DuckLake Security

DuckLake may introduce native security features in future releases:

Potential features being discussed:

Catalog-level RBAC - Define permissions in DuckLake metadata itself
Schema-level grants - GRANT SELECT ON SCHEMA trade TO role_x
Table-level grants - GRANT INSERT ON table TO role_y
Encryption at rest - Encrypted Parquet files with key management
Column masking - Hide sensitive columns from certain users

Current workaround for sensitive data:

# Store sensitive data in a separate schema with restricted folder ACLs
db_write(sensitive_data, schema = "restricted", table = "pii")

# Folder structure:
# /data/restricted/  <- Only authorized users have ACL access
#   pii/*.parquet

Quack Security Considerations

When Quack becomes available as a catalog backend, it will introduce new security considerations:

┌─────────────────────────────────────────────────────────────┐
│                    Quack Security Model                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Client ──── HTTPS ────► DuckDB Server ────► Data Files     │
│                              │                               │
│                              ▼                               │
│                    Authentication                            │
│                    Authorization                             │
│                    Audit logging                             │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Expected Quack security features:

Feature	Status
HTTPS transport	Available now
Token authentication	Available now
Reverse proxy support	Available now (nginx, Caddy)
Native RBAC	Planned
Query auditing	Planned

Securing Quack with a reverse proxy:

# nginx configuration for Quack
server {
    listen 443 ssl;
    server_name duckdb.example.com;

    ssl_certificate /etc/ssl/certs/duckdb.crt;
    ssl_certificate_key /etc/ssl/private/duckdb.key;

    location / {
        proxy_pass http://localhost:4242;
        proxy_set_header Authorization $http_authorization;
    }
}

Security Recommendations by Use Case

Use Case	Recommended Backend	Security Approach
Personal/Dev	DuckDB	Local file permissions
Team, low sensitivity	SQLite	Folder ACLs per schema
Team, sensitive data	PostgreSQL	RBAC + folder ACLs + SSL
Enterprise, compliance	PostgreSQL	Full RBAC + RLS + audit + encryption
Multi-tenant	PostgreSQL + Quack	Row-level security + API gateway

Data Cannot Be Fully Controlled via Catalog

Critical understanding: The catalog backend only controls metadata access. The actual data files (Parquet) are controlled by file system permissions.

# Even if PostgreSQL restricts a user from seeing table metadata...
# ...they could still read the parquet files directly if they have folder access:

# Direct file access bypasses DuckLake entirely
arrow::read_parquet("/data/trade/imports/file-001.parquet")

Mitigations:

Folder ACLs - Ensure Parquet folders have appropriate permissions
Network isolation - Store data on restricted network shares
Encryption - Use encrypted storage (BitLocker, LUKS, cloud KMS)
Monitoring - Audit file access at the OS level

Future: SQL Server Support

Current Status

SQL Server is not yet supported as a DuckLake catalog backend. This is tracked in:

DuckLake Discussion #892

Why SQL Server?

Many enterprise environments standardise on SQL Server:

Existing infrastructure and expertise
Azure SQL Database / Managed Instance availability
Corporate licensing agreements
Integration with Microsoft ecosystem (Power BI, Azure Data Factory)

Technical Path

The hugr-lab/mssql-extension provides native SQL Server connectivity for DuckDB:

Native TDS protocol (no ODBC/JDBC required)
Full DML support (INSERT, UPDATE, DELETE)
Transaction support
High-performance bulk loading via BCP protocol
Azure AD authentication

What’s missing: DuckLake needs to add ducklake:mssql: DSN support and generate SQL Server-compatible DDL.

What SQL Server Would Offer

# Future (once DuckLake supports it)
db_connect(
  catalog_type = "mssql",
  metadata_path = "Server=sql.example.com;Database=ducklake;User Id=app;Password=secret",
  data_path = "/data/lake"
)

Feature	Benefit
Enterprise integration	Use existing SQL Server infrastructure
Azure native	Azure SQL Database, Managed Instance
Familiar tooling	SSMS, Azure Data Studio
Existing permissions	AD/Entra ID authentication
High availability	Always On, geo-replication

Current Workarounds

Until SQL Server support arrives:

PostgreSQL - Use Azure Database for PostgreSQL in Azure environments
SQLite - Store catalog file on Azure Files or shared storage
Hybrid approach - Keep DuckLake metadata in PostgreSQL, source data in SQL Server

Quack Remote Protocol (EXPERIMENTAL)

What is Quack?

Quack is DuckDB’s client-server protocol, available since DuckDB v1.5.2:

RPC protocol for DuckDB-to-DuckDB communication
HTTP-based for network compatibility (port 9494 by default)
Native serialisation of DuckDB’s internal data vectors
Multi-writer support without locking

As of DuckDB 1.5.3, DuckLake supports Quack as a catalog backend. datapond includes experimental support for this.

How Quack Changes Everything

Before Quack (DuckDB catalog):                After Quack:
┌─────────────────────────────┐             ┌─────────────────────────────┐
│     Single Client Only      │             │   Multiple Clients          │
│                             │             │                             │
│  Client ──► DuckDB File     │             │  Client A ──┐               │
│             (locked)        │             │  Client B ──┼──► DuckDB     │
│                             │             │  Client C ──┘    Server     │
└─────────────────────────────┘             └─────────────────────────────┘

Using Quack with datapond

datapond supports Quack as an experimental catalog backend:

# Connect to a DuckDB server running Quack
db_connect(
  catalog_type = "quack",
  metadata_path = "quack:duckdb-server.example.com:9494/catalog.ducklake",
  data_path = "/data/lake"
)

You will see a warning about the experimental status. This is expected.

Benefits:

Feature	Impact
Multi-writer DuckDB	Native DuckDB speed with concurrent access
Inlining performance	Much faster than SQLite/PostgreSQL for streaming
Simplified stack	DuckDB for everything - no PostgreSQL needed
Edge deployment	Run DuckDB server close to data

Running a Quack Server

To use Quack, you need a DuckDB server:

# Start DuckDB with Quack server enabled
duckdb -cmd "INSTALL quack; LOAD quack; CALL quack_start_server();"

The server listens on port 9494 by default. See the Quack documentation for configuration options.

Status and Timeline

Version	Status
DuckDB 1.5.2	Quack beta available
DuckDB 1.5.3	DuckLake + Quack integration
DuckDB 2.0 (Fall 2026)	Production-ready Quack

Warning: The protocol, function names, and defaults may change before v2.0. Use in production at your own risk.

Quack Authentication

datapond supports basic token authentication for Quack:

# Option 1: Explicit token parameter
db_connect(
  catalog_type = "quack",
  metadata_path = "quack:server:9494/catalog",
  data_path = "/data/lake",
  quack_token = "my-secret-token"
)

# Option 2: Environment variable (recommended for scripts)
Sys.setenv(QUACK_TOKEN = "my-secret-token")
db_connect(
  catalog_type = "quack",
  metadata_path = "quack:server:9494/catalog",
  data_path = "/data/lake"
)

Running a Quack Server

To use Quack as a catalog backend, you need a running DuckDB server. A minimal server:

duckdb -cmd "
  INSTALL quack; LOAD quack;
  INSTALL ducklake; LOAD ducklake;
  SET quack_token = 'my-secret-token';
  ATTACH 'ducklake:catalog.ducklake' AS lake (DATA_PATH '/data/lake');
  SELECT * FROM quack_serve();
"

For production deployments (systemd, Docker, nginx, cloud), see vignette("quack-server").

Status and Timeline

Version	Status
DuckDB 1.5.2	Quack beta available
DuckDB 1.5.3	DuckLake + Quack integration
DuckDB 2.0 (Fall 2026)	Production-ready Quack

Warning: The protocol, function names, and defaults may change before v2.0. Use in production at your own risk.

Geospatial Data Considerations

Native Geometry Support

DuckLake 1.0 includes native support for geometry types via the DuckDB spatial extension:

# Load spatial extension
DBI::dbExecute(con, "INSTALL spatial; LOAD spatial;")

# Write geospatial data
db_write(
  geo_data,
  table = "boundaries",
  mode = "overwrite"
)

Supported geometry types:

POINT, LINESTRING, POLYGON
MULTIPOINT, MULTILINESTRING, MULTIPOLYGON
GEOMETRYCOLLECTION

GeoParquet Storage

DuckDB reads and writes GeoParquet natively:

# Spatial data stored as GeoParquet
boundaries <- db_read(table = "boundaries")

boundaries |>
  filter(ST_Intersects(geometry, ST_Point(-6.26, 53.35))) |>
  collect()

Formats That Can’t Be Stored Directly

Some geospatial formats don’t fit the tabular model:

Format	Issue	Solution
Raster (GeoTIFF, COG)	Grid data, not rows	Store file reference + metadata
Point clouds (LAS/LAZ)	Billions of points	Store summary + file path
3D models (CityGML)	Complex hierarchical	Store metadata + file reference
Vector tiles (MVT)	Pre-rendered tiles	Store tile index + paths

Metadata-Only Approach

For unsupported formats, store metadata in DuckLake while keeping files external:

# Create a metadata table for raster files
raster_catalog <- data.frame(
  file_id = 1:3,
  file_path = c(
    "/data/rasters/dem_2024.tif",
    "/data/rasters/landcover_2024.tif",
    "/data/rasters/ndvi_2024_q1.tif"
  ),
  format = "GeoTIFF",
  crs = "EPSG:2157",
  bounds_wkt = c(
    "POLYGON((-10.5 51.4, -5.5 51.4, -5.5 55.4, -10.5 55.4, -10.5 51.4))",
    "POLYGON((-10.5 51.4, -5.5 51.4, -5.5 55.4, -10.5 55.4, -10.5 51.4))",
    "POLYGON((-10.5 51.4, -5.5 51.4, -5.5 55.4, -10.5 55.4, -10.5 51.4))"
  ),
  resolution_m = c(10, 30, 10),
  acquisition_date = as.Date(c("2024-01-15", "2024-06-01", "2024-03-15")),
  file_size_mb = c(450, 120, 380),
  description = c(
    "Digital Elevation Model - 10m resolution",
    "CORINE Land Cover 2024",
    "NDVI composite Q1 2024"
  )
)

db_write(raster_catalog, table = "raster_catalog", mode = "overwrite")

# Document the catalog
db_comment(
  table = "raster_catalog",
  comment = list(
    description = "Metadata catalog for raster files stored externally",
    owner = "GIS Team",
    tags = c("geospatial", "raster", "metadata-only")
  )
)

Querying Spatial Metadata

# Find rasters that intersect an area of interest
raster_catalog <- db_read(table = "raster_catalog")

aoi <- "POLYGON((-7 53, -6 53, -6 54, -7 54, -7 53))"

raster_catalog |>
  filter(ST_Intersects(ST_GeomFromText(bounds_wkt), ST_GeomFromText(aoi))) |>
  select(file_path, description, acquisition_date) |>
  collect()

Hybrid Architecture for Geospatial

┌─────────────────────────────────────────────────────────────┐
│                  Geospatial Data Architecture               │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  DuckLake (Tabular + Vector Geometry)                       │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  boundaries     (POLYGON geometries)                 │   │
│  │  points_of_interest (POINT geometries)               │   │
│  │  road_network   (LINESTRING geometries)              │   │
│  │  raster_catalog (metadata only - file references)    │   │
│  │  pointcloud_index (metadata only - file references)  │   │
│  └─────────────────────────────────────────────────────┘   │
│                           │                                 │
│                           ▼                                 │
│  External Storage (Non-tabular formats)                     │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  /data/rasters/          (GeoTIFF, COG)              │   │
│  │  /data/pointclouds/      (LAS, LAZ)                  │   │
│  │  /data/tiles/            (MVT, PMTiles)              │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Decision Guide

Choosing the Right Backend for Your Environment

Most organisations deploying DuckLake will run it inside their corporate network on existing infrastructure. This changes the calculus significantly from cloud deployments.

Internal Network Reality

┌─────────────────────────────────────────────────────────────┐
│                    Corporate Firewall                        │
│  ┌───────────────────────────────────────────────────────┐  │
│  │                  Internal Network                      │  │
│  │                                                        │  │
│  │  ┌──────────┐    ┌──────────┐    ┌────────────────┐   │  │
│  │  │ Analysts │───►│ Catalog  │───►│ Network Share  │   │  │
│  │  │ (R users)│    │ Database │    │ (Parquet data) │   │  │
│  │  └──────────┘    └──────────┘    └────────────────┘   │  │
│  │                                                        │  │
│  │  Security: Windows ACLs / Active Directory             │  │
│  │  Infrastructure: Existing file servers                 │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

What you already have:

Network file shares with folder-based permissions
IT team managing Windows/AD infrastructure
Firewall protecting from external threats
Users authenticated via domain credentials

Key Questions

Ask these questions to choose the right backend:

1. How many people will write data concurrently?

Writers	Recommendation
1 (just you)	DuckDB - simplest possible
2-5 occasionally	SQLite - handles this fine
5-20 frequently	SQLite usually works, monitor for contention
20+ heavy writes	PostgreSQL or Quack

2. Do you have IT support for running servers?

IT Support	Recommendation
No dedicated IT	SQLite (just files, no servers)
IT can run PostgreSQL	PostgreSQL (if you need concurrency)
IT can run custom services	Quack (when stable)

3. What infrastructure already exists?

Existing Infrastructure	Recommendation
Network shares only	SQLite
PostgreSQL already in use	PostgreSQL
Strong Microsoft/Azure stack	SQLite now, SQL Server when supported

Decision Flowchart

Are you the only user?
├─ Yes ──► DuckDB (simplest)
└─ No
   │
   How many concurrent writers?
   ├─ 1-5 occasional ──► SQLite (no server needed)
   │
   ├─ 5-20 frequent
   │  └─ Is SQLite causing write contention?
   │     ├─ No ──► Stay with SQLite
   │     └─ Yes ──► PostgreSQL
   │
   └─ 20+ heavy writes
      └─ Can IT provision a database server?
         ├─ Yes ──► PostgreSQL
         └─ No ──► SQLite + accept queuing

Internal Network: SQLite is Usually Right

For most internal analytics teams, SQLite on a network share is the best choice:

db_connect(
  catalog_type = "sqlite",
  metadata_path = "//FILE-SERVER/DataLake/catalog.sqlite",
  data_path = "//FILE-SERVER/DataLake/data"
)

Why SQLite wins internally:

Factor	SQLite Advantage
Infrastructure	Zero - just a file on existing share
Permissions	Use existing Windows/NFS ACLs
IT overhead	None - IT already manages file servers
Backup	Included in existing file server backups
Concurrency	Multi-read, single-write (usually enough)
Familiarity	IT knows files; they may not know PostgreSQL

When SQLite isn’t enough:

You’re seeing actual write contention errors (not theoretical)
You need streaming/inlining at high throughput
Multiple automated jobs write simultaneously

When to Consider Quack Internally

Quack adds complexity. Only consider it if:

SQLite write contention is actually happening (not just theoretically possible)
You need high-performance inlining for streaming data
IT is willing to run and maintain a DuckDB server process
You have a plan for token distribution to authorized users

Internal Quack considerations:

Consideration	Implication
New server to manage	IT must provision, monitor, update, restart on failure
Token management	How do you securely distribute tokens to users?
HTTP on internal network	Traffic readable without TLS (may or may not matter)
No Windows auth integration	Tokens are separate from AD credentials
Backup strategy	Need to back up DuckDB database on server

Recommended Starting Point

┌─────────────────────────────────────────────────────────────┐
│  Start Here: SQLite on Network Share                         │
│                                                              │
│  //FILE-SERVER/DataLake/                                     │
│  ├── catalog.sqlite    ← Catalog (multi-reader)             │
│  └── data/                                                   │
│      ├── trade/        ← Team A: Read/Write ACL             │
│      ├── labour/       ← Team B: Read/Write ACL             │
│      └── shared/       ← Everyone: Read-only ACL            │
│                                                              │
│  • Zero infrastructure beyond existing file server           │
│  • Familiar permissions model (IT already knows this)        │
│  • Easy backup (part of normal file server backups)          │
│  • Upgrade to PostgreSQL/Quack later if needed               │
└─────────────────────────────────────────────────────────────┘

Migration Path

Start simple, scale up only when needed:

Development     Team Pilot      Production       Scale-Out
    │               │               │                │
    ▼               ▼               ▼                ▼
 DuckDB ────► SQLite ────► SQLite ────────► PostgreSQL
 (local)     (network     (still           (if actually
              share)       works!)          needed)
                              │
                              └────────────► Quack (v2.0+)
                                            (if inlining
                                             perf needed)

Key insight: The data files (Parquet) remain unchanged. You can switch catalog backends without migrating data - just point to a new catalog database.

Summary

Backend	Best For	Key Limitation
DuckDB	Solo development	Single client
SQLite	Small teams, network drives	Single writer
PostgreSQL	Enterprise, high concurrency	Requires server
Quack	DuckDB-native multi-writer	EXPERIMENTAL (stable in v2.0)
SQL Server	Microsoft shops	Not yet supported

For most users starting out, SQLite is the recommended default - it’s simple, requires no infrastructure, and handles typical team usage patterns well. Once Quack reaches production status (DuckDB 2.0, Fall 2026), it may become the preferred option for teams wanting multi-writer without PostgreSQL infrastructure.

What is the Catalog?

Current Backends

DuckDB (Native)

SQLite

PostgreSQL

Comparison Matrix

Security & Access Control

Current Model: File System Permissions

Security by Backend

PostgreSQL: Enhanced Security Options

Future: Native DuckLake Security

Quack Security Considerations

Security Recommendations by Use Case

Data Cannot Be Fully Controlled via Catalog

Future: SQL Server Support

Current Status

Why SQL Server?

Technical Path

What SQL Server Would Offer

Current Workarounds

Quack Remote Protocol (EXPERIMENTAL)

What is Quack?

How Quack Changes Everything

Using Quack with datapond

Running a Quack Server

Status and Timeline

Quack Authentication

Running a Quack Server

Status and Timeline

Geospatial Data Considerations

Native Geometry Support

GeoParquet Storage

Formats That Can’t Be Stored Directly

Metadata-Only Approach

Querying Spatial Metadata

Hybrid Architecture for Geospatial

Decision Guide

Choosing the Right Backend for Your Environment

Internal Network Reality

Key Questions

Decision Flowchart

Internal Network: SQLite is Usually Right

When to Consider Quack Internally

Recommended Starting Point

Migration Path

Summary

References