Skip to content
learn
Global menu

DuckDB — Mission

Why I’m learning this

Section titled “Why I’m learning this”

Every question I ask about the CI of this site — which workflow fails most, how long the Windows jobs take, which step got slower — ends in a pile of JSON from gh run list and gh api. I read it with jq, or paste it into a spreadsheet. DuckDB promises to answer those questions with SQL, straight from the files, without a server to install.

Who this course is for

Section titled “Who this course is for”

You know SQL from SQL Server, PostgreSQL or SQLite: SELECT, JOIN, GROUP BY. You write C# or Java. You don’t need to know anything about analytics or data engineering.

What DuckDB is, in one table

Section titled “What DuckDB is, in one table”
SQL Server / PostgreSQL SQLite DuckDB
Runs as a server inside your process inside your process
Built for transactions (OLTP) transactions, small apps analytics (OLAP): scans, aggregates, joins over many rows
Storage rows rows columns
Database a server instance one file one file, or none: it also queries CSV, JSON and Parquet files directly
From .NET Microsoft.Data.SqlClient, Npgsql Microsoft.Data.Sqlite DuckDB.NET (ADO.NET)
From Java JDBC driver JDBC driver JDBC driver

The first row of the table, drawn: SQL Server and PostgreSQL run as a server, in a process of their own, while DuckDB runs inside your process, next to the files it reads.

flowchart LR
    subgraph p1["your process"]
        app1["C# or Java code"] --> driver["driver"]
    end
    subgraph p2["server process"]
        server["SQL Server or PostgreSQL"]
    end
    driver --> server
    subgraph p3["your process"]
        app2["C# or Java code"] --> duck["DuckDB, through DuckDB.NET or JDBC"]
    end
    dbfile[("one database file, or none")]
    files[("CSV, JSON and Parquet files")]
    duck --> dbfile
    duck --> files

The data

Section titled “The data”

The course queries a snapshot of this repository’s own GitHub Actions history, exported on 2026-09-14 into code/duckdb/data:

  • runs.json: 125 workflow runs, from gh run list --limit 1000 --json databaseId,workflowName,event,status,conclusion,createdAt,updatedAt,startedAt,headBranch,headSha,attempt;
  • jobs.json: the 315 jobs of those runs, with their steps, from the GitHub REST API.

They are the runs written by the GitHub Actions course: real failures, real timings, three operating systems.

By the end of this course, I will be able to

Section titled “By the end of this course, I will be able to”
  • query CSV, JSON and Parquet files with SQL from the DuckDB CLI;
  • use DuckDB’s SQL extensions to write shorter analytical queries;
  • flatten nested JSON with STRUCT, LIST and unnest;
  • convert between formats and read many files at once;
  • use DuckDB from C# and from Java;
  • read a query plan and know when DuckDB is the wrong tool.

Outline

Section titled “Outline”
# Lesson If you know SQL Server
1 First queries sqlcmd, OPENROWSET, SELECT INTO
2 Friendly SQL: dates, windows, QUALIFY, PIVOT window functions, PIVOT
3 Nested data: STRUCT, LIST, unnest OPENJSON, CROSS APPLY
4 Files: CSV, Parquet, globs, remote files BULK INSERT, external tables
5 DuckDB from C# ADO.NET
6 DuckDB from Java JDBC
7 Performance: plans and columnar storage execution plans, columnstore indexes
8 Persistence, transactions and concurrency isolation, locks
— Journal

Resources

Section titled “Resources”