tags : Data Engineering, Information Retrieval, Database
Resources
- Practical Applications for DuckDB (with Simon Aubury & Ned Letcher) - YouTube
- Summing columns in remote Parquet files using DuckDB | Hacker News
- Querying Parquet with Precision using DuckDB - DuckDB
- Build sub-second data applications with MotherDuck’s Wasm SDK
- Implementing Hardware-Friendly Databases (with DuckDB co-creator, Hannes Mühleisen) - YouTube
- DuckDB Doesn’t Need Data to Be a Database | Hacker News
Streaming data and DuckDB
- In-Memory Analytics for Kafka using DuckDB – Robert Yokota
- https://github.com/mistercrunch/duckstreams
- Redpanda | Cloud Topics: Efficiently stream data through object storage
Plugins / Extensions
DBT
dbt-duckdb- DuckDB & dbt | End-To-End Data Engineering Project
- This additionally has extensions such as milicevica23/dbt-duckdb-delta-plugin-demo
- Eg. delta plugin uses the delta-rs package, which enables reading directly from different cloud object stores.
- You can write your own plugins
Writing to object store
- See httpsfs extension
Others
Others
- Merge
import duckdb duckdb.execute(""" COPY (SELECT * FROM '*.parquet') TO 'merge.parquet' (FORMAT 'parquet'); """)