DuckDB

tags : Data Engineering, Information Retrieval, Database

Resources §

• Summing columns in remote Parquet files using DuckDB | Hacker News
• Querying Parquet with Precision using DuckDB - DuckDB

Plugins / Extensions §

DBT §

• dbt-duckdb
  • DuckDB & dbt | End-To-End Data Engineering Project
  • This additionally has extensions such as milicevica23/dbt-duckdb-delta-plugin-demo
  • Eg. delta plugin uses the delta-rs package, which enables reading directly from different cloud object stores.
  • You can write your own plugins

Writing to object store §

• See httpsfs extension

Others §

• GitHub - jwills/buenavista: A Postgres Proxy Server in Python (For duckdb)

Others §

• Merge

  import duckdb
  
  duckdb.execute("""
  COPY (SELECT * FROM '*.parquet') TO 'merge.parquet' (FORMAT 'parquet');
  """)