SEC Filings Lakehouse
AI-ready Iceberg lakehouse on real EDGAR filings
7.07M point-in-time financial facts with every restated version kept, so "what did we
know on date D" is one predicate. dbt with an enforced contract, 10-K and 10-Q text
chunked and embedded into pgvector as immutable index versions behind a retrieval-eval
gate, a FastAPI RAG endpoint with citations, Dagster orchestration, Terraform for three clouds.
What went wrong, and what caught it
The promotion gate rejected two of my three embedding indexes and was right both times:
one was a retrieval problem, fixed with hybrid search, and one was two bugs of mine in
the chunker and the eval itself. 17,977 restated facts in two quarters is why the
point-in-time model exists.
- Filings
- 13,240
- Facts
- 7.07M
- Promoted index recall@5
- 1.00
- Apache Iceberg
- dbt
- Dagster
- DuckDB
- pgvector
- FastAPI
- Terraform
- AWS · Azure · GCP
Repository
GitHub Archive Streaming Lakehouse
Kafka to Iceberg, exactly once, with CDC and FinOps
GH Archive events replayed onto Redpanda as Avro with a schema registry, Spark
Structured Streaming writing exactly-once into Iceberg with watermarks and RocksDB
state, Debezium CDC folded into an SCD2 dimension, replay and backfill, Prometheus and
Grafana, and a cost model that prices three AWS designs and says which one to pick.
What went wrong, and what caught it
The first run put every event in 1970. A freshness gauge I had added because consumer
lag says nothing about correctness caught it; lag did not. After a full-hour replay the
table held 169,753 events with 169,753 distinct ids: zero duplicates. The README lists
five things that broke.
- Events, distinct ids
- 169,753
- Duplicates after replay
- 0
- Files after compaction
- 75 → 15
- Kafka / Redpanda
- Avro
- Spark Structured Streaming
- Apache Iceberg
- Debezium
- Prometheus · Grafana
- Terraform
- AWS · Azure · GCP
Repository
Stack Overflow Retention
Causal inference on 1.77M new contributors
Does a fast first answer make new contributors stay? A five-step BigQuery pipeline
(44 GB scanned for $0.22) and four estimators: controlled OLS, IPW, propensity
stratification and 2SLS. The estimators live in a tested module, the dashboard reads
committed summaries, and the whole thing ships as a Docker image.
What the analysis found
The three controlled estimators agree on about +7.7 points of 30-day retention. The
instrumental-variable estimate came out at −20 points with a strong first stage,
which says the exclusion restriction does not hold. It is reported as a failure, not dropped.
- Users
- 1.77M
- D30 retention lift
- +7.7 pp
- BigQuery cost
- $0.22
- BigQuery
- statsmodels
- linearmodels
- Streamlit
- pytest
- Docker
Repository