Observed Signal · Jul 15, 2026 · Technical Release · Source: DEV Community · Impact: 3/5 · Sentiment: Positive
Infrastructure / Data Movement (ETL) Market: Open-source apitap moves 10M rows in 9.9s
The author published apitap, an open-source Rust-core with Python bindings engine that copies whole tables between databases with no config. Using the pip-distributed wheel on a 16-core machine, apitap moved 10 million rows from Postgres to ClickHouse in 9.9 seconds and sustained 100M rows / 46 GB in 139 seconds (1.2 TB/hour) in later runs. apitap supports Postgres→Postgres, Postgres→ClickHouse, and MySQL→ClickHouse/Postgres routes, uses bounded streaming memory to run in very small containers (0.5 vCPU / 256 MB), stages into shadow tables for atomic swaps, and includes a reproducible benchmark comparing it to dlt and ingestr. The project is MIT-licensed and open for contributors; additional connectors and platforms are on the roadmap.
A performant, low-resource open-source data-movement engine can materially reduce cloud pipeline costs and change how teams architect ETL for large single-table transfers; the release and reproducible benchmarks are relevant to data infrastructure and MarTech teams.
Key Takeaways & Evidence Grounding
- apitap is an open-source engine (Rust core, Python bindings, MIT license) that moves whole tables between databases with a single function call.
- Using the pip-installed wheel on a 16-core box, apitap moved 10 million rows from Postgres to ClickHouse in 9.9 seconds.
- Supported routes today: Postgres→Postgres (binary COPY passthrough), Postgres→ClickHouse (transcodes COPY to ClickHouse RowBinary), and MySQL→ClickHouse/Postgres (decodes MySQL wire protocol).
- apitap is memory-bounded by design and can move large tables from a 0.5 vCPU / 256 MB container by streaming bytes with TCP backpressure and splitting work inside a table.
- Benchmarks were run reproducibly against ingestr and dlt under identical conditions; apitap reported sustained throughput up to 1.2 TB/hour on a larger run (100M rows / 46 GB in 139 s).
Connected Companies & Entities
4 Entities mappedClickHouse
Open-source analytics database with managed cloud and observability.
“On a 16-core box, that call moves 10 million rows from Postgres to ClickHouse in 9.9 seconds - measured from the exact wheel you get with pi...”
Airbyte
Open-source data integration and AI context platform for enterprises.
“Airbyte first....”
GitHub
Developer platform for code collaboration, automation and AI coding.
“Code: [github.com/apitap/apitap-lib]...”
Snowflake
Managed enterprise data cloud for analytics, sharing, AI, and clean rooms.
“Snowflake/BigQuery and Arrow export are coming (the connector architecture makes a new destination one file)....”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.