Observed Signal · Jul 15, 2026 · Technical Release · Source: DEV Community · Impact: 3/5 · Sentiment: Positive
Open-source apitap moves 10M rows in 9.9s
The author published apitap, an open-source Rust-core with Python bindings engine that copies whole tables between databases with no config. Using the pip-distributed wheel on a 16-core machine, apitap moved 10 million rows from Postgres to ClickHouse in 9.9 seconds and sustained 100M rows / 46 GB in 139 seconds (1.2 TB/hour) in later runs. apitap supports Postgres→Postgres, Postgres→ClickHouse, and MySQL→ClickHouse/Postgres routes, uses bounded streaming memory to run in very small containers (0.5 vCPU / 256 MB), stages into shadow tables for atomic swaps, and includes a reproducible benchmark comparing it to dlt and ingestr. The project is MIT-licensed and open for contributors; additional connectors and platforms are on the roadmap.
A performant, low-resource open-source data-movement engine can materially reduce cloud pipeline costs and change how teams architect ETL for large single-table transfers; the release and reproducible benchmarks are relevant to data infrastructure and MarTech teams.
Track Airbyte Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your Polaris7 workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- apitap is an open-source engine (Rust core, Python bindings, MIT license) that moves whole tables between databases with a single function call.
- Using the pip-installed wheel on a 16-core box, apitap moved 10 million rows from Postgres to ClickHouse in 9.9 seconds.
- Supported routes today: Postgres→Postgres (binary COPY passthrough), Postgres→ClickHouse (transcodes COPY to ClickHouse RowBinary), and MySQL→ClickHouse/Postgres (decodes MySQL wire protocol).
- apitap is memory-bounded by design and can move large tables from a 0.5 vCPU / 256 MB container by streaming bytes with TCP backpressure and splitting work inside a table.
- Benchmarks were run reproducibly against ingestr and dlt under identical conditions; apitap reported sustained throughput up to 1.2 TB/hour on a larger run (100M rows / 46 GB in 139 s).
Connected Companies & Entities
4 Entities mapped“Airbyte first.”
“On a 16-core box, that call moves 10 million rows from Postgres to ClickHouse in 9.9 seconds - measured from the exact wheel you get with pi...”
“Code: [github.com/apitap/apitap-lib]”
“Snowflake/BigQuery and Arrow export are coming (the connector architecture makes a new destination one file).”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
ClickHouse vs PostgreSQL: OLAP vs OLTP Comparison
A developer-authored technical post comparing ClickHouse and PostgreSQL as part of a '100 Days of ClickHouse' series. The article explains that PostgreSQL is a row-oriented OLTP database optimized for transactional workloads with frequent inserts/updates/deletes and strong transactional guarantees, while ClickHouse is a column-oriented OLAP database designed for large-scale analytics, fast aggregations and time-series/event analytics. It outlines storage and compression differences, scaling considerations, and common deployment patterns where organizations use PostgreSQL for operational data and ClickHouse for analytical/reporting workloads. The piece aims to guide database selection based on workload requirements rather than popularity or benchmarks.
Powering 800 Million ChatGPT Users with PostgreSQL
OpenAI published an engineering post describing how it scaled PostgreSQL to support millions of queries per second and serve 800 million ChatGPT users. The team retained a single-primary Azure PostgreSQL flexible server for writes and operated nearly 50 geo-distributed read replicas, while migrating shardable, write-heavy workloads to sharded systems such as Azure Cosmos DB. Key techniques included aggressive query optimization, workload isolation, PgBouncer connection pooling (reducing average connection time from 50ms to 5ms), cache locking to prevent cache-miss storms, rate limiting, strict schema-change controls, and testing cascading replication with Azure to scale replicas. OpenAI reports low p99 read latency, five-nines availability, and only one SEV-0 Postgres incident in the past 12 months.
High-Performance ETL for Apple Health XML Exports
This technical tutorial describes building a high-concurrency ETL engine to process very large Apple Health XML exports. The author outlines a performance-first architecture: a streaming Rust XML parser (quick-xml) to extract <Record> tags, export data as Apache Arrow record batches via PyO3 for zero-copy consumption in Python/Polars, and bulk-ingest cleaned data into ClickHouse using clickhouse-connect. The post includes code snippets for the Rust parser, Arrow-to-Polars bridge, ClickHouse table schema and bulk insert, and discusses production considerations such as parallelization, schema evolution, malformed XML handling, Grafana visualization, and feeding Arrow buffers into ML frameworks like PyTorch.
Receive all future signals for Airbyte & competitors in your briefing
Never miss material M&A deals, leadership changes, account wins, or product rollouts. Polaris7 monitors 20,000+ primary sources 24/7, delivering verified telemetry directly into your executive briefing.