Observed Signal · Jun 16, 2026 · Technical Guide · Source: DEV Community · Impact: 2/5 · Sentiment: Neutral
SQL vs Python: Choose by What vs How
A data engineering guide by Vinicius Fagundes explaining a simple rule for choosing between SQL and Python: if you are declaring what data you want, use SQL; if you are describing how to transform it step-by-step, use Python. The post illustrates common cases where SQL excels (joins, aggregations, window functions, dedup-to-latest, running totals, sessionization) and where Python is preferable (per-row API/model enrichment, many independent business rules, logic that depends on the computed result of a prior row). It contrasts concise SQL window examples with equivalent imperative Python implementations and emphasizes scale, locality, parallelism, testability, and cost. The author describes a real pipeline rewrite that reduced runtime from 8 hours to 47 minutes by moving set operations into SQL.
Practical guidance for data pipeline design that can materially reduce runtime and cost by choosing the appropriate execution layer (SQL warehouse vs Python), illustrated with concrete examples and a performance win.
Track Snowflake Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Author Vinicius Fagundes published a technical guide explaining when to use SQL vs Python in data pipelines.
- Rule taught: if declaring what data you want, use SQL; if describing how to transform it step-by-step, use Python.
- SQL examples include dedup-to-latest with ROW_NUMBER()/QUALIFY, moving averages with window frames, and sessionization using running SUM over flags.
- Python is recommended for per-row external calls/enrichment, many independent business rules, unit-testable procedural logic, and iteration where row N depends on computed row N-1.
- The author reports a pipeline rewrite (moving joins and a window into SQL) reduced runtime from 8 hours to 47 minutes.
Connected Companies & Entities
2 Entities mappedRelated Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Practical SQL Functions for Data Work
This technical tutorial surveys six categories of SQL functionality commonly used in financial and operational data work: row-level functions, date/time handling, string manipulation, joins, window functions, and set operators. It provides concrete syntax and examples (e.g., CASE, COALESCE/ISNULL, CAST/CONVERT, GETDATE/CURRENT_DATE/NOW, UPPER/LOWER/TRIM, various JOIN types, ROW_NUMBER/RANK/LAG/LEAD, UNION/INTERSECT/EXCEPT) and highlights practical concerns such as portability differences across database engines (date functions), window-function usage patterns, and performance trade-offs (UNION vs UNION ALL). The article emphasizes using these functions to transform, filter, combine, and rank data before analysis, especially in transaction, reconciliation, and KYC contexts.
PostgreSQL for Data Engineers: Indexes & Bulk Loads
This technical guide outlines practical PostgreSQL patterns for production data pipelines, focusing on operations that determine whether scheduled jobs succeed. It compares Python-to-Postgres loading methods (pandas.to_sql, psycopg2.execute_values, psycopg2.COPY) and recommends COPY for large backfills and execute_values for incremental writes. The article covers idempotent upserts with ON CONFLICT (including IS DISTINCT FROM to avoid unnecessary updates), index types and when to use B-tree, GIN, BRIN, partial and expression indexes, and how to read EXPLAIN ANALYZE. It also discusses window functions for time-series, CTE inlining behavior, JSONB indexing, pgvector for embedding search (HNSW vs IVFFlat), materialized views, table partitioning, routine maintenance (VACUUM/ANALYZE, autovacuum), practical pipeline schema patterns, and connection-pool settings for robust pipelines.
Introduction to NoSQL and MongoDB with Python
This tutorial defines NoSQL (commonly phrased today as “Not Only SQL”), explains why it emerged alongside traditional relational databases, and compares NoSQL vs. SQL across structure, schema, scaling, and query/relationship handling. It describes four primary NoSQL types—key-value, document, wide-column, and graph—explaining typical trade-offs and example use cases (e.g., Redis for caching/key-value, MongoDB for document stores, wide-column for time-series/IoT, graph for social/fraud). The piece also demonstrates how to interact with MongoDB from Python using the pymongo library and maps MongoDB concepts (collections, documents) to relational equivalents (tables, rows).
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.