What I build and learn at work: data platforms, Spark, Fabric and the tools around them.
Snowflake's DML error logging offers a robust mechanism to handle data ingestion errors, ensuring data integrity by capturing problematic rows in dedicated error tables. This feature allows for seamless debugging and recovery, crucial for maintaining high-quality data pipelines.
Microsoft Fabric's multitasking capabilities streamline data engineering workflows by allowing users to manage multiple tasks simultaneously. This feature enhances productivity and reduces context-switching, making it a valuable tool for data engineers.
Delta Lake 3.3 introduces row tracking backfill, enabling row-level lineage audits on existing tables. This feature is crucial for maintaining data integrity and auditing changes across table versions.
Apache Spark 4.1 introduces Declarative Pipelines, transforming how data engineers build and manage data flows. Coupled with Real-Time Mode in Structured Streaming, it offers unprecedented low-latency processing for dynamic workloads.
Explore how Databricks' compliance and security settings APIs help safeguard your data. Learn to configure identity management, secure data in transit and at rest, and protect your network infrastructure.
Fabric IQ transforms how organizations unify analytical and operational data by providing a semantic intelligence layer over OneLake. This enables consistent interpretation across analytics tools and AI agents, driving more informed business decisions.
Explore the architectural differences between AWS Capabilities by Region Tool and Azure Resource Manager. Understand their cost models, operational burdens, and scenarios where one might be preferred over the other.
Apache Airflow 3.1 introduces Human-in-the-Loop (HITL) workflows, enabling human decision-making within automated processes. This feature enhances AI/ML model validation, content moderation, and approval workflows by pausing tasks for human review.
Explore how Snowflake's search optimization service can enhance the performance of queries using scalar subqueries. Learn the mechanics behind enabling search optimization and its impact on query execution.
Google Cloud's Bigtable in-memory tier offers sub-millisecond read latency and high throughput, revolutionizing data access for time-sensitive applications. This article delves into how this tier works, its benefits, and potential pitfalls.
Learn how to trace AI agents in Amazon OpenSearch Service using Prometheus integration. This guide explores the unified observability features, including AI agent tracing and Prometheus metrics, to enhance your application's performance monitoring.
Genie Code revolutionizes data engineering by providing AI-driven data discovery and pipeline automation. It integrates deeply with Unity Catalog to offer contextual insights and streamline complex workflows.
As AI reshapes entry-level data engineering roles, new opportunities arise for those equipped with advanced skills. Understanding the AI-driven transformation is key to thriving in these evolving careers.
Delta Lake 4.1.0 introduces enhanced storage management capabilities through Unity Catalog, optimizing data governance and performance. This article explores how these features streamline data operations and improve efficiency.
Explore how Apache Spark's Real-Time Mode enhances declarative pipelines, enabling sub-second latency for complex data processing tasks. Learn how to leverage these capabilities for efficient and scalable data engineering.
Explore how Lakebase's integration with OpenTelemetry enhances observability by exporting metrics and logs to various OTLP-compatible backends. Understand the configuration process and common pitfalls to avoid.
Cloning databases in Snowflake that contain hybrid tables offers powerful capabilities for data restoration and environment hydration. This article explores the nuances of cloning hybrid tables, including performance considerations and common pitfalls.
Asset partitioning in Apache Airflow 3.2 introduces a new level of granularity to data-aware scheduling, allowing Dags to trigger based on specific data partitions. This feature optimizes resource usage and reduces operational noise by ensuring only relevant downstream tasks are executed.
Amazon OpenSearch Serverless Collection Groups allow for cost-effective management of multi-tenant workloads by sharing compute resources while maintaining security through distinct KMS keys. This approach optimizes resource utilization and ensures predictable performance.
The Data Engineering Agent in BigQuery automates complex data pipeline tasks using natural language. This tool simplifies pipeline creation and maintenance, allowing data engineers to focus on strategic tasks.