What I build and learn at work: data platforms, Spark, Fabric and the tools around them.
AWS and Azure offer robust cloud services, but their regional capabilities and service availability differ significantly. This comparison explores their architectural differences, cost models, and operational implications to guide strategic cloud decisions.
Learn how to optimize your Microsoft Fabric CI/CD workflows by parameterizing connections using variable libraries and parameter.yml files. This approach ensures seamless environment transitions and reduces manual updates.
Serverless compute is reshaping the landscape of data engineering by simplifying infrastructure management and enhancing performance. This article explores how it impacts Spark Declarative Pipelines, comparing it with traditional methods and highlighting its benefits.
Microsoft Viva Engage offers data engineers a platform to foster community and engagement within their organizations. By leveraging its features, data engineers can enhance collaboration, share knowledge, and build a stronger organizational culture.
Delta Lake 3.2.0 introduces type widening, a feature that simplifies schema evolution by allowing data type changes without rewriting data. This article explores how type widening works, its benefits, and common pitfalls to avoid.
Apache Spark Connect introduces a client-server architecture that decouples the client from the Spark cluster, offering improved performance and flexibility for Python applications. This article explores how to leverage Spark Connect to optimize Python client performance.
Discover how to efficiently merge DataFrames using PySpark's df.mergeInto API, a powerful tool for handling complex data integration tasks in Delta Lake. Learn the underlying mechanics, common pitfalls, and best practices to optimize your data workflows.
Learn how to migrate Azure Data Factory pipelines to Microsoft Fabric using PowerShell and built-in tools. Understand the best practices, common pitfalls, and when to choose this migration path.
Discover how Delta Change Data Feed in Microsoft Fabric enables real-time data replication, simplifying analytics by continuously syncing changes from source databases to OneLake. Learn about its setup, benefits, and common pitfalls.
Databricks Genie Code revolutionizes data engineering by automating pipeline development with generative AI. This article explores how Genie Code compares to traditional methods and what its capabilities mean for the future of data engineering.
Multifactor authentication (MFA) is becoming a cornerstone in data engineering, enhancing security by requiring multiple forms of verification. As data environments grow more complex, understanding and implementing MFA effectively is crucial for safeguarding sensitive information.
SQL Server 2025 introduces vector search and indexing capabilities, enabling efficient similarity searches across large datasets. This article explores how these features work, their implementation, and the engineering challenges they address.
Delta Lake 4.1.0 introduces conflict-free feature enablement, allowing you to activate Deletion Vectors and Column Mapping on existing tables without disrupting concurrent writes. This advancement simplifies table management and enhances performance.
Agentic AI is transforming data engineering by automating complex, multi-step tasks with minimal human intervention. This approach enhances efficiency and scalability, allowing data teams to focus on strategic initiatives.
AI agents are reshaping data engineering by automating repetitive tasks and enabling more strategic work. However, successful implementation requires robust data foundations and careful integration into existing workflows.
Delta Lake 4.1.0 introduces significant enhancements in storage management through Unity Catalog. This release simplifies table management, improves performance, and ensures consistent governance across platforms.
Apache Spark 4.1's Real-Time Mode introduces millisecond latency, challenging Apache Flink's dominance in low-latency streaming. This article explores their architectures, use cases, and performance to help you choose the right tool.
Real-Time Mode in Apache Spark Structured Streaming offers ultra-low latency processing for immediate data response. This guide explores its architecture, setup, and practical applications, highlighting its advantages over traditional micro-batch processing.
Azure Data Factory's Power Query offers a code-free, scalable approach to data wrangling. Learn how to leverage this tool for efficient data preparation and transformation at cloud scale.
The April 2026 update to Microsoft Fabric introduces significant enhancements to Materialized Lake Views, making them more robust and efficient for data engineers. These improvements streamline medallion architecture implementation, offering new features like multi-schedule support and PySpark authoring.