What I build and learn at work: data platforms, Spark, Fabric and the tools around them.
Spark Declarative Pipelines offer a streamlined approach to building data workflows, focusing on business logic while automating execution. This article explores how to leverage this framework to simplify ETL processes and improve pipeline reliability.
The 2026 Built-On Databricks Startup Challenge offers over $1 million in prizes for startups innovating on the Databricks platform. This competition is a launchpad for early-stage B2B companies to gain exposure and venture investment.
Adaptive Query Execution in Databricks SQL leverages parameter markers to enhance query performance and security. This approach allows dynamic value substitution, improving query reuse and protecting against SQL injection. Learn how to implement and optimize this technique effectively.
Liquid clustering in Delta Lake 3.2 offers a flexible and efficient way to optimize data layout for better write performance. This technique allows for dynamic clustering without rewriting existing data, adapting to evolving query patterns.
Apache Spark 4.1 introduces Real-Time Mode in Structured Streaming, enabling ultra-low-latency processing for critical workloads. This guide explores how Real-Time Mode works, its benefits, and how to implement it.
Databricks offers robust mechanisms for sharing views across platforms, enhancing collaboration while maintaining security. Understanding these opportunities and challenges is crucial for effective data sharing.
Azure Data Factory's new Change Data Capture (CDC) resource simplifies real-time data integration by automatically detecting changes in source data. This feature enables efficient processing of large datasets, reducing the need for full data reloads.
The April 2026 update to Microsoft Fabric introduces enhanced notebook resiliency, improving the robustness of data engineering and data science workflows. This feature ensures that notebooks can handle unexpected disruptions, providing a more reliable experience for users.