What I build and learn at work: data platforms, Spark, Fabric and the tools around them.
The big picture: In 2012, the Harvard Business Review famously crowned data scientist as the "sexiest job of the 21st century." The logic was simple: a massive shortage of talent capable of making sense of big data. Then Generative AI exploded onto the scene. Today, the role isn't dying, but the traditional definition of a data scientist is officially extinct.
Anthropic’s recent pledge to put roughly $48 billion in equity into a public benefit fund, and its CEO's push for an AI-funded Universal Basic Income (UBI), aren't just Silicon Valley thought experiments. They are blaring alarm bells. Generative AI is rapidly obliterating the exact jobs Sri Lanka's developing economy relies on for survival.
The discourse surrounding Sri Lanka’s Information Technology and Business Process Management (IT/BPM) sector frequently oscillates between two extremes: overly optimistic promotional narratives of a "$5 Billion Digital Dawn" and pessimistic warnings about the devastating impact of skilled talent migration. To understand the true trajectory of this sector in 2026, we must look past the anecdotes and analyze the raw data.
If you've worked in data long enough, you know the quiet pain. Your warehouse is full of tables called factsales, dimstore, and dimproducts. Your business, meanwhile, talks about customers, shipments, and cold-chain breaches. Every time someone asks a "simple" question, a domain expert has to sit down and translate it into joins, filters, and column names. That translation tax is the thing slowing everyone down — and it's exactly the thing Fabric IQ is built to remove.
If you have spent any real time running Spark in production, you know the pattern. A job that usually finishes in eight minutes is suddenly stuck at "199/200 tasks completed" for forty-five minutes.
Most data teams hit the same wall sooner or later. You have the same real-world thing — a customer, a company, a patient — sitting in three or four systems, and not one of them agrees on how to identify it. The CRM calls it ACME Corp with id 1042. The billing system has Acme Corporation Ltd with id BX-77. A spreadsheet someone imported last year says acme corp. and has no id worth trusting. You need a single master record, and you need a way to point every transaction back to it.
AWS reskilling programs offer a structured path for individuals seeking to transition into data engineering. By leveraging these resources, you can gain the necessary skills to thrive in this evolving field.
Materialized lake views in Microsoft Fabric offer a powerful way to optimize SQL workloads by precomputing complex transformations. This article explores how to implement these views, their benefits, and considerations for production environments.
Delta Lake 4.0 introduces Delta Kernel, a pivotal step towards seamless multi-engine interoperability. This article explores how Delta Kernel simplifies connector development, enabling diverse data engines to interact with Delta Lake tables efficiently.
Spark 4.1 introduces Real-Time Mode, offering low-latency processing for micro-batch streaming workflows. This article explores how to implement and optimize these workflows, addressing common pitfalls and operational challenges.
Lakewatch by Databricks redefines security management by integrating AI-driven threat detection with a unified data platform. This article explores its architecture, operational benefits, and practical implementation challenges.
Keeping your on-premises data gateway updated is crucial for optimizing Azure Data Factory pipelines. This article explores the implications of gateway updates on performance, security, and operational efficiency.
Microsoft Fabric Data Factory offers robust support for JSON file formats, enabling seamless data integration across various sources. This article explores the practical implementation of JSON handling in Fabric Data Factory, including operational considerations and common pitfalls.
Apache Airflow 3 introduces the Task Execution Interface, a pivotal feature that enhances workflow optimization. This article explores its impact on task management, security, and deployment across diverse environments.
Explore how Jinja2 templating can be integrated with Snowflake's EXECUTE IMMEDIATE to dynamically construct and execute SQL statements. This approach enhances flexibility in SQL execution, particularly in environments where query parameters are determined at runtime.
Model Context Protocol (MCP) offers a standardized way to integrate multiple data engines on Google Cloud Platform, enabling seamless communication and data sharing between AI applications and databases. This article explores the practical implementation of MCP for multi-engine data integration, highlighting its benefits and challenges.
Explore the enhanced change data capture strategies in AWS DMS 3.5.1, focusing on operational improvements and practical implementation guidance for real-world data migration scenarios.
Explore how integrating Dagster+ with Terraform can streamline the deployment of data workflows by using infrastructure as code. Learn about the benefits, challenges, and practical implementation of this integration.
Hybrid quantum computing is reshaping the landscape of computational careers. With Amazon Braket SDK, professionals can explore new opportunities in quantum-classical algorithm development, blending traditional computing with cutting-edge quantum technologies.
The Modern Get Data experience in SQL Server Analysis Services streamlines data import by leveraging Power Query. This approach simplifies connecting, filtering, and transforming data, making it more efficient for building robust tabular models.