# Matching messy records without buying an MDM tool: a practical guide to Splink

> Most data teams hit the same wall sooner or later. You have the same real-world thing — a customer, a company, a patient — sitting in three or four systems, and not one of them agrees on how to identify it. The CRM calls it ACME Corp with id 1042. The billing system has Acme Corporation Ltd with id BX-77. A spreadsheet someone imported last year says acme corp. and has no id worth trusting. You need a single master record, and you need a way to point every transaction back to it.

**Category:** comparisons  
**Published:** 2026-05-21T08:44:28.200590Z  
**Canonical:** https://allaboutspark.com/posts/matching-messy-records-without-buying-an-mdm-tool-a-practical-guide-to-splink
**Tags:** Data Management, MDM, Master Data Management

---

Most data teams hit the same wall sooner or later. You have the same real-world thing, a customer, a company, a patient, sitting in three or four systems, and not one of them agrees on how to identify it. The CRM calls it `ACME Corp` with id `1042`. The billing system has `Acme Corporation Ltd` with id `BX-77`. A spreadsheet someone imported last year says `acme corp.` and has no id worth trusting. You need a single master record, and you need a way to point every transaction back to it.

The textbook answer is "buy a master data management platform." That works, but those tools are expensive, slow to roll out, and frankly overkill if your real problem is "I have a few million rows and a stubborn matching job." This article is about the alternative I keep coming back to: **Splink**, a free, open-source library that does the hard part, deciding which records are the same thing, surprisingly well, and at a scale most paid tools charge a fortune for.

I'll cover what it is, how it actually works (not just the marketing version), a worked example you can adapt, and the practical lessons that the documentation tends to gloss over.

## Why a normal SQL join doesn't save you

The instinct is to write `JOIN ON company_name`. It fails immediately, because `ACME Corp` and `Acme Corporation Ltd` are not equal strings. So you reach for fuzzy logic, maybe a `LIKE`, maybe a Levenshtein distance with a hand-picked cutoff. That gets you a bit further and then collapses under its own weight, for two reasons.

First, a single similarity score on a single column throws away information. Two companies with similar names but different cities and different tax numbers are probably *not* the same; two with slightly different names but an identical registration number almost certainly *are*. A flat name-distance rule can't weigh those signals against each other.

Second, the comparison count explodes. Comparing every record to every other record is an n² problem. A million records means roughly five hundred billion comparisons. No clever `WHERE` clause rescues you from that on its own.

Splink solves both problems with a statistical model and a technique called blocking. Here is the whole pipeline before we go deeper.

![image](https://allaboutspark.com/api/uploads/20260521-19881899443d4620.svg)

*The Splink pipeline. Steps 3–6 are what the library runs for you; steps 1–2 are preparation you own; step 7 hands you the keys to stitch everything back together.*

## What Splink actually is

Splink is a Python package built by data scientists at the UK Ministry of Justice to link very large administrative datasets  court records, prison records, that sort of thing  where no shared identifier exists. It is a Python package for probabilistic record linkage (entity resolution) that lets you deduplicate and link records from datasets that lack unique identifiers. It has been downloaded several million times and is used in production by organisations including NHS England and the UK's Office for National Statistics.

Three things make it a good fit for the company-master problem:

It runs on the engine you already have. You can execute linkage in Python using DuckDB on a single machine, or on big-data backends like Spark for 100-million-row-plus datasets. If you are working in Microsoft Fabric, Databricks, or anything Spark-based, Splink plugs straight into the session you already pay for.

It is genuinely fast. It can link a million records on a laptop in around a minute. That is not a typo, and it changes how you work, you can iterate on a model in seconds rather than waiting for an overnight batch.

And it needs no training data. Model training is unsupervised, so you do not need a labelled set of known matches to get started. This is the part people find hard to believe, so let me explain how it pulls that off.

## How it works, without the hand-waving

Splink is built on the **Fellegi–Sunter model**, a piece of statistics from 1969 that is still the backbone of serious record linkage. The idea is easier than the name suggests.

Take any pair of records and look at them field by field. For each field, ask two questions:

- If these two records *are* the same company, how likely is it that this field would agree? Call that the **m probability**. For a field like a tax number, m is very high true matches almost always share their tax number.
- If these two records are *not* the same company, how likely is it that this field would agree anyway, just by chance? Call that the **u probability**. For tax number, u is tiny random unrelated companies basically never share one. For a field like "country = United States," u is large, because loads of unrelated companies are American.

The ratio of those two numbers is the evidence the field provides. Splink turns it into a **match weight**, on a log2 scale, so the weights from each field can simply be added up. A field where agreement is strong evidence of a match (high m, low u, like a tax number) contributes a large positive weight. A field that agrees all the time anyway (like country) contributes almost nothing. And a field that *disagrees* pushes the weight down. Add the weights across all fields and you get a single score for the pair: strongly positive means "almost certainly the same," strongly negative means "almost certainly not."

The clever part is that Splink learns the m and u values from your data automatically, using an algorithm called **Expectation–Maximisation (EM)**. It starts with rough guesses, scores the pairs, uses those scores to re-estimate the probabilities, and repeats until the numbers stop moving. That loop is why you don't need labelled examples.

Two refinements matter in practice. **Comparison levels** mean a field isn't just "match or no match"  you can define an exact match, a close fuzzy match (say, a Jaro–Winkler similarity above 0.9), and everything else, each with its own weight. And **term-frequency adjustments** let the model know that two records both named "Smith Trading" is weaker evidence than two records both named "Zubrowka Holdings," because the first name is common and the second is rare.

That handles scoring. Blocking handles scale. Instead of comparing every record to every other, you give Splink **blocking rules**  cheap conditions that two records must share before they are even considered, such as "same city" or "same first three letters of the name." Records that share no block are never compared, which is what brings five hundred billion comparisons down to a few million. Crucially, blocking is only about which pairs get *scored*; the statistical model still decides whether each scored pair is actually a match.

The last step is **clustering**. Scoring produces pairs: A matches B, B matches C. Splink treats these as edges in a graph and groups connected records into clusters. Each cluster is one real-world entity, and its cluster id becomes your master key.

## A worked example: building a company master

Let's build the thing. I'll start with DuckDB because it runs anywhere with no setup, then show the change you make for Spark/Fabric. Install it first:

```bash
pip install splink
```

Assume you've already pulled your sources together and cleaned them (more on that in a moment) into one table with these columns: `unique_id`, `source_system`, `source_id`, `company_name`, `city`, `postcode`, `registration_no`. The `unique_id` must be unique across the whole combined table  a simple concatenation of source and source id works fine.

### Define the model

```python
import splink.comparison_library as cl
from splink import DuckDBAPI, Linker, SettingsCreator, block_on

db_api = DuckDBAPI()

settings = SettingsCreator(
    link_type="dedupe_only",
    comparisons=[
        cl.NameComparison("company_name"),
        cl.JaroWinklerAtThresholds("city", score_threshold_or_thresholds=[0.9]),
        cl.ExactMatch("postcode").configure(term_frequency_adjustments=True),
        cl.ExactMatch("registration_no"),
    ],
    blocking_rules_to_generate_predictions=[
        block_on("city"),
        block_on("postcode"),
        block_on("substr(company_name, 1, 4)"),
    ],
)
```

A few words on the choices. `link_type="dedupe_only"` tells Splink everything is in one table and any record can match any other  which is what you want when several sources are already stacked together. The comparisons describe how each field is scored: a purpose-built name comparator for the company name, a fuzzy comparator for city, and exact matches for the structured fields. The term-frequency adjustment on postcode means a shared rare postcode counts for more than a shared common one. The blocking rules say two records are only ever compared if they share a city, a postcode, or the first four letters of the name  so a genuine match will almost always survive at least one of those nets, while most junk pairs are discarded cheaply.

### Train it

```python
linker = Linker(your_dataframe, settings, db_api)

# Rough estimate of how rare matches are overall
linker.training.estimate_probability_two_random_records_match(
    [block_on("registration_no")], recall=0.7
)

# Learn the u values (agreement by chance) from random pairs
linker.training.estimate_u_using_random_sampling(max_pairs=1e6)

# Learn the m values (agreement among true matches) via EM
linker.training.estimate_parameters_using_expectation_maximisation(
    block_on("city")
)
linker.training.estimate_parameters_using_expectation_maximisation(
    block_on("postcode")
)
```

The pattern here is deliberate. The u values are learned by sampling random pairs, which are nearly all non-matches by definition. The m values are learned with EM, and you run it more than once, blocking on a different field each time, so that every field gets trained on pairs where it wasn't the thing being blocked on. After this, the model is fully trained and you can inspect it  `linker.visualisations.match_weights_chart()` shows you exactly how much each field is contributing, which is the first thing I check when results look off.

### Predict and cluster

```python
pairwise = linker.inference.predict(threshold_match_probability=0.9)

clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(
    pairwise, 0.95
)

df = clusters.as_pandas_dataframe()
```

`df` now has every input row with a `cluster_id` attached. That `cluster_id` is your master company id. To produce the cross-reference table that links back to your transactions, you keep exactly three columns:

```python
xref = df[["source_system", "source_id", "cluster_id"]]
```

Every transaction in every source still joins on its original `(source_system, source_id)`, and through this xref it rolls up to one master company. You never touch the source transaction tables.

### Running it on Microsoft Fabric or Spark

The model definition and the training/predict calls are identical. The only change is the backend: you install the Spark extras and hand Splink your existing Spark session instead of a DuckDB connection.

```bash
pip install 'splink[spark]'
```

```python
from splink import Linker, SettingsCreator, SparkAPI, block_on
import splink.comparison_library as cl

# In Fabric a `spark` session already exists in the notebook
spark_df = spark.createDataFrame(your_pandas_df)

linker = Linker(spark_df, settings, db_api=SparkAPI(spark_session=spark))
```

From there, `linker.training...`, `linker.inference.predict()` and the clustering call are the same as before, and results come back with `results.as_spark_dataframe()`. One Fabric-specific wrinkle: some of Splink's string-similarity functions are bundled as compiled Spark functions in a jar, and registering that jar in a managed environment can be fiddly. If a comparator complains, fall back to the comparators that don't need it (exact match and Levenshtein-based ones) while you sort out the jar, or test your model in DuckDB on a sample first and only move to Spark for the full volume.

## The guidance that actually matters

The code is the easy part. Here is where projects succeed or quietly go wrong.

**Clean before you link.** Splink is not a data-cleaning tool, and feeding it raw values is the most common reason for poor results. Do the boring work first: upper-case everything, trim whitespace, strip the legal suffixes (`LTD`, `INC`, `LLC`, `GMBH`, `PLC`) into a separate field rather than leaving them in the name, and standardise addresses. A model fed `Acme` and `ACME LIMITED` has to work much harder than one fed `ACME` and `ACME`.

**Find your strong identifiers and use them.** If any source carries a tax number, company registration number, or website domain, that field is worth more than the name and city combined. A reliable shared identifier gives you a high m and a near-zero u  the textbook definition of strong evidence. Check what you have before you spend a day tuning name-similarity thresholds you may barely need.

**Design blocking rules as a set, not a single rule.** No single blocking rule needs to be perfect; the set just needs to catch every true match in at least one rule. The trade-off is real: loose rules (block on city alone) generate huge numbers of pairs and slow everything down; tight rules (block on city *and* first letter) are fast but risk missing matches that disagree on the blocked field. Splink can tell you how many comparisons each rule will generate before you run anything, so use that to keep the total in a sane range  typically tens of millions, not billions.

**Choose the threshold deliberately, and keep a review band.** Don't treat the match score as a yes/no. Pick two thresholds: above the upper one, auto-accept as a match; below the lower one, auto-reject; in between, route to a human. That middle band is where the genuinely ambiguous cases live, and pretending they don't exist is how bad merges sneak into a master table. Splink's waterfall chart, which breaks a single pair's score down field by field, is the tool I hand to whoever does that review.

**Keep your master ids stable across runs.** This is the lesson people learn the hard way. Splink's `cluster_id` is not guaranteed to be the same value next time you run the pipeline. If you blindly overwrite your master id with each run's cluster id, every transaction link breaks. The fix is to persist your own master id table and, on each run, map new clusters back to existing master ids by their overlap with the previous run  only minting a brand-new master id for a genuinely new entity. Build this in from day one; retrofitting it is painful.

**Evaluate, don't just trust.** Even without a full labelled set, you can sanity-check. Hand-label a few hundred pairs, pull the cluster sizes and look for suspiciously large clusters (often a sign two blocking rules merged things they shouldn't have), and use Splink's cluster studio dashboard to eyeball real clusters. A model that looks great on aggregate metrics can still be quietly merging two different companies that happen to share a building.

## When Splink is the wrong tool

It is not magic, and a couple of situations defeat it. If your only field is a single free-text "company name" column with nothing else  no city, no id, no address  Splink has little to work with, because the whole approach depends on weighing several independent signals against each other; the authors say plainly it is not designed for linking a single bag-of-words column. Highly correlated fields also weaken it, since the model assumes each field adds independent evidence; if your "city" and "postcode" always agree, they are really one signal counted twice. And if you need a polished business-user interface for stewards to approve merges, with workflow and audit baked in, that genuinely is what the paid MDM platforms sell Splink gives you the matching engine, not the front office around it.

For the problem most teams actually have, though  several sources, the same entities under different ids and slightly different spellings, a transaction layer that needs a single key to hang off  Splink does the difficult ninety percent for nothing but the compute you already own. Clean your data, give it a few good signals, watch the thresholds, and keep your master keys stable. That is most of the battle.

[Source](https://moj-analytical-services.github.io/splink/)