Skip to content
Antoine Le LouëtIndependent senior developer (Home)
Back to the Journal

Workshop15 min read

Identity matching: finding and deduplicating customer records

How to find a customer despite different names or contact details, assess similarities and merge duplicates without losing business history.

Updated on

Two fictional records, Jean Marie Yves MARTIN and Jean MARTIN, go through a multi-field comparison and then validation before any merge.
Resemblance identifies a candidate. Duplicate validation and merge rules then determine what can be combined.

A customer signs up again, but their old contract cannot be found

A company has hundreds of thousands, or even millions, of customer records. A former customer returns and creates an account with a different email address, an abbreviated given name or a name they now use. Registration succeeds, but their old contract remains attached to another record. Customer support sees two files and must reconstruct their connection.

Identity matching looks for records that may refer to the same person even when their information does not match exactly. It combines similarities across several fields, such as surname, given names, date and place of birth or contact details, to propose candidates for verification. This is also known as record linkage.

I developed a user-matching algorithm in Java at Crédit Logement, from prototype to industrialisation, while employed by Coreoz. This subject affects many other customer databases. Here I present the difficulties to address and an approach to tackling them. The examples are fictional; the rules and tools described are not a description of Crédit Logement’s internal implementation.

Why surname, given name and email are not enough

An initial rule is to look for equality across several raw fields: surname, given name, email, phone number, date of birth or town of birth. This is a useful foundation when data is identical. But requiring all fields to match also excludes customers whose information has changed or been entered differently.

  • Birth name and name in use: a customer may have signed up under her birth name, then returned with a married name. Comparing only the current name misses the link to the old record.
  • Multiple given names: Jean Marie Yves MARTIN may appear as Jean MARTIN or Marie MARTIN. A given name present in a list is a clue to cross-check against other fields.
  • Compound names: Jean-Marie, Jean Marie and Jean–Marie may differ in spacing or the dash character used. Their similarity deserves examination without losing the original form.
  • Accents and case: Élodie LE LOUËT and Elodie Le Louet may represent the same entry in different forms.
  • Spelling errors: MARTIN and MARTNI may result from transposed letters. Two similar names can also belong to different people.
  • Multiple surnames: someone registered as GARCIA MARTIN may use only one surname when registering again. Systematically reducing every surname to its first element loses that information.

An email address can change, and a phone number may be shared within a household or reassigned. Conversely, two people with the same name may share a birth date and town. An exact match provides evidence whose strength depends on context, reliability and age.

Normalise values while preserving the originals

Preparation preserves the received value and builds comparison forms alongside it. It harmonises case, spaces and some punctuation, then produces an unaccented variant. Displayed names and reference data retain their spelling. The W3C highlights how much name structures vary across people and cultures: one universal splitting rule risks introducing its own errors.

For multiple given names, comparing both the complete sequence and its elements identifies partial overlap. Preserve the distinction between a compound given name and multiple given names when the source supplies it. The same principle applies to birth names, names in use and known former names. Matching two completely different names requires information linking them or other consistent evidence; removing accents does not solve that case.

Other fields need their own rules. A phone number must be interpreted with its country; a date needs an unambiguous format. A town of birth can be matched against a reference dataset including its country, if available. For email, avoid globally removing dots or suffixes: their meaning depends on the provider. A missing value must remain distinct from a different value, and two empty fields provide no evidence of identity.

How do you build a reliable comparison dataset?

A comparison dataset is a set of record pairs whose relationship has been examined. Each pair connects the compared data to a reference decision, with its reason, source and validation date. This reference lets you verify the algorithm’s results. An earlier automatic match does not become ground truth without checking.

The selection covers the sources and flows actually feeding the database: recent registrations, old contracts, imports or customer support entries. It includes common names, multiple surnames and given names, name changes, mistakes and missing fields. The data used remains limited to information needed for comparison, in an environment with controlled access.

  • Same person: confirmed duplicates, including cases with different names or contact details. They check whether matching recovers the expected variants.
  • Different people: easy-to-distinguish pairs, but also people sharing a name or household members sharing contact details. These close cases challenge false positives.
  • Undecidable: pairs where available evidence does not support a conclusion. They remain identified as such and are not forced into one of the other classes to calculate a success rate.

Business validation relies on available evidence without treating the algorithm’s score as proof. Annotation guidelines specify what justifies each decision and how to resolve disagreements. A second review of ambiguous cases helps identify differing interpretations. Splink’s documentation describes evaluation using labelled pairs and notes that measurement representativeness depends on sample representativeness.

Synthetic variants complement confirmed cases: removing an accent, transposing letters or dropping a given name. Their origin remains traceable. They check specific behaviours but do not, on their own, reproduce the diversity of production errors. Difficult cases may also be overrepresented in a regression set; its results stay separate from estimates obtained on a sample representative of real traffic.

The tuning set is used to adjust rules and thresholds. An evaluation set is reserved for checking results. When the reference identity is known, all records and variants of one person stay in a single set, then pairs are formed within each set. This avoids having the same identity on both sides and overestimating the ability to handle new cases. Scikit-learn’s guidance on groups and data leakage explains this principle, which also applies to manually adjusted rules.

The dataset version accompanies the rules version: added cases, corrected annotations and set allocation remain traceable. A case added after a production error enriches the regression set. Once an evaluation set is used to correct the rules, it no longer provides an independent check of those corrections; a new reserved set lets you evaluate them.

Varied pairs are annotated by business reviewers, preserving an undecidable state. Reference identities are divided between a tuning set and a reserved evaluation set, without placing one person’s variants in both.
The diagram shows preparation of the comparison dataset. Difficult cases enrich the tests; measurements on a representative sample remain separately identified.

How do you calculate a similarity score across several fields?

The calculation compares fields separately before combining evidence. For a surname, it can distinguish complete equality, equality after normalisation, partial overlap and spelling similarity. A partially recovered given name provides another piece of information. An identical birth date may strengthen a candidate, whereas a different, verified date is a contradiction to examine.

Trigrams, three-character fragments, are one option for measuring text similarity. PostgreSQL offers this approach in pg_trgm, with indexes for similarity searches. Such tools help retrieve similar spellings; they are insufficient to identify a person or handle name changes on their own.

The combination can use weighted rules or a probabilistic model. Agreement on a very common value is generally less discriminating than agreement on a rare one. Related fields also require care: an email based on the surname does not necessarily provide independent evidence. Splink documents an approach where each comparison level contributes a weight for or against a match.

A similarity score of 0.92 does not automatically mean a 92% chance that the records refer to the same person. A model’s probability must also be evaluated on the relevant data. Weights and thresholds are chosen using validated cases, observed errors and the business cost of a wrong decision. A percentage displayed without this explanation gives misleading confidence.

An example: does Jean MARTIN match the old record?

Take a fictional situation. Record A contains Jean Marie Yves MARTIN, a date and town of birth, and an old email. New account B contains Jean MARTIN, the same birth information and a different email. The system finds the surname and one given name; the birth information strengthens the candidacy. The email change may be consistent with the time elapsed.

Record C is also Jean MARTIN, but its verified birth date differs. Comparing only surname and given name may favour C because its given name matches exactly. Multi-field comparison exposes the contradiction. If B’s date is missing, that absence has neither the strength of an agreement nor that of a disagreement.

A useful result explains the proposal: identical surname, partial given name, matching birth information, different contact details. If several candidates remain plausible, the system preserves that ambiguity. Mechanically picking the first sorted result removes information necessary for the decision.

In a fictional example, account B Jean MARTIN is compared with A Jean Marie Yves MARTIN, whose birth information matches, and C Jean MARTIN, whose birth information differs. A remains a candidate to verify and C has a contradiction to examine.
Fictional example: record C’s exact given name does not compensate for different birth information. Record A still needs verification before linking.

How do you search a database of several million customers?

Comparing every pair in a database of one million records means 499,999,500,000 comparisons, even before calculating similarities. For a new registration, scanning the entire database on every request poses another cost and latency problem. First select a candidate set, then apply detailed comparison.

This selection, often called blocking, can combine several paths: identical contact details, a birth date combined with part of a name, or indexed search on name variants with another field. Combining results from multiple paths avoids making the entire search depend on one potentially incorrect field. Splink’s documentation explains this trade-off between comparison volume and the ability to recover true links.

A candidate excluded at this stage never reaches score calculation. Candidate selection and ranking must therefore be evaluated separately, paying attention to very large blocks formed by common names. Preparing normalised forms and using suitable indexes can reduce work; response time must be measured on the actual database, infrastructure and volumes.

The database is queried through several alternative paths: identical contact details, birth information plus a name element, or a similar name combined with another field. Found candidates are combined without repetition, then their fields are compared to rank them.
Conceptual diagram: selection paths complement each other. A candidate missing from their union escapes scoring even when its data is highly similar.

Finding an old contract does not grant access to it

The search produces a matching hypothesis. During registration, linking a contract and granting access to its documents require appropriate verification. Someone who knows another person’s name and birth date must not be able to recover their file through a high score. Identity checks and access rights are separate steps.

Depending on context, the flow directs the user towards account recovery, verification through an established channel or intervention by authorised customer support. Information visible before verification also needs care: displaying candidate details exposes personal data. The level of checking depends on the information and operations accessible after linking.

Deduplicating the existing database needs a dedicated process

Matching at registration limits the creation of new duplicates. Records already accumulated still need processing. Historical deduplication reuses data preparation and candidate searches, this time across the full stock. It produces proposals to examine before changing any accounts.

Duplicates may form groups of several records. Beware of transitivity: A resembling B and B resembling C does not prove that A and C refer to the same person. A shared household email may bridge two different people. Each group needs checks of agreements and contradictions before being treated as one identity.

In a fictional example, A and B share an email, while B and C share a phone number. These pairwise agreements do not prove that all three records refer to one person. The link between A and C and contradictions within the group still need checking before a merge.
Fictional example: B connects two pairs through shared contact details. The chain of links alone does not make the group a confirmed identity.

An initial pass without merging produces proposals, with the rules version, their reasons and an estimate of review workload. Processing resumes in batches and accounts for records changed between analysis and decision. Previously validated refusals to match are preserved, with an option to reconsider if new information appears.

Which account should be kept during deduplication?

Once a duplicate is confirmed, the reference account still needs choosing. Keeping the oldest preserves identifier continuity. Keeping the one with the most contracts may limit reference moves. Favouring the most complete preserves more fields, but their quantity does not prove accuracy. These criteria must be decided with business owners and the affected applications.

Choosing A as the retained account does not mean keeping every value from A. The rule must specify field by field what is retained: a recently confirmed address, a name from a reference source, a still-valid phone number. When two reliable pieces of information conflict, merging may require further verification. A missing value must not overwrite reliable information.

Some objects need specific rules. Consents or communication preferences cannot be combined simply by giving ‘yes’ priority. Login identifiers and authentication methods also need explicit treatment. A merge proposal must show these choices before execution, with the option to refuse or postpone it.

A comparison interface for deciding to merge A into B

When ambiguous cases are frequent, a business interface allows side-by-side comparison. The operator sees original data, differences and evidence that led to the match. For a group of several accounts, every member and contradiction must remain visible, without hiding a record behind pairwise comparisons.

  • Display identifiers, full surnames and given names, contact details and birth information useful for the decision, subject to the operator’s permissions.
  • Specify data sources and verification dates when available, along with linked contracts and histories.
  • Explain the score by field and flag disagreements, missing data and other possible candidates.
  • Offer ‘Different people’, ‘To verify’ and duplicate confirmation, followed by the choice of retained account and values.
  • Preview the result: merging A into B keeps B as the reference and redirects A’s links under the agreed rules; merging B into A keeps A.

The operator can understand the proposed decision without mentally recalculating every rule. A reason, author and validation date make the origin of the merge traceable later. For a sensitive operation, a second approval may be justified by its impact. A record can still be deferred when available evidence is insufficient to decide.

The merge must remain consistent across the information system

A customer identifier may be used by CRM, contracts, billing, support, exports or partners. Deleting a database row without handling these references may leave orphaned objects and systems still writing under the old identifier. Inventory dependencies and define how old identifiers resolve to the retained account.

Merging customer records does not imply merging contracts: one person can legitimately hold several. Documents, payments and historical events must retain their provenance. A mapping between old identifiers and the reference identifier can help preserve these links, provided consuming applications follow the same rule.

Example organisation of a merge from A into B: B remains the reference account, a mapping resolves the old identifier A to B, values are selected field by field and references are redirected in applications. Contracts remain separate objects.
Example organisation after confirming a duplicate: keeping B requires handling old identifiers, retained values and references throughout the information system. Merging records does not merge their contracts.

Within one database, a transaction can group compatible changes. Across several applications, you need to organise propagation, retries after failure and consistency checks. A retried operation must not move a contract twice or reapply a completed merge. A concurrent change must trigger another check rather than silently overwrite the reviewed record.

Define how to correct an erroneous merge before the first batch. Preserving previous values and moved links helps prepare that correction. Rollback can become complex after new operations or external exchanges; a backup alone does not guarantee automatic separation of accounts. Test the procedure against the dependencies actually involved.

How do you measure false positives and missed duplicates?

A false positive links two different people; a false negative leaves two records of the same person separate. The ONS recommends assessing quality through precision and recall. Precision measures the proportion of true links among proposed ones; recall measures the proportion of true links found. The number of matches alone says nothing about their correctness.

Examining only the highest scores overestimates quality. You must also sample rejected results and cases missing from candidate selection; otherwise some missed duplicates remain invisible. Without exhaustive knowledge of true links, recall remains an estimate over a scope that must be documented. Track errors and review workload by case type and data source.

Thresholds depend on the decision that follows. Suggesting a lead to an operator and automatically merging two accounts do not have the same error cost. An observation phase analyses proposals without changing data, before validating limited batches. Rules then evolve based on observed errors and a review capacity defined with the business team.

Where should a customer-matching project start?

Scoping starts from a specific flow, such as a customer registering again and failing to find their contract. The business team gathers known cases, identifies reliable information and defines the checks needed before linking. This first scope lets you discuss acceptable errors and a merge’s consequences using concrete records.

The first delivery can be a list of explained candidates, without automatic changes. It lets you check whether old records are found, how many proposals remain ambiguous and what an operator needs to see to decide. Before enabling merges, agree retention rules, verify references in the information system and try the procedure for correcting a bad merge.

Sources and useful links

Share on LinkedIn (new tab)

Keep reading
All articles in this category