Duplicate document detection is the process of spotting when the same file, or a near copy of it, enters a mortgage file more than once, so a broker reviews one clean version instead of five. In a mortgage workflow, it works in two layers: an exact check (is this the identical file?) and a content check (is this the same pay stub or notice of assessment, rescanned or renamed?). Done well, it cuts review time, keeps the lender package tidy, and leaves a record of what was kept and why.
This guide explains how detection works, where it fails, how to fit it into intake, and what Canadian privacy and retention rules mean for the copies you remove.
TL;DR
- Exact duplicates are caught by comparing file fingerprints (hashes). Near duplicates, such as rescans, need comparison of the extracted content and key fields.
- No method is perfect. Scans, revised statements and multi-page documents produce both false matches and missed duplicates, so a person confirms before anything is removed.
- Flag first, archive second. Keep an audit trail of what was merged, who approved it and when.
- Retention rules still apply to what you keep. Ontario brokerages, for example, must keep mortgage transaction records for at least six years.
- Detection works best when it sits at intake, right after classification and data extraction.
Table of Contents
- What duplicate document detection means in mortgage processing
- How systems detect exact and near duplicates
- Where detection goes wrong
- Fitting detection into intake and review
- Reviewing flagged pairs and keeping an audit trail
- Privacy and retention: what happens to the copies
- Our perspective
- Try it on your own files
- FAQ
- Sources
What duplicate document detection means in mortgage processing
A typical Canadian mortgage file collects documents from many hands. A client emails a pay stub, then uploads it again to a portal, then a lender asks for it and the client sends a photo from their phone. An assistant saves the bank statement under two names. A revised appraisal arrives with the same file name as the original.
None of this is unusual. The cost shows up later:
- The underwriter opens three versions of the same T4 and wonders which one counts.
- Income gets double counted when two copies of a pay stub are keyed in as separate periods.
- A lender package goes out with a stale version of a document.
- The file becomes harder to audit because you cannot tell which copy supported which decision.
Duplicate detection answers a simple question at the moment a document arrives: have we already seen this? A second question follows: is it the same document, or a new version we need to treat as an update?
How systems detect exact and near duplicates
Most approaches combine a few techniques, from cheapest to most sophisticated.
Exact matching with file hashes
A hash is a short fingerprint calculated from a file's bytes. Two identical files produce the same fingerprint, so checking is instant and almost never wrong. The weakness is that any change at all, even a different PDF export setting or a single re-saved page, gives a different hash. A hash catches the client who uploaded the same file twice. It does not catch the client who rescanned it.
Near-duplicate matching on text
To catch the same content in a different file, systems compare the text itself. Published approaches in the information retrieval field break text into overlapping chunks (often called shingles) and estimate how similar two documents are from the overlap, using techniques such as MinHash and SimHash. You do not need to understand the maths to use the result: the system produces a similarity score, and you choose the score above which a pair is flagged for review.
That works well for text-heavy documents such as purchase agreements or employment letters. It works less well for the short, form-like documents that dominate a mortgage file, where two different pay stubs from the same employer share most of their words.
Field-level matching with extracted data
For mortgage documents, the most reliable signal is often structured data pulled out of the document. If two files both show the same employer, the same pay period end date, the same gross amount and the same employee name, they are almost certainly the same pay stub, however different the files look. If the pay period differs, they are two different documents that happen to look alike.
This is why detection pairs naturally with classification and extraction. Once a system knows a file is a bank statement and has read the account number, statement period and closing balance, comparing records is far more dependable than comparing pixels. If you want the background on that step, see our guide to mortgage OCR for Canadian brokers.
A practical comparison
| Method | Catches | Misses | Best used for |
|---|---|---|---|
| File hash | Identical files uploaded twice | Rescans, re-exports, renamed edits (renamed identical files are still caught) | First pass at intake |
| Text similarity | Same content in a different file format or export | Short form-like documents, poor scans | Contracts, letters, agreements |
| Field-level match | Same document with different appearance | Documents where extraction failed | Pay stubs, statements, T4s, notices of assessment |
| Human review | Context, intent, revisions | Nothing at volume, but slow | Every flagged pair before removal |
Where detection goes wrong
Brokers should expect two kinds of error and plan for both.
False matches happen when two different documents look the same. Two consecutive biweekly pay stubs share the employer, layout and often the gross amount. Two monthly statements from the same account share nearly every word except the dates and balances. A system that is too eager will flag these as duplicates, and removing one would leave the file short of the income evidence a lender needs.
Missed duplicates happen when the same document looks different to the system. Common causes:
- Scans and phone photos. Skew, shadows, cropping and low resolution change the text a system reads, so a rescan may score as only a partial match.
- Revised files. A corrected employment letter or an updated statement is not a duplicate, but it is not new either. It is a version, and it should supersede the earlier one with a record of the change.
- Multi-page documents. A 12-page bank statement uploaded as one PDF and again as twelve separate images has the same content, but no file-level comparison will connect them. Page-level comparison is needed.
- Partial uploads. A client sends pages 1 to 3 first, then the full package. The first is a subset of the second.
- Extraction errors. If a number is misread, a field-level match fails even though the documents are identical.
The lesson is not to distrust detection. It is to treat its output as a recommendation with a confidence level. Systems that show how sure they are about each extracted value, down to the line, make it far easier to decide which flagged pairs need a human look and which can be cleared quickly.
Fitting detection into intake and review
The best place for detection is right after a document arrives and has been classified, before anyone keys data from it. A workable sequence looks like this:
- Receive and log. Record the source (client email, portal, lender request), the date and the file name as received.
- Check for an exact match. If the fingerprint matches a file already in the deal, link the new file to the existing one rather than storing it again.
- Classify and extract. Identify the document type and read the key fields.
- Compare against the same deal. Look for matches on document type plus identifying fields: borrower, institution, period, amount.
- Route by outcome. Clear duplicates go to a merge queue. Possible duplicates go to a reviewer. Revisions are marked as new versions.
- Confirm and record. A person approves the merge, and the system logs who, when and why.
- Proceed to underwriting forms and packages. Only the approved version feeds the forms and the lender package.
This fits a broader intake routine. Our mortgage document checklist and intake workflow covers what to request up front, and five steps from lender requests to retention shows how the file should look by the end.
Scope the comparison carefully
Compare within a deal first. A client's T4 from a prior application is not a duplicate of this year's, and matching across clients would raise privacy concerns of its own. Cross-deal checks, if you use them at all, should be deliberate and limited to what your brokerage's privacy policy allows.
Set thresholds by document type
One similarity threshold for everything is a mistake. A government-issued notice of assessment has a fixed layout, so near-identical text may truly mean a duplicate. A pay stub is the opposite: near-identical text is expected, and the pay period is what matters. Set stricter rules for recurring documents and looser ones for one-off agreements.
Reviewing flagged pairs and keeping an audit trail
A reviewer looking at a flagged pair should be able to answer four questions in under a minute:
- Are these the same document, a revision, or two different periods?
- Which version is the most complete and legible?
- Does anything differ, even slightly, in the amounts, dates or names?
- Does the lender require a specific format or a certified copy?
Then record the decision. A useful log entry holds the two file references, the match type (exact, content, field-level), the confidence or score, the decision (merged, kept both, marked as a revision), the reviewer and the timestamp. Never silently delete the loser. Archive it, linked to the retained document, so you can reconstruct the file if a lender or regulator asks how a figure was reached.
This is the same discipline that makes a file defensible generally. For what examiners tend to look for, see our notes on audit ready digital compliance records and, for Ontario, the proofs brokers must keep on file.
A related point: duplicates are not always innocent. If a document that looks like a copy of a prior one has altered figures, the match is a signal worth escalating rather than merging. Our piece on procurement questions about mortgage document tampering covers that side.
Privacy and retention: what happens to the copies
Removing duplicates is housekeeping, but the copies you keep and the ones you discard are both personal information, so the rules still apply.
Safeguards match sensitivity. The Office of the Privacy Commissioner of Canada's audit of selected mortgage brokers found that brokers should use safeguards proportionate to the sensitivity of the information. Mortgage files hold SINs, income and bank details, which sit at the high end. The same audit recommends retention and secure destruction policies for unapproved mortgage files, and a clear process for destroying unneeded documents within a reasonable time. A pile of uncontrolled duplicate copies in inboxes, downloads folders and shared drives works against all of that, because every extra copy is another place the data can leak.
Retention differs by province. In Ontario, O. Reg. 188/08 requires mortgage brokerages to keep records related to mortgage transactions for at least six years after the transaction's expiry, and records in electronic form must be stored so they can be retrieved promptly in an understandable form. The practical consequence: when you merge duplicates, make sure the retained copy, and the log explaining the merge, stay retrievable for the full period. Our article on FINTRAC's five-year and FSRA's six-year retention periods compares the two clocks.
Quebec has its own framework. The Commission d'accès à l'information du Québec explains that personal information agents must destroy information collected more than seven years earlier, with a narrow exception for investigation files. That rule is aimed at credit files, not brokerages as such, so Quebec brokers should confirm what applies to their own records with their regulator or counsel. British Columbia and Alberta brokers should likewise check provincial requirements and their brokerage's policies.
For a wider look at your obligations, see PIPEDA checklists for Canadian mortgage brokers.
What to assess before you adopt detection
The OPC audit points to a good checklist for any brokerage introducing this step:
- Can you meet your provincial retention period for the version you keep and the log you create?
- Are both physical and digital records stored securely?
- Is there a policy for timely destruction of documents you no longer need?
- Can you still run a diligent search for a client's personal information if they make an access request? Fewer duplicates make this faster and the answer more reliable.
- Where is the data processed and stored? Canadian data residency is a reasonable question to put to any vendor.
Our perspective
We built Autowrite around one belief: a broker's time belongs with clients, not with deciding which of four pay stubs is the real one.
Duplicates are a symptom of a bigger problem. Documents arrive through email, portals and lender requests with no common structure, and every one of them has to be identified, read and checked by hand. When a platform classifies each document and extracts its data with line-level confidence, the duplicate question gets easier, because you are comparing what a document says, not just what it looks like.
That is why we put intake, classification, extraction, underwriting form fill, compliance packages and integrations with Canadian mortgage software in one purpose-built system for Canadian brokers. One flow means fewer copies floating around, and a clearer record of which version fed which decision. Canadian data residency and privacy compliance are part of the design, not an add-on.
Our advice is simple: automate the matching, keep a person on the final call, and write the decision down.
Anant Bawa
Try it on your own files
If your team still sorts uploads by hand, start with a 14-day free trial and run a live deal through intake, classification and data extraction. Plans start at $99 per month for Starter, with Pro at $199 per month and Legend at $399 per month. Additional deals are $20 each, and pay as you go is $35 per deal. Enterprise is available on request. See the Autowrite plans and pricing for what each tier includes.
FAQ
Is a duplicate the same as a new version of a document?
No. A duplicate carries the same content as one you already hold. A new version changes something, such as a corrected amount or an updated statement period. Treat versions as replacements that you log, and keep the earlier one archived so the history is clear.
Can software detect a duplicate if the client rescans a document?
Often, but not always. A file hash will not match a rescan, so the system has to compare extracted text or fields. Poor scan quality can reduce accuracy, so a reviewer should confirm borderline matches before one copy is merged away.
Should I delete duplicates once I find them?
Archive and link them rather than deleting straight away, at least until the file closes and your retention policy says otherwise. Deletion should follow your brokerage's destruction policy and the retention period that applies in your province, such as six years in Ontario.
Does duplicate detection create privacy risk?
It can reduce it, since fewer stray copies mean fewer places for personal information to sit. Keep the comparison within each client's deal, store results in Canada if your policies require it, and limit access to the review queue to people who need it.
Sources
- Audit of Selected Mortgage Brokers (2010), Office of the Privacy Commissioner of Canada
- O. Reg. 188/08 Mortgage Brokerages: Standards of Practice, Ontario e-Laws
- Dossier de crédit, Commission d'accès à l'information du Québec