Data Validation and Independent Benchmark Protocol
No independent accuracy benchmark results are published here. This protocol defines how to evaluate Acren's record matching with independent labels, reproducible samples, disclosed failures and uncertainty.
Documentation updated . This date does not identify a completed benchmark.
Current evidence and its limits
Internal engineering tests are not an independent accuracy benchmark. Tests of authorization, query counts, deterministic replay, field handling and failure states establish software behavior under tested conditions. They do not estimate how often a real-world ownership or entity link is correct across the covered population.
The repository includes workflows for owner-resolution samples, reviewer evidence, adjudication and versioned metric publication. The existence of those workflows is not evidence of completed independent review. No accuracy percentage, universal confidence threshold or independent certification is claimed on this page.
The protocol below is a proposed standard for a reproducible independent evaluation. It is not a report of completed sampling, reviewer agreement, precision, recall or coverage.
Freeze the claim and sampling frame
- Pre-register the question, evaluation period, population, unit of analysis, eligibility rules and metrics before inspecting outcomes. Evaluate a recorded-owner observation, a registration link, a cross-property entity link, a mortgage match and a sale-record match as different tasks.
- Freeze a dated sampling frame with source dataset IDs, property and entity IDs, jurisdiction, source dates, resolver version, query parameters, code revision and content hashes. Define exactly what constitutes the same property, the same registration and a duplicate.
- Use stratified random sampling across states and counties, property types, source ages, coverage gaps and predicted resolution statuses. Include confirmed, review, not_found and not_eligible records; do not sample only successful links. Treat not_evaluated records separately when measuring operational coverage.
- Publish the random seed, sampling code, stratum population counts, inclusion probabilities and selected sample IDs where rights permit. Oversampling difficult cases is useful, but population estimates need the corresponding sampling weights.
- Choose and disclose sample sizes from a stated uncertainty target and review budget, not a handpicked success set. Separate a diagnostic set from a locked holdout, and keep the same owner or connected entity cluster from leaking across development and evaluation sets.
Create independent reference labels
Reviewers should be independent of the matching implementation and disclose organizational and financial relationships. Have two reviewers examine the underlying records without seeing the model's score or decision. Record the source document identifier or URL, jurisdiction, source date, retrieval date, reasoning and reviewer agreement. Adjudicate disagreements with a documented tie-breaking review.
Reference labels should distinguish match, ambiguous, no match and not eligible. Keep inaccessible, missing or unreviewable evidence as an explicit outcome with a reason. A matching company name is not sufficient ground truth. A company registration does not establish beneficial ownership, current title or seller intent.
Recall requires independently finding eligible true links, including links the system did not propose. Reviewing only returned candidates can measure precision on those candidates, but cannot establish recall.
Metrics and denominators
| Metric | Definition | Required disclosure |
|---|---|---|
| Link precision | TP / (TP + FP): correct predicted links divided by all evaluated predicted links. | Publish true positives (TP), false positives (FP), the decision rule and the exact link type. A zero denominator is undefined, not perfect accuracy. |
| Link recall | TP / (TP + FN): recovered true links divided by all independently established eligible true links. | Publish false negatives (FN), the reference-search procedure and any coverage limits. Do not substitute the confirmation rate for recall. |
| False-positive entity matches | Wrong predicted entity links, counted separately from unresolved candidates. | Report false-link counts and FP / (TP + FP). If reporting a false-positive rate FP / (FP + TN), define and sample the eligible non-link population (TN). |
| Duplicate-property rate | Redundant records beyond one per adjudicated property, divided by all evaluated records. | For group size k, count k - 1 redundant records. Publish the grouping rule, dataset boundaries and whether parcel, building or property is the unit. |
| Source coverage | Eligible reference records represented in the evaluated dataset, divided by eligible records in the independent source frame. | Report jurisdiction, source dates, missing fields and unresolved records. A loaded record count is not a claim of complete county coverage. |
| Mortgage and sale-record matching | Evaluate each claimed relationship with its own precision and recall counts. | Define property and instrument identifiers, date tolerances and amendment/duplicate rules. Do not infer debt-match accuracy from owner-resolution results. |
Uncertainty, calibration and failures
Publish 95% confidence intervals with the method and assumptions. Wilson intervals are an option for unweighted binomial proportions from independent observations. Stratified estimates and repeated records for one owner need design-aware uncertainty, such as a stratified cluster bootstrap with disclosed weights, resampling unit, seed and replicate count. Do not apply an independent-record interval to dependent entity links.
Report each stratum as well as any weighted overall result. Disclose small samples, zero denominators, missing reference evidence and unreviewable records. Show their counts and sensitivity to exclusions; do not silently remove failures to improve the result. Track data freshness and missingness separately from matching accuracy.
A resolver score is not automatically a probability. Any calibration claim needs held-out observed outcomes grouped by score range, sample counts and uncertainty. Choosing thresholds after examining the holdout invalidates it as an independent final test.
Reproducibility and publication checklist
| Artifact | Required contents | Boundary |
|---|---|---|
| Frozen inputs and versions | Sampling frame, selected IDs, dataset snapshots, source dates, resolver version, code revision and hashes. | A methodology page date must not stand in for the version or freshness of evaluated records. |
| Labels and evidence | Reference outcomes, review instructions, source citations, disagreement and adjudication records, exclusions and reasons. | Redact or restrict source material as required by rights and privacy; disclose what independent reviewers cannot reproduce. |
| Executable analysis | Environment and dependency versions, sampling seed, metric code, weights, interval method and commands to regenerate tables. | Release aggregate counts and machine-readable results where permitted. A screenshot of a percentage is insufficient. |
| Results and failures | All pre-registered metrics, denominators, confidence intervals, stratum results, unresolved cases and known failure modes. | Publish adverse and inconclusive findings alongside successes. Separate benchmark results from internal regression-test results. |
| Independent review and changes | Evaluator identity and conflicts, review scope, review date, corrections and a versioned change record. | Publish these only after review occurs. A material source or resolver change requires a new evaluation; earlier results do not certify later versions. |
See a source-linked property review.
Review the documented research trail before choosing a next step.
Search is free. Request access for the full research workspace, or open the sample property review.