I Asked Gemini to Read 300-Year-Old Portuguese Parish Records

Can you read this page?

This page was written in a Portuguese parish in 1693.

It holds three baptism records in running cursive, densely abbreviated.

The first child was baptized “aos 30 de agosto de 1693 annos” — on August 30, 1693.

The second, “aos 2 de 7.bro” — 7bre being the period abbreviation for September.

The third entry records only one parent, and the ink has faded to the edge of legibility.

In the left margin, the priest repeated each child’s name and hamlet — An.to / Vila, Mathias / Poutena — a seventeenth-century finding aid.

I ask language models to turn pages like this into structured genealogical data—not a full transcription, but the people and relationships needed to identify someone and connect them to a family.

For this page, the expected answer is:

[
  {
    "arquivo": "PT-ADAVR-PAND13-1-3_m00009.jpg",
    "assentos": [
      {
        "tipo": "batismo",
        "data": "1693-08-30",
        "local": "Bemposta",
        "nome_principal": "Maria",
        "nome_do_pai": "Domingos Jorge",
        "nome_da_mae": "Isabel Francisca"
      },
      {
        "tipo": "batismo",
        "data": "1693-09-02",
        "local": "Vila",
        "nome_principal": "Antonio",
        "nome_do_pai": "Antonio Francisco",
        "nome_da_mae": "Maria Francisca"
      },
      {
        "tipo": "batismo",
        "data": "1693-09-22",
        "local": "Poutena",
        "nome_principal": "Matias",
        "nome_do_pai": "Manoel José"
      }
    ]
  }
]

This JSON is the human reference: my reading of the page, record by record.

Three choices in this reference shape the evaluation.

First, the schema is in Portuguese—the language of the corpus and my native language.

Each assento (record) includes a tipo (type), date, place, nome_principal (the person the record concerns), and any parents named in the document.

Second, the reference records only what the page states. The third entry names a father but no mother, so the annotation preserves that absence rather than filling the gap.

Third, the task requires more than character recognition.

Abbreviations must be expanded (7.bro → September, M.a → Maria), historical spellings modernized, and place names recovered from cramped marginalia—in the third entry, from ink that is barely visible.

This image is one of 84 in the benchmark. The rest of this article examines what models get right, what they get wrong, and what changes when I let them investigate the image instead of answering in one shot.

Why This Extraction Matters

I started building my own family tree in 2022.

That hobby quickly led me to Portuguese parish records, to the slow, rewarding work of reading them by hand, and to the question behind this article: can current AI reliably extract the people mentioned in these documents?

For anyone researching Portuguese ancestry, parish records are the backbone source.

From the sixteenth century until civil registration took over in the early twentieth century, the local parish was where baptisms, marriages and deaths were written down.

Many of these books are now digitized and published online by Portuguese archives, including the two whose collections I use here: the Aveiro District Archive (ADAVR) and the University of Coimbra Archive (AUC).

But digitized pages are not searchable data (yet).

Finding an ancestor today usually means guessing which books might mention them and then reading, page after page of this pleasant calligraphy.

What an (amateur) genealogist actually needs from each record is relational:

An isolated name rarely solves a search; a name, plus parents, plus a place usually does.

That is why this benchmark measures the extraction of people and relationships, not full-text transcription quality.

The Benchmark Data

I built the reference set by reading 84 digitized images from 17 parish books — nine from ADAVR, eight from AUC — and manually extracting the same structured fields shown for the opening page: 434 records spanning 1601 to 1876.

I expect most of it to be correct given my few years of hobbyist experience reading and studying hundreds of these pages, but I don’t expect perfection.

The set was composed to be roughly balanced across the three record types and three centuries.

The 434 records split into 140 baptisms, 135 marriages and 159 deaths.

By century, 173 date from the seventeenth, 131 from the eighteenth and 123 from the nineteenth (some records don’t have dates in the visible page).

An image can have up to two pages and hold anywhere from 2 to 21 records — about 5 on average.

The balance lets me check whether older, harder-to-read records really score worse instead of letting one easy century dominate the average.

What The Model Is Asked To Do

For each benchmark item, the model receives one digitized image and must identify every parish record in reading order.

The instruction itself is short and is sent in Portuguese:

Extraia cada assento desta página seguindo o schema anexado

Normalize os nomes para versões modernas e abreviações para o nome completo. Por exemplo:
Joze - José
Joam - João
Magdalena - Madalena
Anna - Ana
Roiz - Rodrigues
Antônio - Antonio
Manuel - Manoel
Rosaria - Rosália
Baptista - Batista
Izabel - Isabel

[a list of valid place names for the parishes in the benchmark]

Preencha o campo arquivo com: PT-ADAVR-PAND13-1-3_m00009.jpg

In English: extract every record on the page following the attached schema, normalize names to modern versions and abbreviations to full names, and identify the image in the arquivo field.

The output must identify the record type and date, then capture the people and family relationships relevant to that type.

Baptisms and deaths revolve around a principal person; marriages name the bride and groom.

Fields unsupported by the document may be omitted, and the schema provides no place for witnesses or godparents.

Two additional conventions remove ambiguity before scoring.

Historical names and abbreviations are normalized to the expected modern form: “Joseph”, “Joze” and “José” can all be the same name in these books, as can “Manuel” and “Manoel”.

Without an explicit convention, a model that modernizes a spelling and a model that transcribes letter by letter would each be penalized against a reference written in the other style.

The bracketed section is a list of valid place names for the parishes in the benchmark, grouped by municipality and accompanied by historical spelling variants.

I compiled it through research separate from the annotation, mirroring my own practice of checking candidate places when the handwriting is ambiguous or a name is unfamiliar.

A subtle way of leaking data would be using the golden answers to extract the location, but we don’t make this type of mistake here (only other types).

Together, these rules define a valid extraction before the deterministic grader compares it with the human reference.

How Scoring Works

Each image earns two numbers: the ordered Transcription Score and an unordered diagnostic.

(While writing this post I noticed that Transcription Score was not a good name for the metric, as this is an extraction task, but bear with me)

Both operate on the annotated fields in the human reference.

Each field has equal weight, predicted fields without a corresponding annotation are ignored, and the image score is the total credit divided by the number of evaluable reference fields.

Text fields earn 1 − Levenshtein distance / length of the longer string.

This gives partial credit to imperfect readings that may still help a researcher find the right page.

A search for a João Manuel married to a Maria da Silva should still surface a page where the model read “João Martins” but preserved Maria da Silva and the place; a reading unrelated to the real name earns close to zero, and the credit fades smoothly between these extremes.

For dates, string similarity prices errors badly: 1600 and 1700 differ by one character and by a century, while the 7th and the 17th of a month differ more in text and still land a researcher in the right weeks.

So dates use temporal distance instead: max(0, 1 − days of difference / 56).

An exact date earns full credit, and an error of eight weeks or more earns zero. Reference dates are scored only when complete; a missing reference date earns a point only when the model also returns null.

One structural rule overrides field similarity: if the record type is wrong, every field in that record scores zero. A baptism interpreted as a death changes the event that gives its names and dates meaning.

The ordered score pairs predicted and reference records by position. A missing or invented record can therefore shift every record that follows it.

For example, suppose the reference order is Maria, Antonio, Matias, but the prediction skips Antonio and returns Maria, Matias.

The ordered score compares Matias with Antonio and leaves Matias unmatched; the unordered score restores the Maria–Maria and Matias–Matias matches, leaving only Antonio unmatched.

The unordered diagnostic uses the same field credits but finds the highest-scoring one-to-one assignment between complete predicted and reference records.

A large gap between the two scores indicates an ordering problem; two low scores indicate that the extraction itself failed.

Only the model’s final message is scored. Invalid JSON receives zero and is also recorded in the invalid_json metric.

Experiment Results

The experiment runs two conditions on Gemini 3.5 Flash Lite and Gemini 3.6 Flash: T0, a single-pass extraction, and T1, the same task inside an agentic harness.

Chart layout adapted from the beautiful DiligenceBench.

The diagram shows the ordered Transcription Score averaged across the 84 images.

Within each model, the environment helps: +0.17 for the budget model and +0.08 for the stronger one.

The unordered diagnostic follows the same pattern, rising from 0.46 to 0.63 on Gemini 3.5 Flash Lite and from 0.70 to 0.77 on Gemini 3.6 Flash.

The weaker reader gains more from the tools — the same pattern DiligenceBench reports on its harness ladder, where added rungs lift the weakest models most.

Across the two models, capability is the larger effect: the 3.6 Flash single pass already beats the 3.5 Lite with tools, and the best extraction belongs to the strongest model inside the harness.

The unordered score stays only 0.02 to 0.04 above the ordered one, so on average the loss concentrates on field extraction rather than record ordering.

The harness raises total cost from US$ 0.56 to US$ 8.74 on the budget model and from US$ 2.70 to US$ 47.47 on the stronger one, increases of 15.6× and 17.6×.

The incremental dollar buys less as the model improves: US$ 8.2 for +0.17 on the budget model, US$ 44.8 for +0.08 on the stronger one.

Invalid JSON appears only in the Gemini 3.6 Flash T1 run — three of 84 images fail to parse — a direct cost of emitting free-text JSON without the API’s schema enforcement.

Do Older Records Score Worse?

Yes, but the effect looks more like a seventeenth-century penalty than a smooth decline with age.

Field-weighted ordered Transcription Score over dated reference records; seven records without a visible year are excluded.

In every model-environment pair, the seventeenth century scores below both later centuries.

Two baptism entries from Vilarinho do Bairro show what “older and harder to read” means in this corpus:

1693	1795

Crops from ADAVR baptism books PT-ADAVR-PAND13-1-3 and PT-ADAVR-PAND13-1-7.

The later hand is darker and more regular, while the earlier page is faded, densely abbreviated and physically damaged.

Abbreviations get really crazy as you reach the 1600-1500s.

Interestingly, Gemini 3.6 Flash T1 improves clearly from the seventeenth to the eighteenth and then the nineteenth century.

What Separates T0 And T1, Exactly?

In T0, the image and prompt go into one API call, and the JSON comes out. The response is constrained by the strict JSON Schema attached to the request, reasoning effort sits at medium, and every image gets exactly one attempt.

In T1, each model runs inside an agentic harness with the same core prompt and sampling settings.

The harness is pi (v0.82.0), executing in a Docker sandbox that carries image tooling: Python with Pillow and OpenCV, ImageMagick, Tesseract OCR, and a JSON Schema validator, with the same output schema.

To point the agent to that file, the prompt’s first line asks the model to read the schema file before starting and answering with valid JSON.

Across multiple turns, the model can write and run its own scripts: split a double page, cut out individual records, enlarge faded lines, adjust contrast, try OCR on a stubborn passage, and validate the output file before answering.

Only the final message is scored, exactly as in T0.

The design was inspired by DiligenceBench, an equity-research benchmark that grades agent submissions with dense, task-specific rubrics.

Its reference harnesses form a controlled ladder: a base loop, where the model searches the web and answers from conversation context alone; a generic sandbox, which adds shell, Python and writable files on a persistent workspace; and a domain harness, which adds specialized tools and instructions.

There the sandbox rung beat the base loop for fifteen of the sixteen models tested — the same weights scored measurably higher once the model could execute and keep state.

My T0 and T1 mirror the first two rungs: the loop becomes a single vision call, the sandbox gains image tooling, and the question moves from “which model reads best in one shot?” to “what changes when the model can investigate the image?”.

Why Did I Test Only Gemini Models?

In my preliminary tests, Gemini 3.5 Flash Lite — the budget option — read these documents significantly better than every GPT and Claude model I tried, so the comparison stays within the family.

External evidence points the same way: Gemini models are the strongest general-purpose entries on the CHURRO leaderboard for historical document transcription, ahead of the best Claude and GPT results and behind only the benchmark authors’ own fine-tuned specialist model.

Independent testing on eighteenth- and nineteenth-century English manuscripts reached similar conclusions, with Gemini 3 Pro achieving error rates comparable to a professional typist and far ahead of both competitors.

One variation I didn’t test, but might in the future, is to run stronger agentic models with Gemini as a tool for vision.

A Few Interesting Rollouts

When Decomposition Recovers Structure

This image from 1636 and 1637 contains twelve marriage entries across two pages.

In T0, Gemini 3.5 Flash Lite examined the complete image once, drifted between death and baptism, returned a single baptism, and scored 0.00.

In T1, it split the scan into two pages, divided each page into vertical bands, created twelve entry-level crops, and increased the contrast on selected regions.

It also tried Tesseract, found the available OCR unsuitable for the seventeenth-century Portuguese handwriting, and returned to visual inspection.

The smaller views exposed the repeated marriage formula, including recebi por palavras de presente (“received” is a common language formula for marriages that indicate the bride and groom went to meet the record author), and made the document’s structure legible.

The agent returned twelve marriage objects and raised the score to 0.26.

The output still read eleven of the twelve years as 1631 instead of 1636 or 1637, omitted every parent, and recovered many names only partially.

One Page Where T1 Was Worse

On this page of four death entries, T1 performed worse than T0 for Gemini 3.5 Flash Lite.

T0 scored 0.90, while T1 scored 0.60.

Both conditions found four records, classified all four as deaths, and kept them in the correct order.

The entire 0.30 gap came from the dates.

The reference year is 1606, which T0 preserved across all four records, with three exact dates and one six days late.

T1 used 1666 for every record, putting all four dates beyond the scorer’s eight-week window and receiving zero date credit.

It considered 1660 and 1666, opened the complete image, located files, read the schema, and validated the JSON.

A standalone 1606 appears between the first and second entries, yet none of those steps corrected the year.

The validator could establish only that dates such as 1666-03-07 matched the required format.

A Rare Self-Correction

On this heavily crossed-out page, Branca (Blank) is one of the few legible words.

Gemini 3.6 Flash initially constructed a complete but unsupported baptism around it: a child named Branca, baptized on 24 November 1688 in Troviscal or Palhaça, with Manuel and Maria as parents.

Branca is, indeed, a valid first name in Portuguese.

The model then reconsidered the large flourishes, crossed-out writing, and placement of the word.

“I’m now certain ‘Branca’ does not refer to the name of a baptized individual.”

It interpreted the scan as a page marked blank and returned an empty array.

The Gemini 3.5 Flash Lite runs in both T0 and T1 also returned an empty array, although their traces did not show the same explicit construction and withdrawal of a full record.

The Opening Page, Side by Side

If you are curious about the opening page, Gemini 3.6 Flash T1 scored 0.79 on that image. Its final JSON compares with the human reference as follows:

The em dash marks a field absent from the human reference rather than an unread value.

Limitations and Conclusions

Across these four configurations, upgrading the reader buys more quality per dollar than placing Gemini 3.5 Flash Lite inside the harness.

The score itself requires careful interpretation.

A score of 0.74 includes partial credit for near matches and cannot be read as 74% of records being exactly correct.

The benchmark is also small and specific: 84 images, 434 records, 17 books, and two Portuguese archives.

I tried to be unbiased, but the images form a curated evaluation set from locations where I have ancestors rather than a random sample of the archives.

I produced the reference transcription myself, without an independent second annotator or paleographic adjudication.

That keeps the annotation policy consistent, while leaving some historical readings open to human error or interpretation.

Due to self-funded budget constraints, I only generated one answer per image. Without repeated samples, I cannot estimate how much of the image-level difference comes from model stochasticity.

I am keeping the dataset private for now to reduce the chance that it accidentally enters future model training data and contaminate the benchmark.

These are obscure archival scans, and I consider it unlikely that equivalent high-quality annotations or transcripts already exist in a public corpus.

I would love to see reinforcement learning environments targeting historical and genealogical tasks like this one though.

My ideal end state is a model reliable enough to extract every person and relationship in a record—including witnesses and godparents—while linking each claim to its source image.

Downstream systems could use these records to build family trees and follow a person through baptisms, marriages, deaths, and appearances in other people’s records.

Freely searchable online, this evidence-linked archive could become an unprecedented map of ordinary lives and the relationships connecting them across centuries.

And it’s likely easier than reading burned scrolls!

A Note On Verifiers

I first implemented this evaluation with my own custom code.

I later moved the benchmark to Verifiers, Prime Intellect’s open-source framework for building LLM evaluation and reinforcement-learning environments.

I made that migration to learn the framework through a real task whose data and failure modes I already understood, and because the same environment abstraction can later support reinforcement learning.

In the resulting environment, a Verifiers Taskset loads one multimodal task per image, including the image, prompt, and private reference records.

The task exposes the ordered Transcription Score as its reward and records the unordered score and invalid JSON rate as additional metrics.

The execution harness remains a separate choice, which allowed me to run the same taskset through a single model call in T0 or through pi in T1 without rewriting the scoring logic.

Verifiers also preserved the calls, tool interactions, usage, errors, rewards, and metrics that made the case studies and reconstructed cost analysis possible.