# Barbados Historic Handwriting Challenge - Reading 300-year-old deeds with vision-language models **Final result:** Private score: **0.926180** · Public score: 0.92969 · 20th place over 620 ## 1. The challenge The goal of the competition: > Given the image of **one handwritten line** from a 17th- or 18th-century Barbadian notarial deed, transcribe it **exactly as written**. "Exactly" is the hard part. The model has to keep the scribe's spelling, capitals, abbreviations and `^` superscript markers. No normalisation, no modern English. The images make it harder still: - secretary and round hands, faded ink, stains, torn paper; - crops from about 60 px high to high-resolution scans above 200 px; - about 39 % of the lines show fragments of the lines above or below. The data: **4,098** transcribed training lines and **1,374** test lines. The competition score was: $$ \text{Score} = 1 - \frac{1}{2}\left(\frac{\overline{d}_{\text{word}}}{12} + \frac{\overline{d}_{\text{char}}}{55}\right) $$ where $\overline{d}_{\text{word}}$ and $\overline{d}_{\text{char}}$ are the mean word-level and character-level edit distances over the test lines. Since $55 / 12 \approx 4.6$, a word error costs about **4.6 times** as much as a character error, so getting whole words right matters more than getting most letters right. ## 2. The main idea: generate, then select Instead of asking one model for one answer, I split the problem in two: - **Generate.** Two fine-tuned vision-language models (Qwen3-VL 8B and 32B) each propose several transcriptions, about **14 distinct candidates per line**. - **Select.** A scorer picks the best candidate, combining three independent opinions: - small CTC recognisers ("judges") that only look at the letter shapes; - a character language model of the training transcriptions; - the readers' own probability of each candidate ("cross-scores"). On top of that, the readers are improved with **pseudo-labelled test lines**. No external data and no manual transcription were used. ## 3. Architecture **Inference pipeline** ```mermaid flowchart TD IMG["Test line image
contrast ×1.5"] subgraph GEN["1 · Generate"] R32["Reader pseudo32
Qwen3-VL-32B + LoRA"] R8["Reader pseudo8
Qwen3-VL-8B + LoRA"] end POOL["Candidate pool
~14 per line"] subgraph SCORE["2 · Score"] J["3 CTC judges
ConvNeXt + Transformer
central text band, 3 crops"] LM["Character 14-gram LM
training transcriptions"] XS["Cross-scores
each candidate forced
through both readers"] end SEL["3 · Select
weighted sum of z-scores"] OUT[("final_submission.csv")] IMG --> R32 & R8 R32 -->|"beam-3 top-1 + 8-best"| POOL R8 -->|"beam-3 top-1 + 8-best"| POOL POOL --> J & LM & XS J --> SEL LM --> SEL XS --> SEL SEL --> OUT ``` **Training pipeline** ```mermaid flowchart TD DATA[("Train.csv + images
4,098 lines")] subgraph BASE["Base models"] B32["Reader base32"] B8["Reader base8"] J1["Judges dino96, h128"] end R1{{"Round 1
read the test set,
keep lines every reader agrees on"}} P8["Reader pseudo8
Judge dino128p"] R2{{"Round 2
read the test set,
keep lines every reader agrees on"}} P32["Reader pseudo32"] FINAL[("Final models
pseudo32, pseudo8,
dino96, h128, dino128p")] DATA --> BASE BASE --> R1 R1 -->|"round-1 pseudo-labels"| P8 DATA --> P8 BASE --> R2 P8 --> R2 R2 -->|"round-2 pseudo-labels"| P32 DATA --> P32 P32 --> FINAL P8 --> FINAL J1 --> FINAL ``` ## 4. The readers: Qwen3-VL + LoRA A vision-language model reads the whole line image and writes the transcription token by token. I fine-tuned Qwen3-VL-8B and Qwen3-VL-32B with **LoRA on every linear layer**, both in the language model and in the vision tower. | | | |---|---| | LoRA | r 32, α 64, dropout 0.1 | | Training | 3 epochs, lr 1.5e-4 cosine, effective batch 8, bf16 | | Image | RGB, contrast ×1.5, 512-1,536 visual tokens | | Augmentation | small rotations, scaling, brightness/contrast, noise, motion blur | | Prompt | transcribe the central line only, keep spelling, capitals and `^` markers | At inference, each reader returns its beam-3 best answer plus the 8-best list of a beam-8 search. ## 5. The judges: small CTC models that only look at the ink The judges read worse than Qwen, and that is not the point. They share **nothing** with it: a different architecture, different pretraining, no built-in language model. That makes them good at **ranking** Qwen's candidates. - **Architecture:** ConvNeXt-small backbone (downsampling only vertically) → 4-layer Transformer → character CTC head. - **Input:** the ink band nearest the vertical centre of the image, found from the horizontal ink profile, so fragments of neighbouring lines are cut away. - **Scoring:** for a candidate $c$ with $|c|$ characters and a judge $j$, the score is the CTC log-likelihood per character, averaged over 3 crops $x_k$ of the band (margins 0.18, 0.28, 0.40) after z-scoring within the line: $$ s_j(c) = \frac{1}{3}\sum_{k=1}^{3} z\!\left(\frac{\log p_{\text{CTC}}^{(j)}(c \mid x_k)}{|c|}\right) $$ | judge | backbone | band height | |---|---|---| | dino96 | ConvNeXt-small, DINOv3 weights | 96 px | | h128 | ConvNeXt-small, ImageNet weights | 128 px | | dino128p | ConvNeXt-small, DINOv3 weights + pseudo-labels | 128 px | ## 6. The language model and cross-scores **Character 14-gram LM.** The deeds repeat legal formulas word for word. A high-order character model (Witten-Bell smoothing) recognises them without ever seeing the image. **Cross-scores.** This was one of the most useful findings. A reader ranks its own beam badly: it puts only **55 %** of candidate pairs in the right order, because it prefers what it generated itself. But if every candidate, whoever proposed it, is **forced through** the reader and scored by its mean token log-probability, the ranking jumps to **69 %**: $$ x_r(c) = \frac{1}{T}\sum_{t=1}^{T} \log p_r\left(c_t \mid c_{