本地多模態 LLM 辨識台灣便當店手寫收據——能用嗎？Local Multimodal LLM vs Taiwanese Lunch Box Receipts — Is It Usable?

For personal bookkeeping this is the holy grail — snap a photo, store name / items / total filled in automatically. Cloud services (Google Vision, AWS Textract) can do this, but they cost money and require sending receipt images off-device.

Can local models do it?

This article uses Ollama's gemma4:12b (7.6 GB, from Google, vision-capable) on 20 simulated lunch-box receipts. The results were surprising — at a cost.

Why I Switched From minicpm-v to gemma4:12b

The first version of this article used minicpm-v:8b — 5.5 GB, fast, accurate. But it's from OpenBMB / Tsinghua-affiliated ModelBest, a PRC team. Resonance Stack's editorial stance is to not recommend or evaluate PRC-trained models (see knowledge/master/prohibited.md) — minicpm-v isn't on the explicit blocklist but it fits the spirit of hard rule 6.

So I reran the entire experiment with gemma4:12b (Google, non-PRC). The numbers below honestly reflect the real cost of "choosing a non-PRC local multimodal model" — both accuracy and speed take a hit, but the training data and weight provenance are cleanly bounded.

The full comparison is in the accuracy section; here's the snapshot:

Translated: 8× slower, store-name accuracy down 15 points. That's the cost of the "local + non-PRC" path on PIL synthetic data. The two big tests below use a public real-world dataset (CORD-v2) and the same PIL synthetic set, so you can see where the numbers diverge.

Public-Dataset Test: CORD-v2

Before any synthetic data, here's gemma4:12b on a public real-world benchmark. This number is the realistic floor of what readers will see.

Dataset: CORD-v2 (released by Naver Clova, non-PRC team) — Korean and Indonesian real retail receipts, photographed on phones, varying resolution and quality, with structured JSON ground truth. I sampled 20 from the test split, filtering to receipts with 2-6 items. License: research use.

Scoring (slightly different from the PIL version):

• CORD ground truth doesn't reliably include a store name, so store-match isn't scored
• Item-count tolerance widened to ±1 (CORD splits "main item + modifier" in a way that doesn't always match how the model groups them)
• Total tolerance widened to ±5% or ±50 (CORD amounts are in Indonesian rupiah — much larger numbers)

Results:

The three 55%s are identical — because these 20 receipts either pass all three or fail all three. There's no "parse OK but numbers wrong" middle ground. When gemma4:12b responds on CORD, the 11 that respond are 100% accurate; when it doesn't (9 receipts), it's 113 seconds of inference followed by an empty ''.

CORD vs PIL Taiwan, side by side

That 30-40 point gap is the most important number in this article. When you read any piece that says "we tested multimodal on synthetic data," remember that real data's ceiling is roughly this much lower.

CORD Failure Mode: Empty Responses, Image-Size Correlation

All 9 failures are empty responses — 113 seconds of inference followed by "". No JSON parse error, no error message, no partial output. From ollama's view it's HTTP 200 with an empty body.

Original image sizes for success vs failure:

Failures skew larger, but not strictly — #17 failed at only 306 KB.

Hypothesis test: downscale the 9 failures to 1024 px wide and retry:

2 of 9 recovered — downscaling helps but isn't reliable. Meaning gemma4:12b's failures on big images aren't purely "too large to fit"; the visual encoding pipeline is unstable somewhere.

Adding the 2 recovered, the overall pass rate climbs to 13/20 = 65%. Still not production grade.

This matches the "occasional empty responses" pattern from needle-in-haystack and format-reliability — but on vision the frequency is an order of magnitude higher (0 of 30 text runs, 1 of 20 PIL vision runs, 9 of 20 CORD vision runs). Image complexity drives the failure rate up.

Practical recommendations:

1. Timeout ≥ 120 seconds — empty responses on CORD arrive at the 113-115 second mark
2. Retry-on-empty is mandatory, and downscale to 1024 px before retrying
3. Don't feed receipt images > 1.5 MB to gemma4:12b — failure rate exceeds 50%
4. For high-failure scenarios, fall back to cloud vision (Claude / GPT-4o)

Controlled Synthetic Baseline: PIL Taiwan-Style Receipts

The ideal experiment uses real lunch-box receipts, but two practical concerns:

1. High collection cost: shooting 20 receipts across different stores, lighting, and wear states
2. High ground-truth annotation cost: manually labeling store/items/amounts is error-prone

So this experiment uses PIL-generated receipts as a "controlled baseline":

• 10 printed-style (rendered with STHeiti Medium)
• 10 handwritten-style (same font, plus ±3° rotation + 0.7px Gaussian blur, simulating handwritten irregularity and paper creasing)

ground_truth.json is saved alongside (the script knows what it generated). This is the "best-case environment" — if the model fails here, real photos will definitely fail.

What the Receipts Look Like

Each receipt contains:

• Store: random from 10 fixed names (金門排骨飯, 鼎泰豐, 阿婆便當, 三商巧福...)
• Items: 2-5 items drawn from a pool of 14 (排骨飯, 雞腿飯, 酸辣湯, 青菜...)
• Quantity: 1-3 per item
• Total: auto-calculated

The handwritten-style rotation and blur make the image look like "a lunch-box auntie wrote it on a slip of paper with a pen."

Running All 20

Each receipt is base64-encoded and sent to gemma4:12b with the instruction "extract store, items, and total as JSON":

resp = requests.post('http://localhost:11434/api/generate', json={
    'model': 'gemma4:12b',
    'prompt': PROMPT,
    'images': [base64_encoded],
    'stream': False,
    'options': {'temperature': 0.1, 'num_predict': 1500},
}, timeout=600)  # note: timeout raised from 300 to 600 — some receipts take 100+ seconds

Key observations:

• First receipt: 38.2 s (warmup + inference)
• Subsequent receipts: stable 50-70 s — not the minicpm-v "warm up then speed up" curve. Every receipt makes 12B think hard
• 19 of 20 parsed; receipt #12 (handwritten) burned 111 seconds and returned an empty string — consistent with the "sustained 12B runs occasionally return empty" pattern from the format-reliability article

Printed Receipt Results: "Structure right, characters slip"

Sample #05 (山東水餃館), gemma4:12b extracted:

{
  "store": "山東水餃館",
  "items": [
    {"name": "豬腳飯", "qty": 1, "subtotal": 130},
    {"name": "酸辣湯", "qty": 2, "subtotal": 80},
    {"name": "雞腿飯", "qty": 1, "subtotal": 110}
  ],
  "total": 320
}

Perfect field-by-field match. But this is one of the easy ones — 3 of 10 printed receipts had store-name errors, all visual confusions between similar characters:

#07 is particularly telling — store name off by one character, but everything else (4 items with correct quantities and subtotals, total 535) is exact. The structure is reliable, individual characters get guessed. Downstream implication: fine for arithmetic, not fine for "match this store name to a database lookup."

Handwritten Results: 9/10 pass, the missing one is expensive

9 of 10 handwritten receipts passed all three structural metrics. But the 10th (#12) deserves attention — it represents a failure mode for 12B's vision side:

Receipt #12 (handwritten, 金門排骨飯) — after 111.5 seconds, the model returned an empty string. No JSON, no markdown, no text at all. The raw output for this row is "".

For downstream code, an empty response is harder to handle than a wrong one — wrong outputs are validatable, empties just need retry. This matches the "12B occasionally returns empty under sustained load" pattern from gemma-12b-needle-in-haystack, just more frequent on vision tasks (0 empties in 30 text runs vs 1 empty in 20 vision runs).

For comparison: minicpm-v processed the same #12 in 9 seconds, returned valid JSON, every field 100% correct. That's part of the real cost of switching to a non-PRC model.

By the numbers, 9/10 on handwritten is more surprising than 10/10 on printed. Of the 10 handwritten:

• 7 perfect
• 2 with mild store-name errors (including #19 三商巧福 → 三喬巧福)
• 1 total failure (#12)

Implication: gemma4:12b is actually fairly robust against rotation + blur. The main failure is random per-call empties, not a systematic "handwritten is harder" degradation.

Overall Accuracy: gemma4:12b vs minicpm-v Side by Side

Both models scored on all 20:

Two observations:

1. gemma4:12b loses 5 points on the most important metric — totals correct (100 → 95). The entire gap is the empty-response on #12; the other 19 totals are exact. With per-call retry logic, most of that 5% is recoverable.
2. Store-name match drops from 90% to 75% — this one isn't recoverable by retry. It's a real character-recognition gap. gemma4:12b is genuinely weaker at fine-grained Traditional Chinese visual discrimination (婆/凌, 肉/排, 天/夭, 商/喬). If your downstream needs exact store-name joins, add fuzzy match or an LLM rerank step.

Both models share "printed and handwritten accuracy track closely" — the ±3° rotation + 0.7 px blur in this test isn't enough to distinguish them. Whether real lunch-box photos with oil stains and creases would flip the result is still an open question (this article doesn't test that — real-photo experiments are a follow-up).

Failure Mode Analysis

gemma4:12b had 5 imperfect results out of 20. Two categories:

Type A: Character confusion (5 receipts, but only the store-name field is affected)

Pattern: totals, item counts, and item names are unaffected — the slip always lands on the second or third character of the store name. Downstream implication: "totals are reliable, store-name joins are not."

Type B: Empty response (1 receipt)

#12 ran for 111.5 seconds then returned "". No partial result, no fallback, no error. From ollama's side, it looks successful (HTTP 200) with an empty body.

This matches the failure pattern seen in the format-reliability article — 12B occasionally returns empty under sustained load. Vision tasks trigger it more often than text (0 in 30 text runs, 1 in 20 vision runs).

Failure types that did NOT occur (important):

• No hallucinated items (no fictional menu entries)
• No fabricated amounts (no NT$999,999 absurdities)
• No item-count miscounts (other than #12, every receipt's count is correct)
• 19 of 20 JSONs are well-formed (#12 has no JSON at all)

Conclusion: gemma4:12b is highly reliable on structure and numbers, fragile on fine-grained Chinese character discrimination, and has an occasional single-call empty-response failure under sustained load. Three practical actions:

1. Add retry-on-empty (recovers the #12-type failure)
2. Add fuzzy match or LLM rerank on store names (mitigates Type A)
3. Don't use store-name as an exact JOIN key; use (amount + date + approximate store-name token) instead

Decision Framework

Ready to use with gemma4:12b (provided you accept ~60s/receipt):

• Auto-billing (totals-driven) → ✅ 95%, plus retry-on-empty gets you to ~99%
• Bookkeeping assistance (error-tolerant) → ✅ sampling for human review is fine
• Controlled images (PIL-generated, PDF screenshots, clean scans) → ✅ structured accuracy is high
• Batch jobs (overnight, latency-insensitive) → ✅ better suited than interactive

Still not ready:

• Real-time UI (user snaps a photo, expects immediate result) → ❌ 60s/receipt is too slow
• Real oily-handwritten lunch-box photos → ❓ untested; expect performance below PIL-generated
• Accounting / tax compliance receipts → ❌ requires character-exact matching; 75% store-name match isn't enough
• High-frequency batch (1000+ receipts/day) → ❌ 1 min/receipt = 16+ hours of continuous runtime, will hit #12-type empties

If your use case demands minicpm-v-level speed and accuracy:

Consider hosted multimodal (Claude Sonnet, GPT-4o) — sub-5s per receipt, accuracy higher than gemma4:12b, but paid and the data leaves your machine. Resonance Stack doesn't recommend PRC models for local paths, but for "I need the answer right" readers, the honest recommendation is cloud, not local-PRC.

Engineering reminders for using gemma4:12b on vision:

1. timeout must be ≥ 600 seconds (individual receipts can take 110+ seconds)
2. retry-on-empty is mandatory — don't treat the empty string as a valid response
3. Model is 7.6 GB; ollama process resident memory ~9 GB during inference; M1 Pro 16GB works but is tight with other apps open
4. Don't run other heavy apps while a 12B vision batch is going

What This Article Covers

Both endpoints were measured:

• Synthetic (PIL Taiwan lunch-box style) → 95% pass — optimistic ceiling
• Public real-world (CORD-v2 Korean/Indonesian printed) → 55-65% pass — realistic floor

Real Taiwan handwritten lunch-box receipts weren't tested — that's a different corpus I don't currently have. The two numbers above stand on their own; neither depends on the untested third.