The Problem Space

Translating text inside images sounds simple. It isn't. A screenshot of a Japanese manga panel has: speech bubbles with varying shapes, sound effects rendered as part of the artwork, vertical text, handwritten-style fonts, and text overlapping visual elements. A European street photo has: multiple fonts, arbitrary rotations, partial occlusion, varying lighting, and text at multiple scales in a single frame.

Most OCR pipelines handle clean, horizontal, well-lit text on uniform backgrounds. manju handles everything else. Here's how I built the pipeline that makes it work.

Detection Before Recognition

The first mistake in DIY OCR: treating detection and recognition as one step. They're not. Detection finds where text is in the image. Recognition reads what it says. These require different models and different optimization strategies.

For detection, I use a fine-tuned CRAFT (Character Region Awareness for Text Detection) model. CRAFT outputs character-level and word-level heatmaps, which makes it significantly more robust to curved, dense, or stylized text than bounding-box-only approaches. The output is a set of polygonal text regions — not rectangles, which matters for rotated or curved text.

Detection runs first. Only detected regions go to the recognition model. This keeps inference cost predictable even on complex images.

Recognition: The Model Choice That Matters Most

I tested six OCR models before settling on the current stack. The evaluation criteria: accuracy on degraded/stylized text, multilingual support (manju handles 40+ languages), inference latency, and model size for edge deployment.

The winner for production: PaddleOCR's recognition model, with custom post-processing. PaddleOCR's accuracy on CJK languages (Chinese, Japanese, Korean) is substantially better than Tesseract for complex layouts. It's also faster — 40-60ms per text region on CPU, under 10ms on GPU.

For languages with low training data representation, I fall back to a cloud vision API with a 200ms SLA. The routing logic is simple: if the detected script is in the high-confidence local model's coverage, run locally. Otherwise, call the API. This keeps costs manageable while maintaining coverage.

The Layout Reconstruction Problem

Here's the part that took the longest: after OCR, you have a list of text regions with their coordinates and recognized text. You need to reconstruct the reading order and spatial layout to pass meaningful context to the translation model.

For horizontal left-to-right text (most Latin scripts): cluster regions by Y position, sort clusters by X within each row. Straightforward. For vertical Japanese/Chinese: cluster by X position, sort by Y within each column, right-to-left across columns. For mixed layouts (manga panels with both vertical and horizontal elements): detect the dominant text direction per region using the aspect ratio and spacing of character bounding boxes.

This spatial context changes translation quality dramatically. "Go!" and "I'll kill you!" read very differently if the translation model knows they're in adjacent speech bubbles vs. scattered across a panel.

Translation: Why Vanilla LLM Calls Aren't Enough

Once I have ordered, contextual text blocks, they go to a translation LLM. The naive approach — "translate this text: [text]" — produces technically correct but contextually flat output. Manga characters have distinct speech patterns. Signs have brevity constraints. UI elements need functional accuracy over literary quality.

The prompt I settled on includes: detected text with spatial context, image type classification (manga / UI / photo / document), source language confidence, and character count constraints for text that needs to fit back in the original region. The LLM output is structured JSON with translation, reading order index, and confidence score.

The structured output matters for the next step: rendering the translation back into the image.

Inpainting and Overlay Rendering

The final — and most visually impactful — step: remove the original text from the image and render the translation in its place. This requires inpainting: filling the removed text regions with the underlying image background.

For clean backgrounds (solid colors, simple patterns), flood-fill inpainting is fast and looks perfect. For complex backgrounds (detailed artwork, photos), I use a lightweight diffusion inpainting model. It's slower (~300ms per region) but produces results that are visually indistinguishable from the original for most inputs.

Total pipeline latency for a typical manga panel: detection (80ms) + recognition (120ms) + translation (400ms) + inpainting (200ms) = ~800ms. Under a second, end-to-end, from image to translated image. That's fast enough to feel real-time in the UI.

What I'd Build Differently

The biggest architectural mistake: not designing for batching from day one. Processing regions sequentially made the math work in development. In production, parallel region processing reduced total latency by 45%. If you're building a multi-step CV pipeline, design the parallelism into the architecture from the start — retrofitting it is painful.