TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/microsoft/unilm/tree/master/trocrOfficialpytorch★ 0
- github.com/huggingface/transformerspytorch★ 158,292
- github.com/oleehyo/textellerpaddle★ 726
- github.com/d-gurgurov/im2latexpytorch★ 19
- github.com/prathameshza/TrOCR_FineTuningnone★ 8
- github.com/MindCode-4/code-5/tree/main/trocrmindspore★ 0
- github.com/pwc-1/Paper-9/tree/main/1/trocrmindspore★ 0
- github.com/pwc-1/Paper-10/tree/main/trocrmindspore★ 0
Abstract
Text recognition is a long-standing research problem for document digitalization. Existing approaches are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on the printed, handwritten and scene text recognition tasks. The TrOCR models and code are publicly available at https://aka.ms/trocr.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| IAM | TrOCR-large 558M | CER | 2.89 | — | Unverified |
| IAM | TrOCR-base 334M | CER | 3.42 | — | Unverified |
| IAM | TrOCR-small 62M | CER | 4.22 | — | Unverified |
| IAM(line-level) | TrOCR | Test CER | 3.4 | — | Unverified |
| LAM(line-level) | TrOCR | Test CER | 3.6 | — | Unverified |