AdvancedLiterateMachinery

AlibabaResearch/AdvancedLiterateMachinery is ranked 9,399 among these 8 peers; Kyubyong/tacotron is ranked 9,401.

Advanced Literate Machinery Introduction The ultimate goal of our research is to build a system that has high-level intelligence, i. , possessing the abilities to ***read, think and create***, so advanced that it could even surpass human intelligence one day in the future. We name this kind of systems **Advanced Literate Machinery (ALM)**. To start with, we currently focus on teaching machines to ***read*** from images and documents. In years to come, we will explore the possibilities of endowing machines

with the intellectual capabilities of ***thinking and creating***, catching up with and surpassing GPT-4 and GPT-4V. 12 Release** **CC-OCR** (*CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy*. paper): The CC-OCR benchmark is specifically designed for evaluating the OCR-centric capabilities of Large Multimodal Models. CC-OCR possesses a diverse range of scenarios, tasks, and challenges, which comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document

parsing, and key information extraction. It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, being released for the first time. 9 Release** **Platypus** (*Platypus: A Generalized Specialist Model for Reading Text in Various Forms,* ECCV 2024. paper): Platypus introduces a novel approach to text reading from images, addressing limitations of both specialist and generalist models. Platypus leverages **a single unified architecture** to effectively recognize text in **various forms**, maintaining high accuracy

and efficiency. We also introduce a **new dataset Worms** which combines and partially re-labels previous datasets to support the model's development and evaluation. **SceneVTG** (*Visual Text Generation in the Wild,* ECCV 2024. paper): We propose a visual text generator (termed SceneVTG), which can produce **high-quality text images in the wild**. Following a **two-stage paradigm**, SceneVTG leverages a Multimodal Large Language Model to recommend reasonable text regions and contents across multiple scales and levels, which are used by

a conditional diffusion model as conditions to generate text images. To train SceneVTG, we also contribute a **new dataset SceneVTG-Erase** with detailed OCR annotations. **WebRPG** (*WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentation,* ECCV 2024. paper): We introduce WebRPG, a novel task that focuses on **automating the generation of visual presentations** for web pages based on HTML code. In the absence of a benchmark, we created a new dataset via an **automated pipeline**. Our proposed models, built on

**VAE architecture** and **custom HTML embeddings**, efficiently manage numerous web elements and rendering parameters. Comprehensive experiments, including customized quantitative evaluations, demonstrate the effectiveness of WebRPG model in generating web presentations. **ProcTag** (*ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data,* arXiv 2024. paper): An effective evaluation method for document instruction data is crucial in constructing instruction data with high efficacy, which, in turn, facilitates the training of LLMs and MLLMs for document understanding. We propose ProcTag, a data

-oriented method that **tags the execution process of instructions** rather than the text itself, enabling more effective assessment and selective