TinyGPT-V

DLYuanGod/TinyGPT-V is ranked 10,733 among these 8 peers; NVlabs/prismer is ranked 10,757.

TinyGPT-V **TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones** Zhengqing Yuan✟, Zhaoxu Li❁, Weiran Huang❋, Yanfang Ye✟, Lichao Sun❁ ✟University of Notre Dame, ❁Lehigh University, ❋Shanghai Jiao Tong University Zhaoxu is a visiting student in the LAIR lab at Lehigh University. We revised some type errors, provided more details and updated TinyGPT-V lastest results. 20 2024] Update the Phi-2 weight download link. 22 2024] Welcome to Hugging Face online demo to try out our models

(for Stage-4 v1)! We are officially releasing v1 of TinyGPT-V! After our evaluation, the performance of TinyGPT-V has reached 98% of InstructBLIP's performance and exceeds the performance of other models of the same period! 03 2024] Welcome to Hugging Face online demo to try out our models (for Stage-3)! TinyGPT-V Model Structure Whole Model Structure Language Model Structure TinyGPT-V Traning Process TinyGPT-V Results Radar Chart Performance and Efficiency Getting Started Installation **1. Prepare

the code and the environment** Git clone our repository, creating a python environment and activate it via the following command **2. Prepare the pretrained LLM weights** **TinyGPT-V** is based on Phi-2. Download the corresponding LLM weights from the following huggingface space via clone the repository using git-lfs. 7B: Download Then, set the variable *phi_model* in the model config file to the LLM weight path. Set the LLM path here at Line 14, here at Line 18

and here at Line 16. yaml at Line 8 for Stage 1, 2 and 3 version or tinygptv_stage4_eval. Update the Phi-2 Modeling for transformers lib. ** Linux system: Windows system Find your conda yourself: conda_sit/envs/tinygptv/lib/python3. 9/site-packages/transformers/models/phi/ Replace modeling_phi. py in that directory with the one in TinyGPT-V/modeling_phi. Launching Demo Locally For Stage 4, run Note: Stage 4 will have some Grounding abilities. But the

performance is not very good, we are working on this! For Stage 1, 2 and 3, run To perfer more powerful model, LLMs loads as 16 bit by default. This configuration requires about 8G GPU memory. To more save GPU memory, you can run the model in 8 bit below 8G device by setting to in the relevant config file: Stage 4 tinygptv_stage4_eval. yaml Stage 1, 2 and 3 tinygptv_stage1_2_3_eval. yaml Training First you

need to adjust all the updated weights in the LLM to be calculated with full precision:Here. Remove the comments from the following lines: **Stage 1 and 2:** Datasets: first stage dataset preparation instruction Then run: You need to execute the above code 17 times to complete the first stage of training. Then run: **Stage 3:** Datasets: stage 3 dataset preparation instruction Then run: **Stage 4:** Datasets: stage 4 dataset preparation instruction. details of TinyGPT-V, check here Star History

Acknowledgement MiniGPT A very versatile model of MLLMs. If you're using TinyGPT-V in your research or applications, please cite