CVNets: A library for training computer vision networks

CVNets is a computer vision toolkit that allows researchers and engineers to train standard and novel mobile- and non-mobile computer vision models for variety of tasks, including object classification, object detection, semantic segmentation, and foundation models (e.g., CLIP).

Table of contents

• What's new?
• Installation
• Getting started
• Supported models and tasks
• Maintainers
• Research effort at Apple using CVNets
• Contributing to CVNets
• License
• Citation

What's new?

• July 2023: Version 0.4 of the CVNets library includes
  • Bytes Are All You Need: Transformers Operating Directly On File Bytes
  • RangeAugment: Efficient online augmentation with Range Learning
  • Training and evaluating foundation models (CLIP)
  • Mask R-CNN
  • EfficientNet, Swin Transformer, and ViT
  • Enhanced distillation support

Installation

We recommend to use Python 3.10+ and PyTorch (version >= v1.12.0)

Instructions below use Conda, if you don't have Conda installed, you can check out How to Install Conda.

# Clone the repo
git clone git@github.com:apple/ml-cvnets.git
cd ml-cvnets

# Create a virtual env. We use Conda
conda create -n cvnets python=3.10.8
conda activate cvnets

# install requirements and CVNets package
pip install -r requirements.txt -c constraints.txt
pip install --editable .

Getting started

• General instructions for working with CVNets are given here.
• Examples for training and evaluating models are provided here and here.
• Examples for converting a PyTorch model to CoreML are provided here.

Supported models and Tasks

To see a list of available models and benchmarks, please refer to Model Zoo and examples folder.

ImageNet classification models

• CNNs
  • MobileNetv1
  • MobileNetv2
  • MobileNetv3
  • EfficientNet
  • ResNet
  • RegNet
• Transformers
  • Vision Transformer
  • MobileViTv1
  • MobileViTv2
  • SwinTransformer

Multimodal Classification

• ByteFormer

Object detection

• SSD
• Mask R-CNN

Semantic segmentation

• DeepLabv3
• PSPNet

Foundation models

• CLIP

Automatic Data Augmentation

• RangeAugment
• AutoAugment
• RandAugment

Distillation

• Soft distillation
• Hard distillation

Maintainers

This code is developed by Sachin, and is now maintained by Sachin, Maxwell Horton, Mohammad Sekhavat, and Yanzi Jin.

Previous Maintainers

• Farzad

Research effort at Apple using CVNets

Below is the list of publications from Apple that uses CVNets:

• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer, ICLR'22
• CVNets: High performance library for Computer Vision, ACM MM'22
• Separable Self-attention for Mobile Vision Transformers (MobileViTv2)
• RangeAugment: Efficient Online Augmentation with Range Learning
• Bytes Are All You Need: Transformers Operating Directly on File Bytes

License

For license details, see LICENSE.

Citation

If you find our work useful, please cite the following paper:

@inproceedings{mehta2022mobilevit,
     title={MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer},
     author={Sachin Mehta and Mohammad Rastegari},
     booktitle={International Conference on Learning Representations},
     year={2022}
}

@inproceedings{mehta2022cvnets, 
     author = {Mehta, Sachin and Abdolhosseini, Farzad and Rastegari, Mohammad}, 
     title = {CVNets: High Performance Library for Computer Vision}, 
     year = {2022}, 
     booktitle = {Proceedings of the 30th ACM International Conference on Multimedia}, 
     series = {MM '22} 
}