AI Rank
EN
← 开源
yoshitomo-matsubara

torchdistill

A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆26 knowledge distillation methods presented at TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV, AAAI, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.

Model DevelopmentTraining toolkitsPython
在 GitHub 打开
增长势头
+024 小时新增 Star0.0%
1.63k
Star
144
Fork
+-1
本周
5
贡献者
创建于 2019-12-18 · 更新于 2026-10-12 · 今日第 9821 名
主要开发者
yoshitomo-matsubararezafurubot66

yoshitomo-matsubara/torchdistill 排名第 9,821，有 1,633 个星标。

同类别比较

yoshitomo-matsubara/torchdistill 排名第 9,821，此处有 8 个同类条目；csinva/imodels 排名第 9,825。

同类别比较
名称排名指标
lucidrains/alphafold29,805
语言:
Python
星标数:
1,640
分叉数:
264
简介:
To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released
datamade/usaddress9,812
语言:
Python
星标数:
1,637
分叉数:
309
简介:
:us: a python library for parsing unstructured United States address strings into address components
autonomio/talos9,813
语言:
Python
星标数:
1,635
分叉数:
265
简介:
Hyperparameter Experiments with TensorFlow and Keras
openai/evolution-strategies-starter9,814
语言:
Python
星标数:
1,635
分叉数:
281
简介:
Code for the paper "Evolution Strategies as a Scalable Alternative to Reinforcement Learning"
csinva/imodels9,825
语言:
Jupyter Notebook
星标数:
1,627
分叉数:
148
简介:
Interpretable ML package 🔍 for concise, transparent, and accurate predictive modeling (sklearn-compatible).
facebookresearch/pytorch_GAN_zoo9,826
语言:
Python
星标数:
1,627
分叉数:
268
简介:
A mix of GAN implementations including progressive growing
tensorflow/mesh9,828
语言:
Python
星标数:
1,627
分叉数:
255
简介:
Mesh TensorFlow: Model Parallelism Made Easier
pavlin-policar/openTSNE9,831
语言:
Python
星标数:
1,625
分叉数:
179
简介:
Extensible, parallel implementations of t-SNE
关键参数与规格
关键参数与规格
参数值
语言Python
类别Model Development
子类别Training toolkits
星标数1,633
分叉数144
贡献者5
创建日期2019-12-18
更新日期2026-10-12
1 天星标数0
7 天星标数-1
排名9,821
上次排名9,855
排名变化34
7 天星标排名17,068
上次 7 天星标排名17,071
7 天星标排名变化3
总星标排名7,543
上次总星标排名7,537
总星标排名变化-6
国家USA
README 摘录

torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation ***torchdistill*** (formerly *kdkit*) offers various state-of-the-art knowledge distillation methods and enables you to design (new) experiments simply by editing a declarative yaml config file instead of Python code. Even when you need to extract intermediate representations in teacher/student models, you will **NOT** need to reimplement the models, that often change the interface of the forward, but instead specify the module path(s) in the yaml file. In

addition to knowledge distillation, this framework helps you design and perform general deep learning experiments (**WITHOUT coding**) for reproducible deep learning studies. , it enables you to train models without teachers simply by excluding teacher entries from a declarative yaml config file. You can find such examples below and in configs/sample/. In December 2023, ***torchdistill*** officially joined PyTorch Ecosystem. When you refer to ***torchdistill*** in your paper, please cite these papers instead of this GitHub repository. **If you use** ***torchdistill

*** **as part of your work, your citation is appreciated and motivates me to maintain and upgrade this framework! ** Documentation You can find the API documentation and research projects that leverage ***torchdistill*** at Forward hook manager Using **ForwardHookManager**, you can extract intermediate representations in model without modifying the interface of its forward function. will give you a better idea of the usage such as knowledge distillation and analysis of intermediate representations. , extract intermediate representations (feature map) of ResNet-18 for a

random input batch 1 experiment → 1 declarative PyYAML config file In ***torchdistill***, many components and PyTorch modules are abstracted e. , models, datasets, optimizers, losses, and more! You can define them in a declarative PyYAML config file so that can be seen as a summary of your experiment, and in many cases, you will **NOT need to write Python code at all**. Take a look at some configurations available in configs/. You'll see what modules are abstracted and how they are

defined in a declarative PyYAML config file to design an experiment. , instantiate CIFAR-10 datasets with a declarative PyYAML config file **test. yaml** If you want to use your own modules (models, loss functions, datasets, etc) with this framework, you can do so without editing code in the local package . See the official documentation and Discussions for more details. Benchmarks Top-1 validation accuracy for ILSVRC 2012 (ImageNet) Examples Executable code can be found in examples/ such as Image classification

: ImageNet (ILSVRC 2012), CIFAR-10, CIFAR-100, etc Object detection: COCO 2017, etc Semantic segmentation: COCO 2017, PASCAL VOC, etc Text classification: GoEmotions, etc GLUE: CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI, AX For CIFAR-10 and CIFAR-100, some models are reimplemented and available as pretrained models in ***torchdistill***. Some Transformer models fine-tuned by ***torchdistill*** for GLUE tasks are available at Hugging Face Model Hub. Sample GLUE benchmark results and details can be found

here. Google Colab Examples The following examples are available in demo/. Note that these examples are for Google Colab users

常见问题

记录了多少星标和分叉？

yoshitomo-matsubara/torchdistill 有 1,633 个星标和 144 个分叉。

仓库排名是多少？

yoshitomo-matsubara/torchdistill 排名第 9,821。

记录的语言和类别是什么？

yoshitomo-matsubara/torchdistill 有 1,633 个星标，语言为 Python，类别为 Model Development。

哪个仓库排名最接近？

yoshitomo-matsubara/torchdistill 与 csinva/imodels 比较，后者排名第 9,825，有 1,627 个星标。

README

torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

torchdistill (formerly kdkit) offers various state-of-the-art knowledge distillation methods and enables you to design (new) experiments simply by editing a declarative yaml config file instead of Python code. Even when you need to extract intermediate representations in teacher/student models, you will NOT need to reimplement the models, that often change the interface of the forward, but instead specify the module path(s) in the yaml file. Refer to these papers for more details.

In addition to knowledge distillation, this framework helps you design and perform general deep learning experiments (WITHOUT coding) for reproducible deep learning studies. i.e., it enables you to train models without teachers simply by excluding teacher entries from a declarative yaml config file. You can find such examples below and in configs/sample/.

In December 2023, torchdistill officially joined PyTorch Ecosystem.

When you refer to torchdistill in your paper, please cite these papers instead of this GitHub repository.
If you use torchdistill as part of your work, your citation is appreciated and motivates me to maintain and upgrade this framework!

Documentation

You can find the API documentation and research projects that leverage torchdistill at https://yoshitomo-matsubara.net/torchdistill/

Forward hook manager

Using ForwardHookManager, you can extract intermediate representations in model without modifying the interface of its forward function.
This example notebook will give you a better idea of the usage such as knowledge distillation and analysis of intermediate representations.

E.g., extract intermediate representations (feature map) of ResNet-18 for a random input batch

import torch
from torchvision import models
from torchdistill.core.forward_hook import ForwardHookManager

# Define a model and choose torch device
model = models.resnet18(pretrained=False)
device = torch.device('cpu')

# Register forward hooks for modules of your interest
forward_hook_manager = ForwardHookManager(device)
forward_hook_manager.add_hook(model, 'conv1', requires_input=True, requires_output=False)
forward_hook_manager.add_hook(model, 'layer1.0.bn2', requires_input=True, requires_output=True)
forward_hook_manager.add_hook(model, 'fc', requires_input=False, requires_output=True)

# Define a random input batch and run the model
x = torch.rand(32, 3, 224, 224)
y = model(x)

# Extract input and/or output of the modules
io_dict = forward_hook_manager.pop_io_dict()
conv1_input = io_dict['conv1']['input']
layer1_0_bn2_input = io_dict['layer1.0.bn2']['input']
layer1_0_bn2_output = io_dict['layer1.0.bn2']['output']
fc_output = io_dict['fc']['output']

1 experiment → 1 declarative PyYAML config file

In torchdistill, many components and PyTorch modules are abstracted e.g., models, datasets, optimizers, losses, and more! You can define them in a declarative PyYAML config file so that can be seen as a summary of your experiment, and in many cases, you will NOT need to write Python code at all. Take a look at some configurations available in configs/. You'll see what modules are abstracted and how they are defined in a declarative PyYAML config file to design an experiment.

E.g., instantiate CIFAR-10 datasets with a declarative PyYAML config file

from torchdistill.common import yaml_util
config = yaml_util.load_yaml_file('./test.yaml')
train_dataset = config['datasets']['cifar10/train']
test_dataset = config['datasets']['cifar10/test']

test.yaml

datasets:
  cifar10/train: !import_call
    key: 'torchvision.datasets.CIFAR10'
    init:
      kwargs:
        root: &root_dir '~/datasets/cifar10'
        train: True
        download: True
        transform: !import_call
          key: 'torchvision.transforms.Compose'
          init:
            kwargs:
              transforms:
                - !import_call
                  key: 'torchvision.transforms.RandomCrop'
                  init:
                    kwargs:
                      size: 32
                      padding: 4
                - !import_call
                  key: 'torchvision.transforms.RandomHorizontalFlip'
                  init:
                    kwargs:
                      p: 0.5
                - !import_call
                  key: 'torchvision.transforms.ToTensor'
                  init:
                - !import_call
                  key: 'torchvision.transforms.Normalize'
                  init:
                    kwargs: &normalize_kwargs
                      mean: [0.49139968, 0.48215841, 0.44653091]
                      std: [0.24703223, 0.24348513, 0.26158784]
  cifar10/test: !import_call
    key: 'torchvision.datasets.CIFAR10'
    init:
      kwargs:
        root: *root_dir
        train: False
        download: True
        transform: !import_call
          key: 'torchvision.transforms.Compose'
          init:
            kwargs:
              transforms:
                - !import_call
                  key: 'torchvision.transforms.ToTensor'
                  init:
                - !import_call
                  key: 'torchvision.transforms.Normalize'
                  init:
                    kwargs: *normalize_kwargs

If you want to use your own modules (models, loss functions, datasets, etc) with this framework, you can do so without editing code in the local package torchdistill/.
See the official documentation and Discussions for more details.

Benchmarks

Top-1 validation accuracy for ILSVRC 2012 (ImageNet)

Examples

Executable code can be found in examples/ such as

For CIFAR-10 and CIFAR-100, some models are reimplemented and available as pretrained models in torchdistill. More details can be found here.

Some Transformer models fine-tuned by torchdistill for GLUE tasks are available at Hugging Face Model Hub. Sample GLUE benchmark results and details can be found here.

Google Colab Examples

The following examples are available in demo/. Note that these examples are for Google Colab users and compatible with Amazon SageMaker Studio Lab. Usually, examples/ would be a better reference if you have your own GPU(s).

CIFAR-10 and CIFAR-100

  • Training without teacher models
  • Knowledge distillation

GLUE

  • Fine-tuning without teacher models
  • Knowledge distillation

These examples write out test prediction files for you to see the test performance at the GLUE leaderboard system.

PyTorch Hub

If you find models on PyTorch Hub or GitHub repositories supporting PyTorch Hub, you can import them as teacher/student models simply by editing a declarative yaml config file.

e.g., If you use a pretrained ResNeSt-50 available in huggingface/pytorch-image-models (aka timm) as a teacher model for ImageNet dataset, you can import the model via PyTorch Hub with the following entry in your declarative yaml config file.

models:
  teacher_model:
    key: 'resnest50d'
    repo_or_dir: 'huggingface/pytorch-image-models'
    kwargs:
      num_classes: 1000
      pretrained: True

How to setup

  • Python >= 3.9
  • pipenv (optional)

Install by pip/pipenv

pip3 install torchdistill
# or use pipenv
pipenv install torchdistill

Install from this repository (not recommended)

git clone https://github.com/yoshitomo-matsubara/torchdistill.git
cd torchdistill/
pip3 install -e .
# or use pipenv
pipenv install "-e ."

Issues / Questions / Requests / Pull Requests

Feel free to create an issue if you find a bug.
If you have either a question or feature request, start a new discussion here. Please search through Issues and Discussions and make sure your issue/question/request has not been addressed yet.

Pull requests are welcome. Please start with an issue and discuss solutions with me rather than start with a pull request.

Citation

If you use torchdistill in your research, please cite the following papers:
[Paper] [Preprint]

@inproceedings{matsubara2021torchdistill,
  title={{torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation}},
  author={Matsubara, Yoshitomo},
  booktitle={International Workshop on Reproducible Research in Pattern Recognition},
  pages={24--44},
  year={2021},
  organization={Springer}
}

[Paper] [OpenReview] [Preprint]

@inproceedings{matsubara2023torchdistill,
  title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},
  author={Matsubara, Yoshitomo},
  booktitle={Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)},
  publisher={Empirical Methods in Natural Language Processing},
  pages={153--164},
  year={2023}
}

Acknowledgments

This project has been supported by Travis CI's OSS credits since November 2021. JetBrain's Free License Programs (Open Source) has supported the project since June 2022.

References

排名相邻
同类项目