Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

FlexFlow Train

[!WARNING] The FlexFlow repository has been split into separate flexflow-train and flexflow-serve repositories. You are currently viewing flexflow-train. For anything inference/serving-related, go to flexflow-serve.

FlexFlow Train is a deep learning framework that accelerates distributed DNN training by automatically searching for efficient parallelization strategies.

Contributing

Please let us know if you encounter any bugs or have any suggestions by submitting an issue.

For instructions on how to contribute code to FlexFlow Train, see CONTRIBUTING.md.

We welcome all contributions to FlexFlow Train from bug fixes to new features and extensions.

Citations

Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI), July 2022.
Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond Data and Model Parallelism for Deep Neural Networks. In Proceedings of the 2nd Conference on Machine Learning and Systems (MLSys), April 2019.
Zhihao Jia, Sina Lin, Charles R. Qi, and Alex Aiken. Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks. In Proceedings of the International Conference on Machine Learning (ICML), July 2018.

The Team

FlexFlow Train is developed and maintained by teams at CMU, Facebook, Los Alamos National Lab, MIT, Stanford, and UCSD (alphabetically).

License

FlexFlow Train uses Apache License 2.0.