You are viewing v0.32.0 version.
A newer version
v0.34.2 is available.
Example Zoo
Below contains a non-exhaustive list of tutorials and scripts showcasing đ€ Accelerate
Official Accelerate Examples:
Basic Examples
These examples showcase the base features of Accelerate and are a great starting point
- Barebones NLP example
- Barebones distributed NLP example in a Jupyter Notebook
- Barebones computer vision example
- Barebones distributed computer vision example in a Jupyter Notebook
- Using Accelerate in Kaggle
Feature Specific Examples
These examples showcase specific features that the Accelerate framework offers
- Automatic memory-aware gradient accumulation
- Checkpointing states
- Cross validation
- DeepSpeed
- Fully Sharded Data Parallelism
- Gradient accumulation
- Memory-aware batch size finder
- Metric Computation
- Using Trackers
- Using Megatron-LM
Full Examples
These examples showcase every feature in Accelerate at once that was shown in âFeature Specific Examplesâ
- Complete NLP example
- Complete computer vision example
- Very complete and extensible vision example showcasing SLURM, hydra, and a very extensible usage of the framework
- Causal language model fine-tuning example
- Masked language model fine-tuning example
- Speech pretraining example
- Translation fine-tuning example
- Text classification fine-tuning example
- Semantic segmentation fine-tuning example
- Question answering fine-tuning example
- Beam search question answering fine-tuning example
- Multiple choice question answering fine-tuning example
- Named entity recognition fine-tuning example
- Image classification fine-tuning example
- Summarization fine-tuning example
- End-to-end examples on how to use AWS SageMaker integration of Accelerate
- Megatron-LM examples for various NLp tasks
Integration Examples
These are tutorials from libraries that integrate with đ€ Accelerate:
Donât find your integration here? Make a PR to include it!
Amphion
- Training Text-to-Speech Models with Amphion
- Training Singing Voice Conversion Models with Amphion
- Training Vocoders with Amphion
Catalyst
DALLE2-pytorch
đ€ diffusers
fastai
- Distributed training from Jupyter Notebooks with fastai
- Basic distributed training examples with fastai
GradsFlow
imagen-pytorch
Kornia
PyTorch Accelerated
PyTorch3D
Stable-Dreamfusion
Tez
trlx
Comfy-UI
In Science
Below contains a non-exhaustive list of papers utilizing đ€ Accelerate.
Donât find your paper here? Make a PR to include it!
- Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy: âPick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationâ, 2023; arXiv:2305.01569.
- Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim: âPlan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelsâ, 2023; arXiv:2305.04091.
- Arthur CĂąmara, Claudia Hauff: âMoving Stuff Around: A study on efficiency of moving documents into memory for Neural IR modelsâ, 2022; arXiv:2205.08343.
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher RĂ©, Ion Stoica, Ce Zhang: âHigh-throughput Generative Inference of Large Language Models with a Single GPUâ, 2023; arXiv:2303.06865.
- Peter Melchior, Yan Liang, ChangHoon Hahn, Andy Goulding: âAutoencoding Galaxy Spectra I: Architectureâ, 2022; arXiv:2211.07890.
- Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, Diyi Yang: âA Cheaper and Better Diffusion Language Model with Soft-Masked Noiseâ, 2023; arXiv:2304.04746.
- Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa: âInstruct-NeRF2NeRF: Editing 3D Scenes with Instructionsâ, 2023; arXiv:2303.12789.
- Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, Andrea Vedaldi: âRealFusion: 360° Reconstruction of Any Object from a Single Imageâ, 2023; arXiv:2302.10663.
- Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, Hongsheng Li: âBetter Aligning Text-to-Image Models with Human Preferenceâ, 2023; arXiv:2303.14420.
- Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang: âHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFaceâ, 2023; arXiv:2303.17580.
- Yue Yang, Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, Jianshu Chen: âZ-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationâ, 2022; arXiv:2210.12261.
- Sheng-Yen Chou, Pin-Yu Chen, Tsung-Yi Ho: âHow to Backdoor Diffusion Models?â, 2022; arXiv:2212.05400.
- Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, Seungryong Kim: âLet 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationâ, 2023; arXiv:2303.07937.
- Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, Daniel Cohen-Or: âLocalizing Object-level Shape Variations with Text-to-Image Diffusion Modelsâ, 2023; arXiv:2303.11306.
- DĂdac SurĂs, Sachit Menon, Carl Vondrick: âViperGPT: Visual Inference via Python Execution for Reasoningâ, 2023; arXiv:2303.08128.
- Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, Qifeng Chen: âFateZero: Fusing Attentions for Zero-shot Text-based Video Editingâ, 2023; arXiv:2303.09535.
- Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, Yejin Choi: âNaturalProver: Grounded Mathematical Proof Generation with Language Modelsâ, 2022; arXiv:2205.12910.
- Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, Daniel Cohen-Or: âTEXTure: Text-Guided Texturing of 3D Shapesâ, 2023; arXiv:2302.01721.
- Puijin Cheng, Li Lin, Yijin Huang, Huaqing He, Wenhan Luo, Xiaoying Tang: âLearning Enhancement From Degradation: A Diffusion Model For Fundus Image Enhancementâ, 2023; arXiv:2303.04603.
- Shun Shao, Yftah Ziser, Shay Cohen: âErasure of Unaligned Attributes from Neural Representationsâ, 2023; arXiv:2302.02997.
- Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, Minjoon Seo: âIn-Context Instruction Learningâ, 2023; arXiv:2302.14691.
- Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, Anima Anandkumar: âPrismer: A Vision-Language Model with An Ensemble of Expertsâ, 2023; arXiv:2303.02506.
- Haoyu Chen, Zhihua Wang, Yang Yang, Qilin Sun, Kede Ma: âLearning a Deep Color Difference Metric for Photographic Imagesâ, 2023; arXiv:2303.14964.
- Van-Hoang Le, Hongyu Zhang: âLog Parsing with Prompt-based Few-shot Learningâ, 2023; arXiv:2302.07435.
- Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui: âDo Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?â, 2023; arXiv:2302.07866.
- Ruoyao Wang, Peter Jansen, Marc-Alexandre CĂŽtĂ©, Prithviraj Ammanabrolu: âBehavior Cloned Transformers are Neurosymbolic Reasonersâ, 2022; arXiv:2210.07382.
- Martin Wessel, TomĂĄĆĄ Horych, Terry Ruas, Akiko Aizawa, Bela Gipp, Timo Spinde: âIntroducing MBIB â the first Media Bias Identification Benchmark Task and Dataset Collectionâ, 2023; arXiv:2304.13148. DOI: [https://dx.doi.org/10.1145/3539618.3591882 10.1145/3539618.3591882].
- Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or: âAttend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelsâ, 2023; arXiv:2301.13826.
- Marcio Fonseca, Yftah Ziser, Shay B. Cohen: âFactorizing Content and Budget Decisions in Abstractive Summarization of Long Documentsâ, 2022; arXiv:2205.12486.
- Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, Daniel Cohen-Or: âTEXTure: Text-Guided Texturing of 3D Shapesâ, 2023; arXiv:2302.01721.
- Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, Yulia Tsvetkov: âOn the Blind Spots of Model-Based Evaluation Metrics for Text Generationâ, 2022; arXiv:2212.10020.
- Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham: âIn-Context Retrieval-Augmented Language Modelsâ, 2023; arXiv:2302.00083.
- Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P. Xing, Hao Zhang: âMPCFormer: fast, performant and private Transformer inference with MPCâ, 2022; arXiv:2211.01452.
- Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, Jianfeng Gao: âGODEL: Large-Scale Pre-Training for Goal-Directed Dialogâ, 2022; arXiv:2206.11309.
- Egil RĂžnningstad, Erik Velldal, Lilja Ăvrelid: âEntity-Level Sentiment Analysis (ELSA): An exploratory task surveyâ, 2023, Proceedings of the 29th International Conference on Computational Linguistics, 2022, pages 6773-6783; arXiv:2304.14241.
- Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, Sergey Levine: âOffline RL for Natural Language Generation with Implicit Language Q Learningâ, 2022; arXiv:2206.11871.
- Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig: âExecution-Based Evaluation for Open-Domain Code Generationâ, 2022; arXiv:2212.10481.
- Minh-Long Luu, Zeyi Huang, Eric P. Xing, Yong Jae Lee, Haohan Wang: âExpeditious Saliency-guided Mix-up through Random Gradient Thresholdingâ, 2022; arXiv:2212.04875.
- Jun Hao Liew, Hanshu Yan, Daquan Zhou, Jiashi Feng: âMagicMix: Semantic Mixing with Diffusion Modelsâ, 2022; arXiv:2210.16056.
- Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao: âLiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learnersâ, 2021; arXiv:2110.06274.