Papers
arxiv:2308.06103

Composable Function-preserving Expansions for Transformer Architectures

Published on Aug 11, 2023
· Submitted by akhaliq on Aug 14, 2023
#3 Paper of the day
Authors:
,

Abstract

Training state-of-the-art neural networks requires a high cost in terms of compute and time. Model scale is recognized to be a critical factor to achieve and improve the state-of-the-art. Increasing the scale of a neural network normally requires restarting from scratch by randomly initializing all the parameters of the model, as this implies a change of architecture's parameters that does not allow for a straightforward transfer of knowledge from smaller size models. In this work, we propose six composable transformations to incrementally increase the size of transformer-based neural networks while preserving functionality, allowing to expand the capacity of the model as needed. We provide proof of exact function preservation under minimal initialization constraints for each transformation. The proposed methods may enable efficient training pipelines for larger and more powerful models by progressively expanding the architecture throughout training.

Community

I was thinking about how humans learn by training small models (baby brains), and then having brains expand as they then get more data, and how that could be applied to AI! This seems like a super interesting approach! Congrats to the authors!

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2308.06103 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2308.06103 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2308.06103 in a Space README.md to link it from this page.

Collections including this paper 2