content.raw: Mixture-of-Depths
content.type: text
content.value: Mixture-of-Depths
content.type: new_line
content.type: new_line
content.raw: Dynamically allocating compute in transformer-based language models
content.type: text
content.value: Dynamically allocating compute in transformer-based language models
content.type: new_line
content.type: new_line
content.label: Mixture-of-Depths: Dynamically allocating compute in transformer-based
    language models (2404.02258)
content.raw: https://huggingface.co/papers/2404.02258
content.id: 2404.02258
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2404.02258
content.type: new_line
content.type: new_line
content.raw: Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens (k) that can participate in the self-attention and MLP computations at a given layer. The tokens to be processed are determined by the network using a top-k routing mechanism. Since k is defined a priori, this simple procedure uses a static computation graph with known tensor sizes, unlike other conditional computation techniques. Nevertheless, since the identities of the k tokens are fluid, this method can expend FLOPs non-uniformly across the time and model depth dimensions. Thus, compute expenditure is entirely predictable in sum total, but dynamic and context-sensitive at the token-level. Not only do models trained in this way learn to dynamically allocate compute, they do so efficiently. These models match baseline performance for equivalent FLOPS and wall-clock times to train, but require a fraction of the FLOPs per forward pass, and can be upwards of 50\% faster to step during post-training sampling.
content.type: text
content.value: Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens (k) that can participate in the self-attention and MLP computations at a given layer. The tokens to be processed are determined by the network using a top-k routing mechanism. Since k is defined a priori, this simple procedure uses a static computation graph with known tensor sizes, unlike other conditional computation techniques. Nevertheless, since the identities of the k tokens are fluid, this method can expend FLOPs non-uniformly across the time and model depth dimensions. Thus, compute expenditure is entirely predictable in sum total, but dynamic and context-sensitive at the token-level. Not only do models trained in this way learn to dynamically allocate compute, they do so efficiently. These models match baseline performance for equivalent FLOPS and wall-clock times to train, but require a fraction of the FLOPs per forward pass, and can be upwards of 50\% faster to step during post-training sampling.
content.type: new_line
rawContent: Mixture-of-Depths
  
  Dynamically allocating compute in transformer-based language models
  
  https://huggingface.co/papers/2404.02258
  
  Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens (k) that can participate in the self-attention and MLP computations at a given layer. The tokens to be processed are determined by the network using a top-k routing mechanism. Since k is defined a priori, this simple procedure uses a static computation graph with known tensor sizes, unlike other conditional computation techniques. Nevertheless, since the identities of the k tokens are fluid, this method can expend FLOPs non-uniformly across the time and model depth dimensions. Thus, compute expenditure is entirely predictable in sum total, but dynamic and context-sensitive at the token-level. Not only do models trained in this way learn to dynamically allocate compute, they do so efficiently. These models match baseline performance for equivalent FLOPS and wall-clock times to train, but require a fraction of the FLOPs per forward pass, and can be upwards of 50\% faster to step during post-training sampling.
  
reactions.reaction: ❤️
reactions.users: samusenps taufiqdp clem mick-bitovi iancos csouza nikgr rupeshs sugatoray dillfrescott ceshine
reactions.reaction: 👀
reactions.users: stormchaser
reactions.reaction: 👍
reactions.users: dashfunnydashdash
commentators._id: 6382255fcae34727b9cc149e
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382255fcae34727b9cc149e/PYiwi8LVZParYvImmcGez.png
commentators.fullname: Ram
commentators.name: 0-hero
commentators.type: user