<p>This was pointed out to me: <a href="https://huggingface.co/blog/accelerate-large-models" rel="nofollow noreferrer">https://huggingface.co/blog/accelerate-large-models</a></p>
<p>and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution script.</p>
<p>Does anyone know of a good library, or example code that implements Accelerate for large models to use less VRAM / RAM?</p>
<p>This is what I tried to run last:</p>
<pre><code>from accelerate import infer_auto_device_map, init_empty_weights

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

import torch

checkpoint = &quot;bigscience/mt0-small&quot;

config = AutoConfig.from_pretrained(checkpoint)

with init_empty_weights():

model = AutoModelForCausalLM.from_config(config)

device_map = infer_auto_device_map(model, no_split_module_classes=[&quot;OPTDecoderLayer&quot;], dtype=&quot;float16&quot;)

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

device_map[&quot;model.decoder.layers.37&quot;] = &quot;disk&quot;

model = AutoModelForCausalLM.from_pretrained(

checkpoint, device_map=device_map, offload_folder=&quot;offload&quot;, offload_state_dict = True, torch_dtype=torch.float16

)
</code></pre>
<p>Goal is to run <strong>Flan-T5</strong> or <strong>BLOOMZ</strong> locally, using disk space, to run the models on lower resources.</p>
<p>This code throws the error:</p>
<pre><code>ValueError: Unrecognized configuration class &lt;class 'transformers.models.mt5.configuration_mt5.MT5Config'&gt; for this kind of AutoModel: AutoModelForCausalLM
</code></pre>
<p>That is another issue, it would be good to know how to supply a configuration or convert the config to a supported type.
And not sure if the code I put together would work once that is addressed.</p>
<p>Any suggestions?</p>

CC BY-SA 4.0
2023-06-05T16:34:35.997Z
hi, do you solved? same problem here, i found another strategy bud did not work for me https://huggingface.co/docs/accelerate/usage_guides/big_modeling
Danilow
CC BY-SA 4.0
2023-06-06T17:43:37.610Z
Still foggy on this, but thanks for posting the link. It could be pretty useful. These days I would be most likely to use something like GPT4ALL or  oobabooga's webui if possible
jmhead
<machine-learning><language-model>
How to best implement Accelerate for vary large models
jmhead
# How to best implement Accelerate for vary large models

(Asked by: username_0 on 2023-02-28)

This was pointed out to me: https://huggingface.co/blog/accelerate-large-models
and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution script.
Does anyone know of a good library, or example code that implements Accelerate for large models to use less VRAM / RAM?
This is what I tried to run last:
from accelerate import infer_auto_device_map, init_empty_weights

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

import torch

checkpoint = "bigscience/mt0-small"

config = AutoConfig.from_pretrained(checkpoint)

with init_empty_weights():

model = AutoModelForCausalLM.from_config(config)

device_map = infer_auto_device_map(model, no_split_module_classes=["OPTDecoderLayer"], dtype="float16")

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

device_map["model.decoder.layers.37"] = "disk"

model = AutoModelForCausalLM.from_pretrained(

checkpoint, device_map=device_map, offload_folder="offload", offload_state_dict = True, torch_dtype=torch.float16

)

Goal is to run Flan-T5 or BLOOMZ locally, using disk space, to run the models on lower resources.
This code throws the error:
ValueError: Unrecognized configuration class  for this kind of AutoModel: AutoModelForCausalLM

That is another issue, it would be good to know how to supply a configuration or convert the config to a supported type.
And not sure if the code I put together would work once that is addressed.
Any suggestions?

**Tags:** machine-learning, language-model

> **Comments:**
> **username_1**: hi, do you solved? same problem here, i found another strategy bud did not work for me https://huggingface.co/docs/accelerate/usage_guides/big_modeling  
> **username_0**: Still foggy on this, but thanks for posting the link. It could be pretty useful. These days I would be most likely to use something like GPT4ALL or  oobabooga's webui if possible  

---

## 0 answers

