Body: <p>This was pointed out to me: <a href="https://huggingface.co/blog/accelerate-large-models" rel="nofollow noreferrer">https://huggingface.co/blog/accelerate-large-models</a></p>
  <p>and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution script.</p>
  <p>Does anyone know of a good library, or example code that implements Accelerate for large models to use less VRAM / RAM?</p>
  <p>This is what I tried to run last:</p>
  <pre><code>from accelerate import infer_auto_device_map, init_empty_weights
  
  from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
  
  import torch
  
  checkpoint = &quot;bigscience/mt0-small&quot;
  
  config = AutoConfig.from_pretrained(checkpoint)
  
  with init_empty_weights():
  
  model = AutoModelForCausalLM.from_config(config)
  
  device_map = infer_auto_device_map(model, no_split_module_classes=[&quot;OPTDecoderLayer&quot;], dtype=&quot;float16&quot;)
  
  tokenizer = AutoTokenizer.from_pretrained(checkpoint)
  
  device_map[&quot;model.decoder.layers.37&quot;] = &quot;disk&quot;
  
  model = AutoModelForCausalLM.from_pretrained(
  
  checkpoint, device_map=device_map, offload_folder=&quot;offload&quot;, offload_state_dict = True, torch_dtype=torch.float16
  
  )
  </code></pre>
  <p>Goal is to run <strong>Flan-T5</strong> or <strong>BLOOMZ</strong> locally, using disk space, to run the models on lower resources.</p>
  <p>This code throws the error:</p>
  <pre><code>ValueError: Unrecognized configuration class &lt;class 'transformers.models.mt5.configuration_mt5.MT5Config'&gt; for this kind of AutoModel: AutoModelForCausalLM
  </code></pre>
  <p>That is another issue, it would be good to know how to supply a configuration or convert the config to a supported type.
  And not sure if the code I put together would work once that is addressed.</p>
  <p>Any suggestions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-05T16:34:35.997Z
Comments.Text: hi, do you solved? same problem here, i found another strategy bud did not work for me https://huggingface.co/docs/accelerate/usage_guides/big_modeling
Comments.UserDisplayName: Danilow
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-06T17:43:37.610Z
Comments.Text: Still foggy on this, but thanks for posting the link. It could be pretty useful. These days I would be most likely to use something like GPT4ALL or  oobabooga's webui if possible
Comments.UserDisplayName: jmhead
Tags: <machine-learning><language-model>
Title: How to best implement Accelerate for vary large models
UserDisplayName: jmhead
ThreadText: # How to best implement Accelerate for vary large models
  
  (Asked by: username_0 on 2023-02-28)
  
  This was pointed out to me: <https://huggingface.co/blog/accelerate-large-models>
  
  and seems it could be a great resource, but it is broken down into steps as a tutorial vs as out of the box solution script.
  
  Does anyone know of a good library, or example code that implements Accelerate for large models to use less VRAM / RAM?
  
  This is what I tried to run last:
  
  ```
  from accelerate import infer_auto_device_map, init_empty_weights
  
  from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
  
  import torch
  
  checkpoint = "bigscience/mt0-small"
  
  config = AutoConfig.from_pretrained(checkpoint)
  
  with init_empty_weights():
  
  model = AutoModelForCausalLM.from_config(config)
  
  device_map = infer_auto_device_map(model, no_split_module_classes=["OPTDecoderLayer"], dtype="float16")
  
  tokenizer = AutoTokenizer.from_pretrained(checkpoint)
  
  device_map["model.decoder.layers.37"] = "disk"
  
  model = AutoModelForCausalLM.from_pretrained(
  
  checkpoint, device_map=device_map, offload_folder="offload", offload_state_dict = True, torch_dtype=torch.float16
  
  )
  
  ```
  
  Goal is to run **Flan-T5** or **BLOOMZ** locally, using disk space, to run the models on lower resources.
  
  This code throws the error:
  
  ```
  ValueError: Unrecognized configuration class <class 'transformers.models.mt5.configuration_mt5.MT5Config'> for this kind of AutoModel: AutoModelForCausalLM
  
  ```
  
  That is another issue, it would be good to know how to supply a configuration or convert the config to a supported type.
  And not sure if the code I put together would work once that is addressed.
  
  Any suggestions?
  
  **Tags:** machine-learning, language-model
  
  > **Comments:**
  > **username_1**: hi, do you solved? same problem here, i found another strategy bud did not work for me https://huggingface.co/docs/accelerate/usage_guides/big_modeling  
  > **username_0**: Still foggy on this, but thanks for posting the link. It could be pretty useful. These days I would be most likely to use something like GPT4ALL or  oobabooga's webui if possible  
  
  ---
  
  ## 0 answers
  
  