allenai

/

OLMo-1B

like 105

Follow

Ai2 1,336

I'm trying to evaluate OLMo-1B on HuggingFace's evaluation harness. When I try with a command like the following:

lm_eval --model hf \
    --model_args pretrained=allenai/OLMo-1B,revision=step20000-tokens84B \
    --tasks mmlu_business_ethics \
    --device cuda:2 \
    --batch_size auto:4 \
    --output_path /lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/OLMo_1B_84B/mmlu_business_ethics \
    --log_samples

I hit the error:

  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/__main__.py", line 241, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/evaluator.py", line 107, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/api/model.py", line 134, in create_from_arg_string
    return cls(**args, **args2)
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/models/huggingface.py", line 181, in __init__
    self._get_config(
  File "/lfs/skampere1/0/rschaef/KoyejoLab-Predictable-LLM-Evals/submodules/lm-evaluation-harness/lm_eval/models/huggingface.py", line 438, in _get_config
    self._config = transformers.AutoConfig.from_pretrained(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1103, in from_pretrained
    trust_remote_code = resolve_trust_remote_code(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 621, in resolve_trust_remote_code
    raise ValueError(
ValueError: Loading allenai/OLMo-1B requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.

I then try loading OLMO in an interactive python session:

>>> model = AutoModel.from_pretrained("allenai/OLMo-1B", trust_remote_code=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1109, in from_pretrained
    config_class = get_class_from_dynamic_module(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 488, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 315, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 175, in check_imports
    importlib.import_module(imp)
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/hf_olmo/__init__.py", line 1, in <module>
    from .configuration_olmo import OLMoConfig
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/hf_olmo/configuration_olmo.py", line 8, in <module>
    from olmo.config import ModelConfig
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/olmo/__init__.py", line 1, in <module>
    from .config import *
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/olmo/config.py", line 20, in <module>
    from omegaconf import DictConfig, ListConfig
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/__init__.py", line 1, in <module>
    from .base import Container, DictKeyType, Node, SCMode, UnionNode
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/base.py", line 11, in <module>
    from ._utils import (
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/_utils.py", line 34, in <module>
    from .grammar_parser import SIMPLE_INTERPOLATION_PATTERN, parse
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/grammar_parser.py", line 12, in <module>
    from .grammar_visitor import (  # type: ignore
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/grammar_visitor.py", line 25, in <module>
    from omegaconf.grammar.gen.OmegaConfGrammarLexer import OmegaConfGrammarLexer
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/grammar/gen/OmegaConfGrammarLexer.py", line 258, in <module>
    class OmegaConfGrammarLexer(Lexer):
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/omegaconf/grammar/gen/OmegaConfGrammarLexer.py", line 260, in OmegaConfGrammarLexer
    atn = ATNDeserializer().deserialize(serializedATN())
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 28, in deserialize
    self.checkVersion()
  File "/lfs/skampere1/0/rschaef/miniconda3/envs/pred_llm_evals_env/lib/python3.10/site-packages/antlr4/atn/ATNDeserializer.py", line 50, in checkVersion
    raise Exception("Could not deserialize ATN with version " + str(version) + " (expected " + str(SERIALIZED_VERSION) + ").")
Exception: Could not deserialize ATN with version  (expected 4).

What am I doing wrong?