answerdotai

/

ModernBERT-base

like 716

Follow

Answer.AI 374

I think, this error happens when Python 3.9 (AWS Lambda’s default) clashes with ModernBERT’s requirement for Python ≥3.10. Try this:

Upgrade Lambda to Python 3.10/3.11.

Add these lines before loading the model:

import os
os.environ["OMP_NUM_THREADS"] = "1" # Disable multiprocessing
os.environ["TOKENIZERS_PARALLELISM"] = "false"
model = AutoModel.from_config(config, cache_dir="/tmp")

Also be sure to include ModernBERT files directly in your Lambda deployment (AWS might not see the repo code)

Thanks for the suggestions @khusrav13 .

I'm running a docker container with python 3.10 already. So that part is fine. I tried the other flags you suggested but getting the same error.

AutoModel.from_config(..) for modernbert tries to import transformers.models.modernbert.modeling_modernbert. This in turn tries to do a torch dynamic compilation, which uses multiprocessing, which isn't supported on AWS Lambda. See the stack below when trying to import transformers.models.modernbert.modeling_modernbert:

"  File \"/workspace/lambda.py\", line 2, in <module>\n    import transformers.models.modernbert.modeling_modernbert\n",
    "  File \"/workspace/transformers/models/modernbert/modeling_modernbert.py\", line 194, in <module>\n    class ModernBertEmbeddings(nn.Module):\n",
    "  File \"/workspace/transformers/models/modernbert/modeling_modernbert.py\", line 207, in ModernBertEmbeddings\n    def compiled_embeddings(self, input_ids: torch.LongTensor) -> torch.Tensor:\n",
    "  File \"/workspace/torch/init.py\", line 1808, in fn\n    return compile(model,\n",
    "  File \"/workspace/torch/init.py\", line 1826, in compile\n    return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)\n",
    "  File \"/workspace/torch/_dynamo/eval_frame.py\", line 795, in optimize\n    compiler_config=backend.get_compiler_config()\n",
    "  File \"/workspace/torch/init.py\", line 1671, in get_compiler_config\n    from torch._inductor.compile_fx import get_patched_config_dict\n",
    "  File \"/workspace/torch/_inductor/compile_fx.py\", line 37, in <module>\n    from torch._inductor.codecache import code_hash, CompiledFxGraph, FxGraphCache\n",
    "  File \"/workspace/torch/_inductor/codecache.py\", line 2492, in <module>\n    AsyncCompile.warm_pool()\n",
    "  File \"/workspace/torch/_inductor/codecache.py\", line 2402, in warm_pool\n    pool = cls.process_pool()\n",
    "  File \"/workspace/torch/_inductor/codecache.py\", line 2385, in process_pool\n    pool = ProcessPoolExecutor(\n",
    "  File \"/var/lang/lib/python3.10/concurrent/futures/process.py\", line 662, in init\n    self._call_queue = _SafeQueue(\n",
    "  File \"/var/lang/lib/python3.10/concurrent/futures/process.py\", line 168, in init\n    super().init(max_size, ctx=ctx)\n",
    "  File \"/var/lang/lib/python3.10/multiprocessing/queues.py\", line 43, in init\n    self._rlock = ctx.Lock()\n",
    "  File \"/var/lang/lib/python3.10/multiprocessing/context.py\", line 68, in Lock\n    return Lock(ctx=self.get_context())\n",
    "  File \"/var/lang/lib/python3.10/multiprocessing/synchronize.py\", line 162, in init\n    SemLock.init(self, SEMAPHORE, 1, 1, ctx=ctx)\n",
    "  File \"/var/lang/lib/python3.10/multiprocessing/synchronize.py\", line 57, in init\n    sl = self._semlock = _multiprocessing.SemLock(\n"

I believe the following code in modeling_moderbert.py is causing the dynamic compilation:

    

@torch
	.compile(dynamic=True)
    def compiled_embeddings(self, input_ids: torch.LongTensor) -> torch.Tensor:
        return self.drop(self.norm(self.tok_embeddings(input_ids)))

This workaround (disabling dynamo) got rid of the error:

        os.environ['TORCHDYNAMO_DISABLE'] = '1'

But what is the right fix here?

Hello!
I reckon the "proper" fix is to use reference_compile=False when loading the AutoModel or AutoConfig. That disabled the use of the compilation step.

• Tom Aarsen