Hello @yosiasz , welcome!
Amharic is definitely an under-served language and one for which we’re working on improving coverage, but we do have some resources to get you started.
First, you can look at the list of datasets that have some Amharic text here 12
For example, you can get about 28M words of Amharic text from the CommonCrawl in the OSCAR dataset 6 with:
import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])

We don’t have any models trained exclusively on Amharic so far, but we do have a few translation models as well as a Language-Agnostic Sentence Encoder 1 you can check out:
import torch
from transformers import BertModel, BertTokenizerFast


tokenizer = BertTokenizerFast.from_pretrained("setu4993/LaBSE")
model = BertModel.from_pretrained("setu4993/LaBSE")
model = model.eval()

am_sentences = oscar_am["train"][:4]["text"]
am_inputs = tokenizer(am_sentences , truncation=True, return_tensors="pt", padding=True)

with torch.no_grad():
    am_outputs = model(**am_inputs )

This will give you vector representations of all of the texts in am_sentences
Hope that helps get you started, would you like to tell us more about what you want to do in Amharic so we can help further?