--model meta-llama/Llama-2-7b-hf --quantization bfloat16 --output_dir ./llama-2-7b-ct2 --force    Loading checkpoint shards: 100%|‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà| 2/2 [00:01<00:00,  1.81it/s]from langchain.llms import CTranslate2llm = CTranslate2(    # output_dir from above:    model_path="./llama-2-7b-ct2",    tokenizer_name="meta-llama/Llama-2-7b-hf",    device="cuda",    # device_index can be either single int or list or ints,    # indicating the ids of GPUs to use for inference:    device_index=[0,1],     compute_type="bfloat16")Single call‚Äãprint(    llm(        "He presented me with plausible evidence for the existence of unicorns: ",        max_length=256,        sampling_topk=50,        sampling_temperature=0.2,        repetition_penalty=2,        cache_static_prompt=False,    ))    He presented me with plausible evidence for the existence of unicorns: 1) they are mentioned in ancient texts; and, more importantly to him (and not so much as a matter that would convince most people), he had seen one.    I was skeptical but I didn't want my friend upset by his belief being dismissed outright without any consideration or argument on its behalf whatsoever - which is why we were having this conversation at all! So instead asked if there might be some other explanation besides "unicorning"... maybe it could have been an ostrich? Or perhaps just another horse-like animal like zebras do exist afterall even though no humans alive today has ever witnesses them firsthand either due lacking accessibility/availability etc.. But then again those animals aren‚Äô t exactly known around here anyway‚Ä¶‚Äù And thus began our discussion about whether these creatures actually existed anywhere else outside Earth itself where only few scientists ventured before us nowadays because technology allows exploration beyond borders once thought impossible centuries ago when travel meant walking everywhere yourself until reaching destination point A->B via
CTranslate2 is a C++ and Python library for efficient inference with Transformer models.
CTranslate2 is a C++ and Python library for efficient inference with Transformer models. ->: --model meta-llama/Llama-2-7b-hf --quantization bfloat16 --output_dir ./llama-2-7b-ct2 --force    Loading checkpoint shards: 100%|‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà‚ñà| 2/2 [00:01<00:00,  1.81it/s]from langchain.llms import CTranslate2llm = CTranslate2(    # output_dir from above:    model_path="./llama-2-7b-ct2",    tokenizer_name="meta-llama/Llama-2-7b-hf",    device="cuda",    # device_index can be either single int or list or ints,    # indicating the ids of GPUs to use for inference:    device_index=[0,1],     compute_type="bfloat16")Single call‚Äãprint(    llm(        "He presented me with plausible evidence for the existence of unicorns: ",        max_length=256,        sampling_topk=50,        sampling_temperature=0.2,        repetition_penalty=2,        cache_static_prompt=False,    ))    He presented me with plausible evidence for the existence of unicorns: 1) they are mentioned in ancient texts; and, more importantly to him (and not so much as a matter that would convince most people), he had seen one.    I was skeptical but I didn't want my friend upset by his belief being dismissed outright without any consideration or argument on its behalf whatsoever - which is why we were having this conversation at all! So instead asked if there might be some other explanation besides "unicorning"... maybe it could have been an ostrich? Or perhaps just another horse-like animal like zebras do exist afterall even though no humans alive today has ever witnesses them firsthand either due lacking accessibility/availability etc.. But then again those animals aren‚Äô t exactly known around here anyway‚Ä¶‚Äù And thus began our discussion about whether these creatures actually existed anywhere else outside Earth itself where only few scientists ventured before us nowadays because technology allows exploration beyond borders once thought impossible centuries ago when travel meant walking everywhere yourself until reaching destination point A->B via