Camarim Logo

Canarim-7B-VestibulAide

For more details on the model, performance test examples, data set, and training process, visit: canar.im or nlp.rocks.

Model Description

Canarim-7B-VestibulAide is a "decoder-only" model with 7 billion parameters, designed specifically for handling questions, exercises, and answers from Brazilian university entrance exams. Tailored for the Portuguese language, its aim is to assist students in understanding and solving complex questions commonly found in these exams.

Usage

from transformers import AutoTokenizer, pipeline
import torch

model_id = "dominguesm/canarim-7b-vestibulaide"

tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

system_message = """Você é um assistente prestativo, respeitoso e honesto, especializado na análise de questões de múltipla escolha, juntamente com a opção considerada correta que sempre responde as perguntas na lingua portugues (portugues). Você oferece uma resposta abrangente, detalhada e bem fundamentada, explicando por que a opção escolhida é a correta. Garanta a abordagem de todos os aspectos relevantes da questão e forneça uma justificativa sólida que exponha com clareza por que a opção selecionada é a resposta correta. Suas resoluções visam proporcionar um entendimento completo da questão, permitindo que os leitores compreendam plenamente o raciocínio subjacente à resposta correta."""

def make_prompt(instruction):
    return (
        f"[INST] <<SYS>>\n{system_message.strip()}\n<</SYS>>\n\n"
        + instruction
        + " [/INST] "
    )

question = """
É comum em viagens de avião sermos solicitados a
desligar aparelhos cujo funcionamento envolva a emissão
ou a recepção de ondas eletromagnéticas, como celulares.
A justificativa dada para esse procedimento é, entre outras
coisas, a necessidade de eliminar fontes de sinais eletro-
magnéticos que possam interferir nas comunicações, via
rádio, dos pilotos com a torre de controle.
Essa interferência poderá ocorrer somente se as ondas
emitidas pelo celular e as recebidas pelo rádio do avião

a) forem ambas audíveis.
b) tiverem a mesma potência.
c) tiverem a mesma frequência.
d) tiverem a mesma intensidade.
e) propagarem-se com velocidades diferentes.
"""

prompt = make_prompt(question)

sequences = pipe(
   prompt,
   do_sample=True,
   num_return_sequences=1,
   eos_token_id=tokenizer.eos_token_id,
   max_length=2048,
   temperature=0.9,
   top_p=0.6,
   repetition_penalty=1.15
)

Question 52 from the blue exam of the first day of Enem 2013.
See the question and answer in English here

Original Solution

Resolução:

Caso os equipamentos de comunicação da aeronave
operem na mesma frequência dos telefones celulares,
pode ocorrer o fenômeno de interferência entre as
radiofrequências utilizadas por esses dispositivos.

Resposta: **C**

Canarim-7B-VestibulAide Solution

Para que haja interferência eletromagnética no avião,
devemos ter duas ondas electromagneticas com vibrações semelhantes,
mas não idênticas, ou seja, devemos ter duas ondas com frequências próximas.

Resposta Correta: **C**

Training Data Set

The data used to train the model consists of a wide range of entrance exams from Brazilian universities and public competitions, spanning multiple years and editions of over 15 distinct entrance exams and 50 public contests. This diversified collection of data from various sources provides a comprehensive understanding of the characteristics and complexity of entrance exam questions over time. Each dataset was meticulously processed and incorporated into the model's training, ensuring the representation of questions found in actual entrance exams.

To view the complete list of years and exams used, access the full list of entrance exams.

Performance

The model's performance was assessed by its ability to provide accurate suggestions for multiple-choice questions related to the ENEM 2022 examination.

ENEM 2022 and 2023 Evaluation

This evaluation focused on the model's ability to accurately suggest choices for multiple-choice questions from the ENEM 2022 and ENEM 2023 exams. The model's success was assessed based on its capability to recommend options that matched the correct answers to the questions.

For the evaluation, the model was tested using the ENEM 2022 exam dataset, which consists of 84 multiple-choice questions. The model achieved an accuracy of 35.71% in correctly suggesting answers, accurately answering 30 out of the 84 questions.

Next, the model was evaluated using the ENEM 2023 - DAY 1 exam dataset, comprising 90 multiple-choice questions. Here, the model demonstrated an accuracy of 43.33% in suggesting correct choices, correctly answering 39 of the 90 questions.

The dataset used for calculating this metric is available at: canar.im

Use and Limitations

Intended Use

This model is intended for students aiming to enhance their skills in solving university entrance exam questions. It can be used in the following ways:

Solution Generation: The model can produce step-by-step solutions for specific questions, aiding students in grasping the solution process and the underlying concepts.
Review and Study: The model can be used for reviewing and studying various topics covered in entrance exam questions, providing detailed explanations when needed.

Limitations

The model performs notably in generating solutions, offering in-depth explanations on the solution process of entrance exam questions. However, it's essential to point out that it may currently have some limitations in suggesting the correct option in multiple-choice questions.

The quality of suggestions for the right choice may not always meet the desired standards, possibly resulting in inaccurate or inappropriate answers (even when the solution/explanation is accurate). It's vital to stress that I'm aware of these limitations and am committed to enhancing this ability in future model versions.

I'm dedicated to investing time and effort to improve the quality of the correct option suggestions, but this might take a while. For now, the model can be used to generate solutions and in-depth explanations, but it's advised that users verify the right choice independently.

I appreciate the understanding of all model users and value everyone's feedback. If you have any suggestions or comments, please do not hesitate to reach out to me.

How to Cite

If you want to cite Canarim-7B-VestibulAide, you could use this:

@misc {maicon_domingues_2023,
    author       = { {Maicon Domingues} },
    title        = { canarim-7b-vestibulaide (Revision 2fb86c2) },
    year         = 2023,
    url          = { https://huggingface.co/dominguesm/canarim-7b-vestibulaide },
    doi          = { 10.57967/hf/1357 },
    publisher    = { Hugging Face }
}

Citations

@misc{touvron2023llama,
      title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
      author={Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton Ferrer and Moya Chen and Guillem Cucurull and David Esiobu and Jude Fernandes and Jeremy Fu and Wenyin Fu and Brian Fuller and Cynthia Gao and Vedanuj Goswami and Naman Goyal and Anthony Hartshorn and Saghar Hosseini and Rui Hou and Hakan Inan and Marcin Kardas and Viktor Kerkez and Madian Khabsa and Isabel Kloumann and Artem Korenev and Punit Singh Koura and Marie-Anne Lachaux and Thibaut Lavril and Jenya Lee and Diana Liskovich and Yinghai Lu and Yuning Mao and Xavier Martinet and Todor Mihaylov and Pushkar Mishra and Igor Molybog and Yixin Nie and Andrew Poulton and Jeremy Reizenstein and Rashi Rungta and Kalyan Saladi and Alan Schelten and Ruan Silva and Eric Michael Smith and Ranjan Subramanian and Xiaoqing Ellen Tan and Binh Tang and Ross Taylor and Adina Williams and Jian Xiang Kuan and Puxin Xu and Zheng Yan and Iliyan Zarov and Yuchen Zhang and Angela Fan and Melanie Kambadur and Sharan Narang and Aurelien Rodriguez and Robert Stojnic and Sergey Edunov and Thomas Scialom},
      year={2023},
      eprint={2307.09288},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

dominguesm
/

canarim-7b-vestibulaide