sabia-7b / README.md
rodrigo-nogueira's picture
Update README.md
629d642
|
raw
history blame
2.19 kB
metadata
language:
  - pt

Sabiá-7B is Portuguese language model developed by Maritaca AI.

Input: The model accepts only text input.

Output: The Model generates text only.

Model Architecture: Sabiá-7B 2 is an auto-regressive language model that uses the same architecture of LLaMA-1-7B.

Tokenizer: It uses the same tokenizer as LLaMA-1-7B.

Maximum sequence length: 2048 tokens.

Pretraining data: The model was pretrained on 7 billion tokens from the Portuguese subset of ClueWeb22, starting with the weights of LLaMA-1-7B and further trained for an additional 10 billion tokens, approximately 1.4 epochs of the training dataset.

Data Freshness: The pretraining data has a cutoff of mid-2022.

License: The licensing is the same as LLaMA-1's, restricting the model's use to research purposes only.

Paper: For more details, please refer to our paper: Sabiá: Portuguese Large Language Models

Given that Sabiá-7B was trained solely on a language modeling objective without fine-tuning for instruction following, it is recommended for few-shot tasks rather than zero-shot tasks.

Results in Portuguese

Below we show the results on the Poeta benchmark, which consists of 14 Portuguese datasets.

For more information on the Normalized Preferred Metric (NPM), please refer to our paper.

Model NPM
LLaMA-1-7B 33.0
LLaMA-2-7B 43.7
Sabiá-7B 48.5

Results in English

Below we show the average results on 6 English datasets: PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OpenBookQA.

Model NPM
LLaMA-1-7B 50.1
Sabiá-7B 49.0

Please use the following bibtex to cite our paper:

@InProceedings{10.1007/978-3-031-45392-2_15,
    author="Pires, Ramon
    and Abonizio, Hugo
    and Almeida, Thales Sales
    and Nogueira, Rodrigo",
    editor="Naldi, Murilo C.
    and Bianchi, Reinaldo A. C.",
    title="Sabi{\'a}: Portuguese Large Language Models",
    booktitle="Intelligent Systems",
    year="2023",
    publisher="Springer Nature Switzerland",
    address="Cham",
    pages="226--240",
    isbn="978-3-031-45392-2"
}