|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# ELC-ParserBERT |
|
|
|
This model is an adaptation of the [Every Layer Counts BERT model](<https://aclanthology.org/2023.conll-babylm.20/>), but it incorporates the `Parser Network` from the [StructFormer](<https://arxiv.org/abs/2012.00857>). It was trained for the [BabyLM 2024 challenge](https://babylm.github.io/index.html)'s Strict-Small track. |
|
|
|
## Dataset |
|
|
|
The training data for the challenge can be accessed through OSF [here](https://osf.io/ad7qg/). This model was trained on the 10M token training dataset. |
|
|
|
### Order in Pretraining |
|
|
|
After the segmentation of the data, the segements are ordered in increasing difficulty according to the flesch_reading_ease metric. This ordering can either be maintained by not including the shuffle flag when training or rejected (and allowing shuffling of the data to happen); this model did shuffle the data. |
|
|
|
## Hyperparameters |
|
|
|
### Base Model |
|
|
|
| Hyperparameter | Value | |
|
| -------------- | ----- | |
|
| Initial learning rate | 5e-3 | |
|
| Batch size | 256 | |
|
| Steps | 13495 | |
|
| shuffled | True | |
|
|attention_probs_dropout_prob | 0.1 | |
|
| classifier_dropout | 0.2 | |
|
| hidden_dropout_prob | 0.1 | |
|
| hidden_size | 384 | |
|
| intermediate_size | 1024 | |
|
| layer_norm_eps | 1e-07 | |
|
| max_position_embeddings | 512 | |
|
| num_attention_heads | 6 | |
|
| num_hidden_layers | 12 | |
|
| vocab_size | 16384 | |
|
| n_parser_layers | 4 | |
|
| parser_conv_size |9 | |
|
|
|
### Fine-tuning |
|
|
|
The fine-tuning parameters were unchanged from the organizer outside of following the ELC-BERT model's patience approach for last year, in particular: |
|
|
|
| Hyperparameter | Value | |
|
| -------------- | ----- | |
|
| Initial learning rate | 5e-5 | |
|
| Batch size | 64 | |
|
| Maximum epochs | 10 | |
|
| Evaluate every (epochs) | 1 | |
|
| Patience | 10 (for CoLA, MRPC, RTE, BoolQ, MultiRC, and WSC), 100 (for MNLI, MNLI-MM, QQP, QNLI, and SST-2) | |
|
| Seed | 12 | |
|
|
|
## Credit |
|
|
|
As mentioned above, this model is an adapatation of Every Layer Counts (ELC) BERT and StructFormer, the citations and code repositories for which can be found here |
|
|
|
* StructFormer |
|
* [StructFormer Github](<https://github.com/google-research/google-research/tree/master/structformer>) |
|
|
|
* ```bibtex |
|
@misc{shen2020structformer, |
|
title={StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling}, |
|
author={Yikang Shen and Yi Tay and Che Zheng and Dara Bahri and Donald Metzler and Aaron Courville}, |
|
year={2020}, |
|
eprint={2012.00857}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}}``` |
|
* ELC-BERT: |
|
* [ELC-BERT Github](<https://github.com/ltgoslo/elc-bert>) |
|
* [ELC-BERT 10M Hugging Face](https://huggingface.co/lgcharpe/ELC_BERT_small_baby_10M) |
|
* ```bibtex |
|
@inproceedings{georges-gabriel-charpentier-samuel-2023-layers, |
|
title = "Not all layers are equally as important: Every Layer Counts {BERT}", |
|
author = "Georges Gabriel Charpentier, Lucas and |
|
Samuel, David", |
|
editor = "Warstadt, Alex and |
|
Mueller, Aaron and |
|
Choshen, Leshem and |
|
Wilcox, Ethan and |
|
Zhuang, Chengxu and |
|
Ciro, Juan and |
|
Mosquera, Rafael and |
|
Paranjabe, Bhargavi and |
|
Williams, Adina and |
|
Linzen, Tal and |
|
Cotterell, Ryan", |
|
booktitle = "Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning", |
|
month = dec, |
|
year = "2023", |
|
address = "Singapore", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2023.conll-babylm.20", |
|
doi = "10.18653/v1/2023.conll-babylm.20", |
|
pages = "238--252", |
|
}``` |
|
|
