German-English Code-Switching BERT

A BERT-based model trained with masked language modelling on a large corpus of German--English code-switching. It was introduced in this paper. This model is case sensitive.

Overview

Initialized language model: bert-base-multilingual-cased
Training data: The TongueSwitcher Corpus
Infrastructure: 4x Nvidia A100 GPUs
Published: 16 October 2023

Hyperparameters

batch_size = 32
epochs = 1
n_steps = 191,950
max_seq_len = 512
learning_rate = 1e-4
weight_decay = 0.01
Adam beta = (0.9, 0.999)
lr_schedule = LinearWarmup
num_warmup_steps = 10,000
seed = 2021

Performance

During training we monitored the evaluation loss on the TongueSwitcher dev set.

Authors

Igor Sterner: is473 [at] cam.ac.uk
Simone Teufel: sht25 [at] cam.ac.uk

BibTeX entry and citation info

@inproceedings{sterner2023tongueswitcher,
  author    = {Igor Sterner and Simone Teufel},
  title     = {TongueSwitcher: Fine-Grained Identification of German-English Code-Switching},
  booktitle = {Sixth Workshop on Computational Approaches to Linguistic Code-Switching},
  publisher = {Empirical Methods in Natural Language Processing},
  year      = {2023},
}