init commit

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Alibaba Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,144 @@
 ---
+language: en
+tags:
+- speech quantization
 license: mit
+datasets:
+- LibriTTS
 ---
+
+# Highlights
+This model is used for speech codec or quantization on English utterances.
+- Frequency domain model, fully using short-time structures of speech signals
+- Fewer model parameters (4.50 M), lower computation complexity (2.18G flops) while maintaining comparable speech quality (ViSQOL score 3.81 ~ 4.31)
+- Training with structured dropout, enabling various band widths during inference with a single model
+- Quantizing a raw speech waveform into a sequence of discrete tokens
+
+# FunCodec model
+This model is trained with [FunCodec](https://github.com/alibaba-damo-academy/FunCodec), 
+an open-source toolkits for speech quantization (codec) from the Damo academy, Alibaba Group.
+This repository provides a pre-trained model on the LibriTTS corpus.
+It can be applied to low-band-width speech communication, speech quantization, zero-shot speech synthesis 
+and other academic research topics.
+Compared with [EnCodec](https://arxiv.org/abs/2210.13438) and [SoundStream](https://arxiv.org/abs/2107.03312), 
+the following improved techniques are utilized to train the model, resulting in higher codec quality and 
+[ViSQOL](https://github.com/google/visqol) scores under the same band width:
+- The magnitude spectrum loss is employed to enhance the middle and high frequency signals
+- Structured dropout is employed to smooth the code space, as well as enable various band widths in a single model
+- Codes are initialized by k-means clusters rather than random values
+- Codebooks are maintained with exponential moving average and dead-code-elimination mechanism, resulting in high utilization factor for codebooks.
+
+## Model description
+This model is a variational autoencoder that uses residual vector quantisation (RVQ) to obtain 
+several parallel sequences of discrete latent representations. Here is an overview of FunCodec models.
+<p align="center">
+<img src="fig/framework.png" alt="FunCodec architecture"/>
+</p>
+
+In general, FunCodec models consist of five modules: a domain transformation module, 
+an encoder, a RVQ module, a decoder and a domain inversion module.
+- Domain Transformation：transfer signals into time domain, short-time frequency domain, magnitude-angle domain or magnitude-phase domain.
+- Encoder：encode signals into compact representations with stacked convolutional and LSTM layers.
+- Semantic tokens (Optional): augment encoder outputs with semantic tokens to enhance the content information, not used in this model.
+- RVQ：quantize the representations into parallel sequences of discrete tokens with cascaded vector quantizers.
+- Decoder：decode quantized embeddings into different signal domains the same as inputs.
+- Domain Inversion：re-synthesize perceptible waveforms from different domains.
+
+More details can be found at：
+- Paper: [FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec](https://arxiv.org/abs/2309.07405)
+- Codebase: [FunCodec](https://github.com/alibaba-damo-academy/FunCodec)
+
+## Intended uses & sceneries
+### Inference with FunCodec
+
+You can extract codecs and reconstruct them back to waveforms with FunCodec repository.
+
+#### FunCodec installation
+```sh
+# Install Pytorch GPU (version >= 1.12.0):
+conda install pytorch==1.12.0
+# for other versions, please refer: https://pytorch.org/get-started/locally
+
+# Download codebase:
+git clone https://github.com/alibaba-damo-academy/FunCodec.git
+
+# Install FunCodec codebase:
+cd FunCodec
+pip install --editable ./
+```
+
+#### Codec extraction
+```sh
+# Enter the example directory 
+cd egs/LibriTTS/codec
+# Specify the model name
+model_name="audio_codec-encodec-en-libritts-16k-nq32ds320-pytorch"
+# Download the model
+git lfs install
+git clone https://huggingface.co/alibaba-damo/${model_name}
+mkdir exp
+mv ${model_name} exp/$model_name
+# Extracting codec within the input file "input_wav.scp" and the codecs are saved under "outputs/codecs"
+bash encoding_decoding.sh --stage 1 --batch_size 16 --num_workers 4 --gpu_devices "0,1" \
+ --model_dir exp/${model_name} --bit_width 16000 --file_sampling_rate 16000 \
+ --wav_scp input_wav.scp --out_dir outputs/codecs
+# input_wav.scp has the following format:
+# uttid1 path/to/file1.wav
+# uttid2 path/to/file2.wav
+# ...
+```
+
+### Reconstruct waveforms from codecs
+```shell
+# Reconstruct waveforms into "outputs/recon_wavs"
+bash encoding_decoding.sh --stage 2 --batch_size 16 --num_workers 4 --gpu_devices "0,1" \
+ --model_dir exp/${model_name} --bit_width 16000 --file_sampling_rate 16000 \
+ --wav_scp outputs/codecs/codecs.txt --out_dir outputs/recon_wavs 
+# codecs.txt is the output of stage 1, which has the following format：
+# uttid1 [[[1, 2, 3, ...],[2, 3, 4, ...], ...]]
+# uttid2 [[[9, 7, 5, ...],[3, 1, 2, ...], ...]]
+# ...
+```
+
+### Inference with Huggingface Transformers
+Inference with Huggingface transformers package is under development.
+
+### Application sceneries
+Running environment
+- Currently, the model only passed the tests on Linux-x86_64. Mac and Windows systems are not tested.
+
+Intended using sceneries
+- This model is suitable for academic usages
+- Speech quantization, codec and tokenization for English utterances
+
+## Evaluation results
+
+### Training configuration
+- Feature info: raw waveform input
+- Train info: Adam, lr 3e-4, batch_size 32, 2 gpu(Tesla V100), acc_grad 1, 300000 steps, speech_max_length 51200
+- Loss info: L1, L2, discriminative loss
+- Model info: SEANet, Conv, LSTM
+- Train config: config.yaml
+- Model size: 4.50 M parameters
+
+### Experimental Results
+Test set: LibriTTS-test, ViSQOL scores
+
+| testset | 50 tk/s | 100 tk/s | 200 tk/s | 400 tk/s |
+|:--------:|:--------:|:--------:|:--------:|:--------:|
+| LibriTTS | 3.32 | 3.81 | 4.11 | 4.31 |
+
+### Limitations and bias
+- Not very robust to background noises and reverberation
+
+### BibTeX entry and citation info
+```BibTeX
+@misc{du2023funcodec,
+ title={FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec},
+ author={Zhihao Du, Shiliang Zhang, Kai Hu, Siqi Zheng},
+ year={2023},
+ eprint={2309.07405},
+ archivePrefix={arXiv},
+ primaryClass={cs.Sound}
+}
+```

config.yaml

@@ -0,0 +1,226 @@
+config: conf/encodec2d_dwconv_groupratio8_mag_phase_16k_n32_600k_step_rmseg_use_power.yaml
+print_config: false
+log_level: INFO
+dry_run: false
+iterator_type: sequence
+output_dir: exp/encodec2d_dwconv_groupratio8_mag_phase_16k_n32_600k_step_rmseg_use_power_raw_en_libritts
+ngpu: 2
+seed: 0
+num_workers: 8
+num_att_plot: 0
+dist_backend: nccl
+dist_init_method: env://
+dist_world_size: null
+dist_rank: null
+local_rank: 0
+dist_master_addr: null
+dist_master_port: null
+dist_launcher: null
+multiprocessing_distributed: true
+unused_parameters: true
+sharded_ddp: false
+cudnn_enabled: true
+cudnn_benchmark: false
+cudnn_deterministic: false
+collect_stats: false
+write_collected_feats: false
+max_epoch: 60
+max_update: 9223372036854775807
+patience: null
+val_scheduler_criterion:
+- valid
+- loss
+early_stopping_criterion:
+- valid
+- loss
+- min
+best_model_criterion:
+- - valid
+ - generator_multi_spectral_recon_loss
+ - min
+keep_nbest_models: 60
+nbest_averaging_interval: 0
+grad_clip: -1
+grad_clip_type: 2.0
+grad_noise: false
+accum_grad: 1
+no_forward_run: false
+resume: true
+train_dtype: float32
+use_amp: false
+log_interval: 50
+use_tensorboard: true
+use_wandb: false
+wandb_project: null
+wandb_id: null
+wandb_entity: null
+wandb_name: null
+wandb_model_log_interval: -1
+detect_anomaly: false
+pretrain_path: null
+init_param: []
+ignore_init_mismatch: true
+freeze_param: []
+num_iters_per_epoch: 10000
+batch_size: 32
+valid_batch_size: null
+batch_bins: 2000000
+valid_batch_bins: null
+drop_last: true
+train_shape_file:
+- exp/tokenizer_states_16k/train/speech_shape
+valid_shape_file:
+- exp/tokenizer_states_16k/dev/speech_shape
+batch_type: unsorted
+valid_batch_type: null
+speech_length_min: -1
+speech_length_max: -1
+fold_length:
+- 512
+- 150
+sort_in_batch: descending
+sort_batch: descending
+multiple_iterator: false
+chunk_length: 500
+chunk_shift_ratio: 0.5
+num_cache_chunks: 1024
+dataset_type: small
+dataset_conf: {}
+train_data_file: null
+valid_data_file: null
+train_data_path_and_name_and_type:
+- - dump/raw_16k/train/wav.scp.pai
+ - speech
+ - kaldi_ark
+valid_data_path_and_name_and_type:
+- - dump/raw_16k/dev/wav.scp.pai
+ - speech
+ - kaldi_ark
+allow_variable_data_keys: false
+max_cache_size: 0.0
+max_cache_fd: 32
+valid_max_cache_size: null
+optim: adam
+optim_conf:
+ lr: 0.0003
+ betas:
+ - 0.5
+ - 0.9
+scheduler: null
+scheduler_conf:
+ step_size: 8
+ gamma: 0.1
+optim2: adam
+optim2_conf:
+ lr: 0.0003
+ betas:
+ - 0.5
+ - 0.9
+scheduler2: null
+scheduler2_conf:
+ step_size: 8
+ gamma: 0.1
+use_pai: true
+simple_ddp: false
+num_worker_count: 1
+generator_first: false
+input_size: 3
+cmvn_file: null
+disc_grad_clip: -1
+disc_grad_clip_type: 2.0
+gen_train_interval: 1
+disc_train_interval: 1
+stat_flops: false
+use_preprocessor: true
+speech_volume_normalize: null
+speech_rms_normalize: false
+speech_max_length: 40800
+sampling_rate: 16000
+valid_max_length: 40800
+frontend: null
+frontend_conf: {}
+normalize: null
+normalize_conf: {}
+encoder: encodec_seanet_encoder_2d
+encoder_conf:
+ ratios:
+ - - 4
+ - 1
+ - - 4
+ - 1
+ - - 4
+ - 2
+ - - 4
+ - 1
+ norm: time_group_norm
+ norm_params:
+ num_groups: 1
+ causal: false
+ dilation_base: 2
+ kernel_size: 3
+ last_kernel_size: 3
+ seq_model: none
+quantizer: costume_quantizer
+quantizer_conf:
+ codebook_size: 1024
+ num_quantizers: 32
+ ema_decay: 0.99
+ kmeans_init: true
+ sampling_rate: 16000
+ quantize_dropout: true
+ rand_num_quant:
+ - 1
+ - 2
+ - 4
+ - 8
+ - 16
+ - 32
+ use_ddp: true
+ encoder_hop_length: 320
+decoder: encodec_seanet_decoder_2d
+decoder_conf:
+ ratios:
+ - - 4
+ - 1
+ - - 4
+ - 1
+ - - 4
+ - 2
+ - - 4
+ - 1
+ norm: time_group_norm
+ norm_params:
+ num_groups: 1
+ causal: false
+ channels: 3
+ dilation_base: 2
+ kernel_size: 3
+ last_kernel_size: 3
+ tr_conv_group_ratio: 8
+ seq_model: none
+model: freq_codec
+model_conf:
+ odim: 128
+ multi_spectral_window_powers_of_two:
+ - 5
+ - 6
+ - 7
+ - 8
+ - 9
+ - 10
+ target_sample_hz: 16000
+ audio_normalize: true
+ segment_dur: null
+ overlap_ratio: null
+ use_power_spec_loss: true
+ codec_domain:
+ - mag_phase
+ - mag_phase
+discriminator: multiple_disc
+discriminator_conf:
+ input_size: 1
+ disc_conf_list:
+ - filters: 32
+ name: encodec_multi_scale_stft_discriminator
+distributed: true
+version: 0.2.0

model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8aff94a8efc27157df3b1fa33a8e4da5cc3eaceb79f4e0c13a697c33e7cbdfd9
+size 53762961