|
--- |
|
license: mit |
|
tags: |
|
- biology |
|
- protein |
|
--- |
|
# PLTNUM-ESM2-HeLa |
|
PLTNUM is a protein language model trained to predict protein half-lives based on their sequences. |
|
This model was created based on [facebook/esm2_t33_650M_UR50D](https://huggingface.co/facebook/esm2_t33_650M_UR50D) and trained on protein half-life dataset of HeLa human cell line ([paper link](https://pubmed.ncbi.nlm.nih.gov/29414762/)). |
|
|
|
### Model Sources |
|
<!-- Provide the basic links for the model. --> |
|
- **Repository:** https://github.com/sagawatatsuya/PLTNUM |
|
- **Paper:** |
|
- **Demo:** https://huggingface.co/spaces/sagawa/PLTNUM |
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
```python |
|
from torch import sigmoid |
|
import torch.nn as nn |
|
from transformers import AutoModel, AutoConfig, PreTrainedModel, AutoTokenizer |
|
|
|
|
|
class PLTNUM_PreTrainedModel(PreTrainedModel): |
|
config_class = AutoConfig |
|
|
|
def __init__(self, config): |
|
super(PLTNUM_PreTrainedModel, self).__init__(config) |
|
self.model = AutoModel.from_pretrained(self.config._name_or_path) |
|
|
|
self.fc_dropout1 = nn.Dropout(0.8) |
|
self.fc_dropout2 = nn.Dropout(0.4) |
|
self.fc = nn.Linear(self.config.hidden_size, 1) |
|
self._init_weights(self.fc) |
|
|
|
def _init_weights(self, module): |
|
if isinstance(module, nn.Linear): |
|
nn.init.normal_(module.weight, mean=0.0, std=self.config.initializer_range) |
|
if module.bias is not None: |
|
nn.init.constant_(module.bias, 0) |
|
elif isinstance(module, nn.Embedding): |
|
nn.init.normal_(module.weight, mean=0.0, std=self.config.initializer_range) |
|
if module.padding_idx is not None: |
|
nn.init.constant_(module.weight[module.padding_idx], 0.0) |
|
elif isinstance(module, nn.LayerNorm): |
|
nn.init.constant_(module.bias, 0) |
|
nn.init.constant_(module.weight, 1.0) |
|
|
|
def forward(self, inputs): |
|
outputs = self.model(**inputs) |
|
last_hidden_state = outputs.last_hidden_state[:, 0] |
|
output = ( |
|
self.fc(self.fc_dropout1(last_hidden_state)) |
|
+ self.fc(self.fc_dropout2(last_hidden_state)) |
|
) / 2 |
|
return output |
|
|
|
def create_embedding(self, inputs): |
|
outputs = self.model(**inputs) |
|
last_hidden_state = outputs.last_hidden_state[:, 0] |
|
return last_hidden_state |
|
|
|
|
|
model = PLTNUM_PreTrainedModel.from_pretrained("sagawa/PLTNUM-ESM2-HeLa") |
|
tokenizer = AutoTokenizer.from_pretrained("sagawa/PLTNUM-ESM2-HeLa") |
|
seq = "MSGRGKQGGKARAKAKTRSSRAGLQFPVGRVHRLLRKGNYSERVGAGAPVYLAAVLEYLTAEILELAGNAARDNKKTRIIPRHLQLAIRNDEELNKLLGRVTIAQGGVLPNIQAVLLPKKTESHHKPKGK" |
|
input = tokenizer( |
|
[seq], |
|
add_special_tokens=True, |
|
max_length=512, |
|
padding="max_length", |
|
truncation=True, |
|
return_offsets_mapping=False, |
|
return_attention_mask=True, |
|
return_tensors="pt", |
|
) |
|
print(sigmoid(model(input))) |
|
``` |
|
|
|
## Citation |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
