ClinicalGPT-Pubmed-Instruct-V1.0

Overview

ClinicalGPT-Pubmed-Instruct-V1.0 is a specialized language model fine-tuned on the mistralai/Mistral-7B-Instruct-v0.2 base model. While primarily trained on 10 million PubMed abstracts and titles, this model excels at generating responses to life science-related medical questions with relevant citations from various scientific sources.

Key Features

• Built on Mistral-7B-Instruct-v0.2 base model
• Primary training on 10M PubMed abstracts and titles
• Generates answers with scientific citations from multiple sources
• Specialized for medical and life science domains

Applications

• Life Science Research: Generate accurate, referenced answers for biomedical and healthcare queries
• Pharmaceutical Industry: Support healthcare professionals with evidence-based responses
• Medical Education: Aid students and educators with scientifically-supported content from various academic sources

System Requirements

GPU Requirements

• Minimum VRAM: 16-18 GB for inference in BF16 (BFloat16) precision
• Recommended GPUs:
  • NVIDIA A100 (20GB) - Ideal for BF16 precision
  • Any GPU with 16+ GB VRAM
  • Performance may vary based on available memory

Software Prerequisites

• Python 3.x
• PyTorch
• Transformers library

Basic Implementation

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Set parameters
model_dir = "rohitanurag/ClinicalGPT-Pubmed-Instruct-V1.0"
max_new_tokens = 1500
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir).to(device)

# Define your question
question = "What is the role of the tumor microenvironment in cancer progression?"
prompt = f"""Please provide the answer to the question asked.
### Question: {question}
### Answer: """

# Tokenize input
inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True).to(device)

# Generate output
output_ids = model.generate(
    inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1000,
    repetition_penalty=1.2,
    pad_token_id=tokenizer.eos_token_id,
)

# Decode and print
generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(f"Generated Answer:\n{generated_text}")

Sample Output

### Question: What is the role of the tumor microenvironment in cancer progression, and how does it influence the response to therapy?
### Answer:
The tumor microenvironment (TME) refers to the complex network of cells, extracellular matrix components, signaling molecules, and immune cells that surround a growing tumor. It plays an essential role in regulating various aspects of cancer development and progression...

### References:
1. Hanahan D, Weinberg RA. Hallmarks of Cancer: The Next Generation. Cell. 2011;144(5):646-74. doi:10.1016/j.cell.2011.03.019
2. Coussens LM, Pollard JW. Angiogenesis and Metastasis. Nature Reviews Cancer. 2006;6(1):57-68. doi:10.1038/nrc2210
3. Mantovani A, et al. Cancer's Educated Environment: How the Tumour Microenvironment Promotes Progression. Cell. 2017;168(6):988-1001.e15. doi:10.1016/j.cell.2017.02.011
4. Cheng YH, et al. Targeting the Tumor Microenvironment for Improved Therapy Response. Journal of Clinical Oncology. 2018;34(18_suppl):LBA10001. doi:10.1200/JCO.2018.34.18_suppl.LBA10001
5. Kang YS, et al. Role of the Tumor Microenvironment in Cancer Immunotherapy. Current Opinion in Pharmacology. 2018;30:101-108. doi:10.1016/j.ycoop.20

Model Details

• Base Model: Mistral-7B-Instruct-v0.2
• Primary Training Data: 10 million PubMed abstracts and titles
• Specialization: Medical question-answering with scientific citations
• Output: Generates detailed answers with relevant academic references

Future Development

ClinicalGPT-Pubmed-Instruct-V2.0 is under development, featuring:

• Training on new 20 million pubmed articles
• Inclusion of full-text articles from various academic sources
• Enhanced performance for life science tasks
• Expanded citation capabilities across multiple scientific databases

Contributors

• Rohit Anurag – Principal Data Scientist
• Aneesh Paul – Data Scientist

License

Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0