(no commit message)
This commit is contained in:
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2025 Stanford MIMI Lab
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
138
README.md
138
README.md
@@ -1,2 +1,138 @@
|
|||||||
# medec-claude-3.7-sonnet-MIPROv2
|
# Structured Prompting Enables More Robust Evaluation of Language Models
|
||||||
|
|
||||||
|
[](https://arxiv.org/abs/2511.20836)
|
||||||
|
[](LICENSE)
|
||||||
|
|
||||||
|
<img src="assets/fig.png" alt="Overview" width="650">
|
||||||
|
|
||||||
|
**Figure 1** | **Pipeline overview**. (a) DSPy takes HELM's baseline prompt and produces structured prompt variants. (b) HELM evaluates models under each prompt variant. With structured prompting, we observe more robust performance evaluation.
|
||||||
|
|
||||||
|
## What is DSPy-HELM?
|
||||||
|
|
||||||
|
A comprehensive framework for optimizing language model performance on benchmarks using [DSPy](https://github.com/stanfordnlp/dspy), enabling more robust evaluation. This toolkit allows automated prompt optimization (APO) for benchmarks from the [HELM](https://github.com/stanford-crfm/helm) (Holistic Evaluation of Language Models) ecosystem.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone the repository
|
||||||
|
git clone https://github.com/StanfordMIMI/dspy-helm.git
|
||||||
|
cd dspy-helm
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## Basic Usage
|
||||||
|
|
||||||
|
**Configure settings** in `run.sh`:
|
||||||
|
```bash
|
||||||
|
scenarios=("mmlu_pro" "gpqa" "gsm8k" "medcalc_bench" "medec" "head_qa" "medbullets")
|
||||||
|
optimizers=("MIPROv2" "BootstrapFewShotWithRandomSearch")
|
||||||
|
|
||||||
|
# Language model to be optimized
|
||||||
|
model=openai/gpt-4o
|
||||||
|
api_base="your_api_base_here"
|
||||||
|
api_key="your_api_key_here"
|
||||||
|
|
||||||
|
# Teacher language model for proposing instruction candidates (MIPROv2)
|
||||||
|
prompt_model=openai/gpt-4o
|
||||||
|
prompt_api_base="your_api_base_here"
|
||||||
|
prompt_api_key="your_api_key_here"
|
||||||
|
|
||||||
|
max_bootstrapped_demos=3
|
||||||
|
max_labeled_demos=3
|
||||||
|
num_threads=1
|
||||||
|
```
|
||||||
|
|
||||||
|
**Run optimization**:
|
||||||
|
```bash
|
||||||
|
./run.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Supported Benchmarks
|
||||||
|
|
||||||
|
| Benchmark | Input → Output | Task |
|
||||||
|
|-------------------|--------------------------------------|------------------------|
|
||||||
|
| MMLU-Pro | Reasoning Question → Answer | Multi-Task Reasoning |
|
||||||
|
| GPQA | Graduate Question → Answer | Graduate-Level QA |
|
||||||
|
| GSM8K | Math Problem → Solution | Numeric Problem-Solving|
|
||||||
|
| MedCalc-Bench | Patient Note → Computed Value | Computational Reasoning|
|
||||||
|
| Medec | Medical Narrative → Errors | Error Classification |
|
||||||
|
| HeadQA | Medical Question → Answer | USMLE-Style QA |
|
||||||
|
| MedBullets | Medical Question → Answer | USMLE-Style QA |
|
||||||
|
|
||||||
|
## Optimization Parameters
|
||||||
|
|
||||||
|
- `--max_bootstrapped_demos`: Number of bootstrapped demonstrations
|
||||||
|
- `--max_labeled_demos`: Number of labeled demonstrations
|
||||||
|
- `--num_threads`: Parallel processing threads (default: 16)
|
||||||
|
|
||||||
|
## 📁 Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
dspy-helm/
|
||||||
|
├── main.py # Main optimization script
|
||||||
|
├── scenarios.py # Benchmark implementations
|
||||||
|
├── run.sh # Batch optimization runner
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── agents/ # Optimized DSPy agents
|
||||||
|
│ ├── medcalc_bench/
|
||||||
|
│ ├── head_qa/
|
||||||
|
│ ├── medbullets/
|
||||||
|
│ └── ...
|
||||||
|
└── README.md # This file
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📈 Results and Evaluation
|
||||||
|
|
||||||
|
Optimized agents are automatically saved to the `agents/` directory:
|
||||||
|
|
||||||
|
```
|
||||||
|
agents/
|
||||||
|
└── {scenario}/
|
||||||
|
└── {model_name}/
|
||||||
|
├── MIPROv2.json
|
||||||
|
└── BootstrapFewShotWithRandomSearch.json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Creating Custom Scenarios
|
||||||
|
|
||||||
|
To add a new HELM benchmark:
|
||||||
|
|
||||||
|
**Implement the scenario class** in `scenarios.py`:
|
||||||
|
```python
|
||||||
|
class my_benchmark:
|
||||||
|
def __init__(self, test_size=0.1, seed=42):
|
||||||
|
self.test_size = test_size
|
||||||
|
self.seed = seed
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def make_prompt(row):
|
||||||
|
return f"Question: {row['question']}\nAnswer:"
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def metric(example, pred, trace=None):
|
||||||
|
# Your evaluation metric
|
||||||
|
return dspy.evaluate.metrics.answer_exact_match(example, pred, trace)
|
||||||
|
|
||||||
|
def load_data(self):
|
||||||
|
# Load and return trainset, valset
|
||||||
|
pass
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🙏 Acknowledgments
|
||||||
|
|
||||||
|
This repository is built using [DSPy](https://github.com/stanfordnlp/dspy) for prompt optimization.
|
||||||
|
|
||||||
|
## 📎 Citation
|
||||||
|
|
||||||
|
If you find this repository useful for your work, please cite the following paper:
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@article{aali2025structured,
|
||||||
|
title={Structured Prompting Enables More Robust Evaluation of Language Models},
|
||||||
|
author={Aali, Asad and Mohsin, Muhammad Ahmed and Bikia, Vasiliki and Singhvi, Arnav and Gaus, Richard and Bedi, Suhana and Cui, Hejie and Fuentes, Miguel and Unell, Alyssa and Mai, Yifan and others},
|
||||||
|
journal={arXiv preprint arXiv:2511.20836},
|
||||||
|
year={2025}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|||||||
14
config.json
Normal file
14
config.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"model": "openrouter/anthropic/claude-3.7-sonnet",
|
||||||
|
"scenario": "medec",
|
||||||
|
"optimizer": "MIPROv2",
|
||||||
|
"api_base": "https://openrouter.ai/api/v1",
|
||||||
|
"prompt_model": "gpt-4o",
|
||||||
|
"prompt_api_base": null,
|
||||||
|
"max_tokens": 64000,
|
||||||
|
"temperature": null,
|
||||||
|
"max_bootstrapped_demos": 3,
|
||||||
|
"max_labeled_demos": 3,
|
||||||
|
"num_threads": 16,
|
||||||
|
"val_size": null
|
||||||
|
}
|
||||||
62
program.json
Normal file
62
program.json
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
{
|
||||||
|
"agent.predict": {
|
||||||
|
"traces": [],
|
||||||
|
"train": [],
|
||||||
|
"demos": [
|
||||||
|
{
|
||||||
|
"augmented": true,
|
||||||
|
"inputs": "The following is a medical narrative about a patient. You are a skilled medical doctor reviewing the clinical text. The text is either correct or contains one error. The text has a sentence per line. Each line starts with the sentence ID, followed by a space character then the sentence to check. Check every sentence of the text. If the text is correct return the following output: CORRECT. If the text has a medical error, return the sentence ID of the sentence containing the error, followed by a space, and a corrected version of the sentence.\n\nClinical Note: 0 A 61-year-old man comes to the physician because of several episodes of dark urine over the past 2 weeks.\n1 He does not have dysuria or flank pain.\n2 He works in a factory that produces dyes.\n3 Since an accident at work 5 years ago, he has had moderate hearing loss bilaterally.\n4 He takes no medications.\n5 He has smoked a pack of cigarettes daily for 29 years and drinks one alcoholic beverage daily.\n6 Vital signs are within normal limits.\n7 Physical examination shows no abnormalities.\n8 His urine is pink; urinalysis shows 80 RBC/hpf but no WBCs.\n9 Cystoscopy shows a 3-cm mass in the bladder mucosa.\n10 The mass is resected.\n11 Pathologic examination shows an urothelial carcinoma with penetration into the muscular layer.\n12 An x-ray of the chest and a CT scan of the abdomen and pelvis with contrast show a normal upper urinary tract and normal lymph nodes.\n13 Patient was recommended a radical cystectomy.\n\nAnswer:",
|
||||||
|
"reasoning": "Let me analyze this clinical note line by line to identify any potential medical errors:\n\n0. A 61-year-old man comes to the physician because of several episodes of dark urine over the past 2 weeks. - This is consistent with hematuria, which is appropriate for the eventual diagnosis of bladder cancer.\n\n1. He does not have dysuria or flank pain. - This is plausible.\n\n2. He works in a factory that produces dyes. - This is important as exposure to certain industrial dyes is a risk factor for bladder cancer.\n\n3. Since an accident at work 5 years ago, he has had moderate hearing loss bilaterally. - This appears unrelated to the current complaint but is part of the patient's medical history.\n\n4. He takes no medications. - This is straightforward information.\n\n5. He has smoked a pack of cigarettes daily for 29 years and drinks one alcoholic beverage daily. - Smoking is a significant risk factor for bladder cancer, so this is relevant.\n\n6. Vital signs are within normal limits. - This is plausible.\n\n7. Physical examination shows no abnormalities. - This is plausible.\n\n8. His urine is pink; urinalysis shows 80 RBC/hpf but no WBCs. - This indicates hematuria without infection, which is consistent with bladder cancer.\n\n9. Cystoscopy shows a 3-cm mass in the bladder mucosa. - This is an appropriate finding.\n\n10. The mass is resected. - This is an appropriate initial management step.\n\n11. Pathologic examination shows an urothelial carcinoma with penetration into the muscular layer. - This indicates muscle-invasive bladder cancer (stage \u2265T2).\n\n12. An x-ray of the chest and a CT scan of the abdomen and pelvis with contrast show a normal upper urinary tract and normal lymph nodes. - This is appropriate staging workup.\n\n13. Patient was recommended a radical cystectomy. - This is the correct recommendation for muscle-invasive bladder cancer without evidence of metastasis.\n\nAfter reviewing each sentence, I don't see any medical errors in the clinical narrative. The diagnosis, workup, and management recommendations are all appropriate for a patient with muscle-invasive bladder cancer.",
|
||||||
|
"output": "CORRECT",
|
||||||
|
"answer": "CORRECT"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"augmented": true,
|
||||||
|
"inputs": "The following is a medical narrative about a patient. You are a skilled medical doctor reviewing the clinical text. The text is either correct or contains one error. The text has a sentence per line. Each line starts with the sentence ID, followed by a space character then the sentence to check. Check every sentence of the text. If the text is correct return the following output: CORRECT. If the text has a medical error, return the sentence ID of the sentence containing the error, followed by a space, and a corrected version of the sentence.\n\nClinical Note: 0 A 21-year-old Cambodian patient with a history of rheumatic heart disease presents to his primary care physician for a routine check-up.\n1 He reports being compliant with monthly penicillin G injections since being diagnosed with rheumatic fever at age 15.\n2 Patient will continue with intramuscular penicillin therapy as he denies any major side effects from the treatment, except for the inconvenience of organizing transportation to a physician's office every month.\n3 On exam, the patient is found to have a loud first heart sound and a mid-diastolic rumble that is best heard at the apex.\n\nAnswer:",
|
||||||
|
"reasoning": "Let me analyze this clinical note sentence by sentence:\n\n0. \"A 21-year-old Cambodian patient with a history of rheumatic heart disease presents to his primary care physician for a routine check-up.\" - This appears medically accurate.\n\n1. \"He reports being compliant with monthly penicillin G injections since being diagnosed with rheumatic fever at age 15.\" - This is appropriate as penicillin prophylaxis is standard for rheumatic heart disease.\n\n2. \"Patient will continue with intramuscular penicillin therapy as he denies any major side effects from the treatment, except for the inconvenience of organizing transportation to a physician's office every month.\" - This is medically accurate.\n\n3. \"On exam, the patient is found to have a loud first heart sound and a mid-diastolic rumble that is best heard at the apex.\" - This describes the classic findings of mitral stenosis, which is consistent with rheumatic heart disease. However, there's a slight error here. A mid-diastolic rumble at the apex is indeed consistent with mitral stenosis, but rheumatic heart disease typically presents with a loud FIRST heart sound (S1), not second heart sound. I should check if this is what the text actually says... The text does say \"loud first heart sound\" which is correct. On second review, I don't see an error in this sentence.\n\nAfter reviewing all sentences carefully, I don't see any medical errors. The presentation is consistent with rheumatic heart disease with mitral valve involvement (likely mitral stenosis based on the mid-diastolic rumble at the apex and loud S1).",
|
||||||
|
"output": "CORRECT",
|
||||||
|
"answer": "CORRECT"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"inputs": "The following is a medical narrative about a patient. You are a skilled medical doctor reviewing the clinical text. The text is either correct or contains one error. The text has a sentence per line. Each line starts with the sentence ID, followed by a space character then the sentence to check. Check every sentence of the text. If the text is correct return the following output: CORRECT. If the text has a medical error, return the sentence ID of the sentence containing the error, followed by a space, and a corrected version of the sentence.\n\nClinical Note: 0 A 47-year-old woman comes to the physician because of a 5-month history of insomnia.\n1 She frequently experiences leg discomfort when trying to fall asleep that is relieved temporarily by movement.\n2 Her husband tells her that she frequently flexes her ankles upward when she sleeps.\n3 She appears fatigued and anxious.\n4 Physical examination shows no abnormalities.\n5 Laboratory studies including a complete blood count and iron studies are within the reference range.\n6 Patient is prescribed ropinirole.\n\nAnswer:",
|
||||||
|
"output": "CORRECT"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"signature": {
|
||||||
|
"instructions": "Review the provided clinical narrative, which may either be correct or contain one medical error. Each line of the narrative begins with a sentence ID followed by the sentence. Examine every sentence to identify any potential medical error. If the narrative is correct, respond with \"CORRECT.\" If there is a medical error, return the sentence ID and provide a corrected version of the sentence, using logical reasoning pertinent to medical facts and principles to support your conclusion.",
|
||||||
|
"fields": [
|
||||||
|
{
|
||||||
|
"prefix": "Inputs:",
|
||||||
|
"description": "${inputs}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"prefix": "Reasoning: Let's think step by step in order to",
|
||||||
|
"description": "${reasoning}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"prefix": "Output:",
|
||||||
|
"description": "${output}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"lm": {
|
||||||
|
"model": "openrouter/anthropic/claude-3.7-sonnet",
|
||||||
|
"model_type": "chat",
|
||||||
|
"cache": true,
|
||||||
|
"num_retries": 3,
|
||||||
|
"finetuning_model": null,
|
||||||
|
"launch_kwargs": {},
|
||||||
|
"train_kwargs": {},
|
||||||
|
"temperature": null,
|
||||||
|
"max_tokens": 64000,
|
||||||
|
"api_base": "https://openrouter.ai/api/v1"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"dependency_versions": {
|
||||||
|
"python": "3.13",
|
||||||
|
"dspy": "3.0.4",
|
||||||
|
"cloudpickle": "3.1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user