67 lines
6.3 KiB
JSON
67 lines
6.3 KiB
JSON
{
|
|
"assess.predict": {
|
|
"traces": [],
|
|
"train": [],
|
|
"demos": [
|
|
{
|
|
"ground_truth_conversation": "(42) Doctor: I really hope you get better soon.\n(42) Patient: Thanks a lot. Thanks so much, for your time.\n(43) Doctor: OK. Bye bye.\n(43) Patient: OK. Bye.",
|
|
"transcription_conversation": "(42) Doctor: I really hope you get better soon.\n(42) Patient: Thanks a lot. Thanks so much, for your time.\n(43) Doctor: OK. Bye bye.\n(43) Patient: ",
|
|
"clinical_impact": "0"
|
|
},
|
|
{
|
|
"ground_truth_conversation": "(0) Doctor: Hello? Hello. Um, before I go any further, can I confirm your name and your date of birth?\n(0) Patient: Uh, yeah. Uh my name is April, and I'm fifty.\n(1) Doctor: You're fifty, OK. April, how can I help you this afternoon?\n(1) Patient: Well I've just been having this like, cough, for quite a few days. And my nose is running. Um and it's just been super annoying, and it's not going away.",
|
|
"transcription_conversation": "(0) Doctor: Hello? Hello. Um, before I go any further, can I confirm your name and your date of birth?\n(0) Patient: Uh, yeah. Uh my name is April, and I'm fifty.\n(1) Doctor: You're fifty, OK. April, how can I help you this afternoon?\n(1) Patient: well i've just been having this like cough for quite a few days and my nose is running and it's just been super annoying and it's not going away",
|
|
"clinical_impact": "0"
|
|
},
|
|
{
|
|
"ground_truth_conversation": "(44) Doctor: Any pain, or strong smell, or having to go more often than normal?\n(44) Patient: So, I, I, I don't really, I have noticed that I don't really go up that high before, so, yeah, I do drink a lot though.\n(45) Doctor: Right, OK. And any , any weight loss or blood in the stool, or urine?\n(45) Patient: Yes. I have . As I said, I have some, I guess, discharge and pain. But, then I there's isn't a lot of discharge.",
|
|
"transcription_conversation": "(44) Doctor: Any pain, or strong smell, or having to go more often than normal?\n(44) Patient: So, I, I, I don't really, I have noticed that I don't really go up that high before, so, yeah, I do drink a lot though.\n(45) Doctor: Right, OK. And any , any weight loss or blood in the stool, or urine?\n(45) Patient: yes i have i was said i have some i can get some discharge and bleeding but then i do that there is no discharge",
|
|
"clinical_impact": "2"
|
|
},
|
|
{
|
|
"ground_truth_conversation": "(12) Doctor: Bit of tightness, OK, OK. So, I, unfortunately, I don't have much of your past medical history. Do you have any significant past medical or surgical history you'd like to share?\n(12) Patient: Uh, no, not really. Uh, a bit of eczema and I'm allergic to penicillin.\n(13) Doctor: OK. Are you on any regular medication at all? OK. Righty-ho. Are you having any rash on your body at the moment?\n(13) Patient: Yes there's a bit of blotching .",
|
|
"transcription_conversation": "(12) Doctor: Bit of tightness, OK, OK. So, I, unfortunately, I don't have much of your past medical history. Do you have any significant past medical or surgical history you'd like to share?\n(12) Patient: Uh, no, not really. Uh, a bit of eczema and I'm allergic to penicillin.\n(13) Doctor: OK. Are you on any regular medication at all? OK. Righty-ho. Are you having any rash on your body at the moment?\n(13) Patient: yes there's a bit of blocking down here",
|
|
"clinical_impact": "2"
|
|
}
|
|
],
|
|
"signature": {
|
|
"instructions": "As a medical expert, your task is to evaluate the clinical impact of errors in a transcribed medical conversation. You will compare a \"Ground Truth Conversation\" with a \"Transcription Conversation\" to identify and assess any discrepancies.\n\nYour assessment must follow two steps:\n1. **Reasoning:** Provide a brief clinical justification for your assessment. Identify the specific error(s) (e.g., word substitution, omission) and explain why they do or do not affect the medical meaning or potential patient care.\n2. **Clinical Impact:** After your reasoning, assign a single numerical score based on the severity of the error(s).\n\nUse the following strict classification guide for the 'Clinical Impact' score:\n\n* **0 = No impact:** Errors are purely cosmetic and do not alter medical meaning. This includes differences in punctuation, capitalization, or filler words (e.g., 'um', 'uh').\n* **1 = Minimal impact:** Some information is missing or changed, but it is NOT critical to immediate diagnosis or treatment decisions. For example, obscuring details about a resolved past issue or losing minor, non-essential context.\n* **2 = Significant impact:** Critical information is missing or incorrect in a way that COULD affect diagnosis, treatment decisions, or patient safety. Be especially vigilant for errors related to:\n * Medication names (e.g., \"Ibuprofen\" becomes \"upper brooklyn\")\n * Key symptoms (e.g., \"pain\" is changed to \"bleeding\")\n * Dosages, allergies, or critical patient history details.\n\nFirst, write your reasoning, then provide the final number for the clinical impact.",
|
|
"fields": [
|
|
{
|
|
"prefix": "Ground Truth Conversation:",
|
|
"description": "${ground_truth_conversation}"
|
|
},
|
|
{
|
|
"prefix": "Transcription Conversation:",
|
|
"description": "${transcription_conversation}"
|
|
},
|
|
{
|
|
"prefix": "Reasoning:",
|
|
"description": "Brief clinical justification for the assessment."
|
|
},
|
|
{
|
|
"prefix": "Clinical Impact:",
|
|
"description": "Clinical impact class (return ONLY the number):\n 0 = No impact: cosmetic differences only (punctuation, capitalization, filler words)\n 1 = Minimal impact: some information missing/changed but NOT critical to diagnosis or treatment decisions \n 2 = Significant impact: missing/incorrect information that COULD affect diagnosis, treatment, or patient safety\n Return ONLY: 0, 1, or 2"
|
|
}
|
|
]
|
|
},
|
|
"lm": {
|
|
"model": "openrouter/google/gemini-2.5-pro",
|
|
"model_type": "chat",
|
|
"cache": true,
|
|
"num_retries": 3,
|
|
"finetuning_model": null,
|
|
"launch_kwargs": {},
|
|
"train_kwargs": {},
|
|
"temperature": 0.1,
|
|
"max_tokens": 8000
|
|
}
|
|
},
|
|
"metadata": {
|
|
"dependency_versions": {
|
|
"python": "3.13",
|
|
"dspy": "3.0.4",
|
|
"cloudpickle": "3.1"
|
|
}
|
|
}
|
|
} |