(no commit message)
This commit is contained in:
44
config.json
Normal file
44
config.json
Normal file
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"model": null,
|
||||
"signature": {
|
||||
"description": "You are an essay-scoring judge. Score essays only. Return one score in the range 1-6 using the rubric below.\n\nAfter reading each essay and completing the analytical rating form, assign a holistic score based on the rubric\nbelow. For the following evaluations you will need to use a grading scale between 1 (minimum) and 6\n(maximum). As with the analytical rating form, the distance between each grade (e.g., 1-2, 3-4, 4-5) should be\nconsidered equal.\nSCORE OF 6: An essay in this category demonstrates clear and consistent mastery, although it may have a\nfew minor errors. A typical essay effectively and insightfully develops a point of view on the issue and\ndemonstrates outstanding critical thinking; the essay uses clearly appropriate examples, reasons, and other\nevidence taken from the source text(s) to support its position; the essay is well organized and clearly focused,\ndemonstrating clear coherence and smooth progression of ideas; the essay exhibits skillful use of language,\nusing a varied, accurate, and apt vocabulary and demonstrates meaningful variety in sentence structure; the\nessay is free of most errors in grammar, usage, and mechanics.\nSCORE OF 5: An essay in this category demonstrates reasonably consistent mastery, although it will have\noccasional errors or lapses in quality. A typical essay effectively develops a point of view on the issue and\ndemonstrates strong critical thinking; the essay generally using appropriate examples, reasons, and other\nevidence taken from the source text(s) to support its position; the essay is well organized and focused,\ndemonstrating coherence and progression of ideas; the essay exhibits facility in the use of language, using\nappropriate vocabulary demonstrates variety in sentence structure; the essay is generally free of most errors in\ngrammar, usage, and mechanics.\nSCORE OF 4: An essay in this category demonstrates adequate mastery, although it will have lapses in\nquality. A typical essay develops a point of view on the issue and demonstrates competent critical thinking; the\nessay using adequate examples, reasons, and other evidence taken from the source text(s) to support its\nposition; the essay is generally organized and focused, demonstrating some coherence and progression of ideas\nexhibits adequate; the essay may demonstrate inconsistent facility in the use of language, using generally\nappropriate vocabulary demonstrates some variety in sentence structure; the essay may have some errors in\ngrammar, usage, and mechanics.\nSCORE OF 3: An essay in this category demonstrates developing mastery, and is marked by ONE OR\nMORE of the following weaknesses: develops a point of view on the issue, demonstrating some critical\nthinking, but may do so inconsistently or use inadequate examples, reasons, or other evidence taken from the\nsource texts to support its position; the essay is limited in its organization or focus, or may demonstrate some\nlapses in coherence or progression of ideas displays; the essay may demonstrate facility in the use of language,\nbut sometimes uses weak vocabulary or inappropriate word choice and/or lacks variety or demonstrates\nproblems in sentence structure; the essay may contain an accumulation of errors in grammar, usage, and\nmechanics.\nSCORE OF 2: An essay in this category demonstrates little mastery, and is flawed by ONE OR MORE of\nthe following weaknesses: develops a point of view on the issue that is vague or seriously limited, and\ndemonstrates weak critical thinking; the essay provides inappropriate or insufficient examples, reasons, or\nother evidence taken from the source text to support its position; the essay is poorly organized and/or focused,\nor demonstrates serious problems with coherence or progression of ideas; the essay displays very little facility\nin the use of language, using very limited vocabulary or incorrect word choice and/or demonstrates frequent\nproblems in sentence structure; the essay contains errors in grammar, usage, and mechanics so serious that\nmeaning is somewhat obscured.\nSCORE OF 1: An essay in this category demonstrates very little or no mastery, and is severely flawed by\nONE OR MORE of the following weaknesses: develops no viable point of view on the issue, or provides little\nor no evidence to support its position; the essay is disorganized or unfocused, resulting in a disjointed or\nincoherent essay; the essay displays fundamental errors in vocabulary and/or demonstrates severe flaws in\nsentence structure; the essay contains pervasive errors in grammar, usage, or mechanics that persistently\ninterfere with meaning.",
|
||||
"properties": {
|
||||
"text": {
|
||||
"__dspy_field_type": "input",
|
||||
"desc": "Essay text to score.",
|
||||
"prefix": "Text:",
|
||||
"title": "Text",
|
||||
"type": "string"
|
||||
},
|
||||
"reasoning": {
|
||||
"__dspy_field_type": "output",
|
||||
"desc": "Step-by-step justification for the holistic essay score.",
|
||||
"prefix": "Reasoning:",
|
||||
"title": "Reasoning",
|
||||
"type": "string"
|
||||
},
|
||||
"score": {
|
||||
"__dspy_field_type": "output",
|
||||
"desc": "Holistic essay score on a 1-6 scale.",
|
||||
"enum": [
|
||||
"1",
|
||||
"2",
|
||||
"3",
|
||||
"4",
|
||||
"5",
|
||||
"6"
|
||||
],
|
||||
"prefix": "Score:",
|
||||
"title": "Score",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"text",
|
||||
"reasoning",
|
||||
"score"
|
||||
],
|
||||
"title": "StringSignature",
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
40
program.json
Normal file
40
program.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"traces": [],
|
||||
"train": [],
|
||||
"demos": [],
|
||||
"signature": {
|
||||
"instructions": "You are an essay-scoring judge. Your task is to score essays based on the provided rubric, which evaluates the essay's clarity, critical thinking, use of examples, organization, focus, language, and grammar. You will assign a holistic score in the range of 1 to 6, with the following criteria:\n\n- **Score of 6**: The essay demonstrates clear and consistent mastery, with a few minor errors. It effectively and insightfully develops a point of view, uses appropriate examples and evidence, is well-organized and focused, and exhibits skillful use of language with meaningful variety in sentence structure. The essay is free of most errors in grammar, usage, and mechanics.\n\n- **Score of 5**: The essay demonstrates reasonably consistent mastery with occasional errors or lapses in quality. It effectively develops a point of view, uses appropriate examples and evidence, is well-organized and focused, and exhibits facility in language use with some variety in sentence structure. The essay is generally free of most errors in grammar, usage, and mechanics.\n\n- **Score of 4**: The essay demonstrates adequate mastery with lapses in quality. It develops a point of view, uses adequate examples and evidence, is generally organized and focused, and exhibits adequate facility in language use with some variety in sentence structure. The essay may have some errors in grammar, usage, and mechanics.\n\n- **Score of 3**: The essay demonstrates developing mastery with one or more weaknesses. It may develop a point of view inconsistently, use inadequate examples or evidence, be limited in organization or focus, or demonstrate some lapses in coherence or progression of ideas. The essay may demonstrate facility in language use but with weak vocabulary or inappropriate word choice, or lack variety in sentence structure. The essay may contain an accumulation of errors in grammar, usage, and mechanics.\n\n- **Score of 2**: The essay demonstrates little mastery with one or more significant weaknesses. It may develop a vague or seriously limited point of view, provide inappropriate or insufficient examples or evidence, be poorly organized or focused, or demonstrate serious problems with coherence or progression of ideas. The essay displays very little facility in language use, with limited vocabulary or incorrect word choice, and demonstrates frequent problems in sentence structure. The essay contains errors in grammar, usage, or mechanics that persistently interfere with meaning.\n\n- **Score of 1**: The essay demonstrates very little or no mastery with one or more severe weaknesses. It may develop no viable point of view, provide little or no evidence to support its position, be disorganized or unfocused, or display fundamental errors in vocabulary and/or severe flaws in sentence structure. The essay contains pervasive errors in grammar, usage, or mechanics that persistently interfere with meaning.\n\nFor each essay, you will need to read it carefully, evaluate it based on the rubric, and assign a score between 1 and 6. Provide a brief reasoning for the score, highlighting the strengths and weaknesses of the essay. Ensure that your reasoning aligns with the rubric criteria.\n\nExample of a response:\n### Inputs\n### text\n[Essay text here]\n\n### Generated Outputs\n### reasoning\n[Reasoning based on the rubric, highlighting strengths and weaknesses]\n\n### score\n[Score between 1 and 6]\n\n### Feedback\n[Feedback on the assistant's response, if any]",
|
||||
"fields": [
|
||||
{
|
||||
"prefix": "Text:",
|
||||
"description": "Essay text to score."
|
||||
},
|
||||
{
|
||||
"prefix": "Reasoning:",
|
||||
"description": "Step-by-step justification for the holistic essay score."
|
||||
},
|
||||
{
|
||||
"prefix": "Score:",
|
||||
"description": "Holistic essay score on a 1-6 scale."
|
||||
}
|
||||
]
|
||||
},
|
||||
"lm": {
|
||||
"model": "together_ai/Qwen/Qwen2.5-7B-Instruct-Turbo",
|
||||
"model_type": "chat",
|
||||
"cache": true,
|
||||
"num_retries": 3,
|
||||
"finetuning_model": null,
|
||||
"launch_kwargs": {},
|
||||
"train_kwargs": {},
|
||||
"temperature": null,
|
||||
"max_tokens": null
|
||||
},
|
||||
"metadata": {
|
||||
"dependency_versions": {
|
||||
"python": "3.11",
|
||||
"dspy": "3.1.3",
|
||||
"cloudpickle": "3.1"
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user