Sunday, June 19, 2011

Creation of Scoring Rubrics

The following are typical characteristics of good scoring rubrics:
  1. Rubrics may consist of a checklist of items, each with an even number of points. Two-point rubrics would indicate that the student either did or did not perform the specified task. Four or more points in a rubric are common and indicate the degree to which a student performed a given task.
  2. Rubric-based scoring is criterion based. That is, the rubric contains criteria for acceptable performance that are meaningful, clear, concise, unambiguous, and credible -- thus ensuring inter-rater reliability.
  3. A rubric is used to assess only those behaviors that are directly observable -- what a student is doing rather than what he or she is thinking, for instance.
  4. Holistic rubrics require raters to give a single score based on the overall quality of the presentation. This may or may not be used in conjunction with other items.
  5. A multi-dimensional rubric can provide a better assessment and understanding of expected or actual performance.
  6. Create scoring guides that contain the following elements:
  • fully explicit scoring criteria
  • examples or models illustrating each score point
  • an abbreviated one-page version of the criteria for reference during actual rating
  • a form for recording scores
When using scoring rubrics it is important that the following considerations be given:
  1. Students might help create the scoring rubric.
  2. Students should be fully aware of the content and meaning of the scoring rubric.
  3. Rubrics should contain an even number of performance levels.
  4. Each level gives a descriptive meaning of the required performance.
Grading Rubric for Teacher Board Skills -- A Poor Example
Please use the following definitions:
4 = Excellent: 3 = Good: 2 = Fair: 1 = Poor: 0 = Unacceptable
 Dimension
 Descriptor
 3
 2
 1
0
 visibility:
does not hide work with body any more  than necessary    
 writing:
writing is large and clearly written; level    
 speech:
is audible and enunciated clearly; easy to  follow    
 attention to students:
talks to students, not board; frequently  turns to face students    
 interaction:
keeps students actively involved in process  by asking questions    
 modeling:
portrays logical thinking by using a  efficient "think aloud" protocol    
student needs:
responds to student needs to know; asks if  there are any follow-up questions    
Critique: It is unclear what "excellent, good, fair, poor, and unacceptable" behavior actually means. That's up to the discretion of the assessor. This will result in an unfair assessment; there is no inter-rater reliability. More importantly, the one assessed will have no clear idea of how performance can be improved. With five categories and a poor definition of expected performance, raters will go more for the middle assessment if uncertain. With a 4-point scoring system assessors will at least have to come down on one side or the other of just a good old middle-of-the-road "fair."

Grading Rubric for Teacher Board Skills -- A Better, but Still Flawed, Example
Please use the following definitions:
3 = Regularly; 2 = Some Times; 1 = Rarely; 0 = Never

Dimension
Descriptor
3
2
1
0
 text placement:
board work shows logical and sequential  placement    
 drawings:
uses large, well-done drawings and/or  diagrams as needed and as appropriate    
 units:
uses units on all quantities, carries them  through calculations, and cancels as needed    
 shows equations:
writes out equations before attempting to  solve    
 solutions:
solves equation for desired variable before  inserting quantities    
 checks:
refers to units as a test on accuracy of  calculation; makes estimate of reliability  and/or meaningfulness of answer    
 defines variables:
writes down what is known identifying  with variables, and states what is desired    
significant figures:
appropriately and accurately makes use of  significant figures during solution    

Critique: At least the teacher has an idea of how to improve his or her performance. If not done frequently enough, then he or she must do more of it. Unfortunately, quality teaching rarely depends on frequency of performance; rather, it most frequently depends on the quality of the performance. A rubric that actually describes the performance is that for which we looking.


Grading Rubric for Teacher Board Skills -- A Good Example


Dimension
Criteria
Score
 metacognition:
3 pts. Regularly talks about the problem-solving process  just modeled or to be modeled in detail; notes that problem solving is more than just looking for the correct equation; clearly identifies principle used to solve problem.2 pts. Sometimes talks about the problem-solving process  just modeled or to be modeled in detail; clearly identifies principle used to solve problem; does not note that problem solving is more than just looking for the correct equation.1 pt. Rarely talks about the problem-solving process  just modeled or to be modeled; does not identify principle used or that problem solving is more than just looking for equations.0 pts: Does not address problem-solving process from a metacognitive perspective at all.  
 accuracy:
3 pts. Is completely accurate from physical and mathematical  viewpoints; makes no errors in statements or calculations.2 pts. Makes one or two minor errors from mathematical and/or physical viewpoint(s). 1 pt. Makes an egregious error in one problems.0 pt. Makes an egregious error in two or more problems. 
 vectors/scalars:
3 pts. Distinguishes vector from scalar quantities  both physically and numerically2 pts. Sometimes fails to distinguish scalar from vector quantities by not using vector notation and/or appropriate language.1 pt. Regularly fails to distinguish scalar from vector quantities in writing and/or speaking.0 pts. Regularly fails to distinguish scalar from vector quantities in writing and/or speaking. 
 problem statement:
3 pts. Reads the problem clearly, completely, and  unambiguously or has student do the same.2 pts. Describes the problem clearly, completely, and unambiguously.1 pt. Describes the problem but leaves out pertinent details.0 pts. Merely makes reference to written problem; does not read it or have it read aloud. 
Composite Score:
 

Critique: This appears to be a good rubric, at least from a structural perspective. Now, the only real concern is if the criteria actually describe the sort of practice legitimately expected. Creating really good rubrics requires both practice and experience with the rubric in question. Sometimes it becomes quickly clear that critical dimensions are mission. It's nice to have a column for scores, and even a box for the composite score.
Reference this information carefully as you work on you Scoring Rubric Project.

No comments:

Post a Comment