Activities

Materials

Core Learning Goals: Item writers must understand the content that is being assessed and the assessment limits of that content.

Initially, before we had really developed the assessment limits, we had the item writers work in pairs to discuss their understanding of specific indicators and ask questions about the limits of what can be assessed. This helped us clarify the indicators in light of what is appropriate for the level of Government being assessed.

At this stage, we had writers look at Prototype items, Public Release items, and rejected items and match them to the CLG and assessment limits as well as identify them by item type. It is important that they see items that meet the assessment limits and items that do not meet the assessment limits. They should be able to recognize why an item does or does not meet the assessment limits.

Core Learning Goals (CLG) with assessment limits, description of item types, the evolution of an item, Prototype items, Public Release items, and rejected items and rubric.

Qualities of a good Selected Response (SR)

Item writers revise items from activity listed above so that the items do fit the assessment limits and basic format.

Basic Format Sheet, Checklist for Selected Response Items

Maryland Social Studies HSA Rubrics:

Item writers must understand the rubrics that are used to score the BCR and ECR items.

Writers examine the constructed response Public Release items and sample student papers representing each score point.

Public Release items, Social Studies rubric

Qualities of good Constructed Response items (BCR/ECR)

After reviewing the format for Constructed Response Items, writers revise items from activity listed above so that the items do fit the assessment limits, content, and basic format.

Basic Format Sheet, Checklist for Constructed Response Items

Activities

Materials

Writing assignments

Based on the needs of the group, each writer is given an assignment that includes the number of items to be written, the item types and indicators to be assessed.

Assignment sheets, Item templates, Appropriate software and access to resources, approved stimulus material

Editing by content specialists

Item content reviewed and revised by writers.

Revisions and writing

Writers go over the comments from the content editor, discuss them with the content editor, and make any necessary revisions to the items. Writers continue writing.

Peer editing

Writers work in pairs or small groups to review each other's items and give their peers feedback, which the writer then uses to revise the items.

Art/ Graphics

Items must be processed with appropriate software so that graphics are presented correctly.

Activities

Materials

Stimulus review

Teachers, Supervisors and MSDE Content Specialists review the stimuli and suggest edits, revisions and rejection.

Stimuli (headlines, cartoons, newspaper article excerpts, graphs, charts, maps)

Vendor review

Content specialists and Senior content editors review items and suggest edits.

Items, CLG, item review checklists

Content edits

Content specialists at MSDE review the comments from vendor review and make edits to the item.

Items, CLG, item review checklists

Formal content review

Teachers, not involved in the item writing, review the items, give feedback and propose edits.

Items, CLG, item review checklists

Bias and sensitivity review

Teams of people representing various minority groups review the items for bias, sensitivity, and linguistics.

Items from content review

Final edits (side by sides)

Content specialists review feedback from content review and bias and sensitivity.

Items and comments from reviews

Item type

Data collected

SR

Number and percent of students selecting each answer choice (p-value) and discrimination (point bi-serial).

ECR/BCR

Sample student responses that indicates miscuing. After scoring, the number and percent of students receiving each score point.

Range finding is done for each constructed response item. Anchor papers and training sets are developed for each item. Scorers are trained so that scoring will be consistent among all items.

Item type

How data is used

SR

When the point bi-serial number for the correct response is less than 0.15 or if the point bi-serial number for any distractor is positive, the item should be examined. Things to consider: is the content being taught, is one or more of the distractors partially correct, is the item misleading so that the distractors are attractive to the higher-achieving students, does the ‘attractive' distractor(s) represent a common error that students make.

ECR/BCR

Look at the responses students gave and the distribution of students across the score levels. Things to consider: is the content being taught, is the item misleading so that the students are misinterpreting the question, is the item rich enough to get a spread across all score levels. Check to see that the percent of students at the intermediate score levels is appropriate.

All Items

Based on the analysis of the data, either keep the item as is, revise and field test revised item or reject the item.

Items are also identified for bias (DIFF) and flagged for review and revision.