Appropriate Instruments for Educational and Political Constituencies

Equating and scaling accuracy are especially important with additional uses of data that depend on sound vertical scaling and annual equating of forms to measure Annual Yearly Progress. It is especially important that equating and scaling studies be conducted on the appropriate populations to which the data will be generalized. In most cases, this means within-state samples for the equating and scaling studies. If states are using publishers' off-the-shelf standardized tests intact, it is appropriate to use the publisher's national equating and scaling groups. If the states are augmenting publisher's off-the-shelf tests with state-specific items, thus creating new scales that measure state standards more accurately, it is important that the samples for the equating and scaling studies be taken from inside the state, or from a source that can be proven sufficiently similar for statistical purposes.

If requested by states, vendors shall be prepared to explain why the test is an appropriate instrument for its intended uses to and political constituencies, the press or in court. In cases where such activities are extensive, provisions defining compensation should be included in the contract.

The state has primary responsibility for defending its programs and decisions but can be aided substantially by vendor expertise and knowledge. Vendors may be most knowledgeable about their products and may have consultants who can provide a national perspective on the issues of concern to various constituencies. The vendor may also have key information required for defending testing activities in a legal forum. All such involvement should occur only when specifically requested by authorized state staff. Costs for such vendor activities should be explicitly included in the RFP and contract or should be dealt with in a separate addendum as the need arises. Vendors should have adequate and appropriate staff to handle such activities.

The state retains primary responsibility for responding to questions regarding the testing system and instruments. The state must be prepared to answer such questions from its legislature, the public, and/or the media. To this end, states must employ some staff with substantive knowledge of psychometric issues and a familiarity with the particular state testing system.

States may request assistance from developers in explaining tests to important political and educational constituencies, such as state legislators, the press, education organizations, and parent groups. In cases of legal challenge, states may need assistance in court explaining how the test development process satisfied all necessary psychometric and legal standards. Contracts should indicate whether, or under which circumstances, such assistance would require additional compensation.

State agencies must also be proactive when it comes to communicating with the legislature. Longstanding relationships with legislators and legislative staffs will enable the agency to help shape policy that is sound and easily justifiable, rather than having to react to legislation that is thrust upon them.

The following documents shall be provided by vendors to states: a) an annual management plan, including a schedule for all tasks required to carry out the plan; b) test development and construction specifications; c) written status reports at regular, agreed-upon intervals as provided for in the contract. In between, updates should be provided as needed through a medium determined by the state (e.g., via phone or email). Depending on the complexity of the program and contract specifications, monthly or quarterly planning meetings with the vendor should be held to discuss current progress, upcoming tasks, problems, and mid-course corrections.

Regular communication between state and vendor staff is a key component of a successful testing program. Detailed documentation of program activities is also important for creating a historical record, informing policymakers, and providing supporting evidence in the event of a legal challenge. Communication and documentation should not be left to chance but rather should be part of the project plan from the outset.