Appropriate Instruments for Educational and Political Constituencies

The standards and responsibilities in this section describe test development activities necessary to produce psychometrically and legally defensible high stakes tests efficiently and to a high standard of quality. These activities may be completed by either the state or the vendor. If responsibility for any portion of an activity is assigned to the vendor, the RFP (Request for Proposal) and resulting contract should so provide. If the state decides to retain responsibility for the activity, the state may seek advice from the vendor but should clearly indicate that expectation in the RFP and resulting contract.

For each activity or portion of an activity assigned to a vendor, the RFP and resulting contract should describe in detail what is expected of the vendor, any special conditions or limitations, and the compensation to be paid. If a state requests changes or delegates additional responsibilities to the vendor after the contract has been signed, the state will likely need to renegotiate the price.

A for quality assurance for all test instruments shall be developed and implemented. The policy should include specific activities, timelines, and locus of responsibility for evaluating the quality of each assessment instrument, including but not limited to, item quality, graphics quality, print quality, forms quality, equating and scaling accuracy, and quality of ancillary materials (e.g., measuring instruments, lab equipment). It shall be described in the proposal and incorporated into the contract.

In test development, quality assurance is a split responsibility between the state and the vendor. The vendor maintains primary responsibility for the quality of the work it produces, but the state ensures that the vendor's product matches the state's intent. The vendor must have detailed quality control procedures developed prior to contract initiation and be prepared to staff and implement them efficiently and effectively. The vendor should also clearly articulate the expectations for interaction with state staff during this process so the state can adequately schedule and staff required reviews and signoffs to avoid unnecessary delays. All deliverables should be thoroughly checked by knowledgeable staff before being sent in draft form to the state. Following established quality control procedures will decrease the likelihood of errors and increase the vendor's reputation for delivering a quality product. The vendor may consider seeking outside review and/or certification of its quality control procedures to enhance their usefulness and credibility. It is also advisable for the vendor to formally document all reviews and signoffs and to have multiple checks where feasible and appropriate.

The state shall develop and implement a quality assurance policy that serves to monitor the vendor. This policy may incorporate techniques such as signing off on all page proofs, checking final copies of test booklets before distribution to , and sampling score reports to ensure typographic quality.

It is important at all stages of development to check work that has been completed to ensure quality, accuracy, adherence to professional standards, and satisfaction of all program policies, administrative rules and legislative statutes. When tests will be used to make high-stakes decisions, multiple checks and rechecks are imperative to detect and correct any errors. While, vendors maintain primary responsibility for the quality of their work, vendor checks do not entirely replace the need for quality reviews by states, especially in high-stakes situations. Such quality reviews should be systematic, including proper training for staff and adequate enforcement mechanisms, and must be implemented in an effective and timely way.