Steps In Focusing Program Evaluation:

A Child Study Team Project Example

Step 1: Determining the Object of the Evaluation

A Child Study Team (CST) model is being developed in an urban school district in upstate New York. Important aspects of the model include:

• a problem-solving process with teachers and support staff for school-based student difficulties;
• prereferral interventions for at-risk students;
• ongoing, systematic assessment of outcomes related to the identified problems(s); and
• increased parent involvement.

Variations of the Child Study Teams already existed in the district; however, there was an increased interest in revising these multi-disciplinary teams in response to concerns expressed by the Board of Education about the high numbers of students in the district who were designated as having handicapping conditions and were thus assigned special-education services.

Goals of the program include:

decreasing the number of students in the district designated as having school-based disabilities (particularly students classified with as being Learning Disabled or Emotionally Disturbed). Since research had shown that referral for Committee on Special Education (CSE) was correlated highly with subsequent Special Education classification, it was decided that decreasing formal CSE referrals for evaluation could be expected to help to decrease the number of students ultimately designated as having disabilities.

establishing problem-solving teams in schools to assist teachers to more effectively help students with academic or behavioral difficulties. To accomplish this goal, personnel associated with the project would:

·         run staff inservices to increase CST members' consultation skills, ability to design effective prereferral interventions, and employment of assessment techniques best matched to track the student's progress during interventions;

·         elicit referrals to the CST from teachers in participating schools;

·         increase parent involvement in the problem solving process;

·         identify and mobilize resources in schools to support interventions for students at risk, as well as to assess intervention outcomes for these referred students.

increasing academic competence and/or school adjustment for students referred to the problem-solving team.

Step 2: Deciding on the Purpose of the Evaluation

The program evaluators talked with developers and trainers of the Child Study Team project and identified the following series of purposes that would inform the program evaluation:

As examples of formative, or ongoing, assessment purposes, the evaluation would:

• identify staff development needs on an ongoing basis;
• investigate those aspects of the program that were successful (and worth keeping) and those found to be less successful and thus in need of strengthening or revision.
• provide guidance to project planners in allocating resources to participating buildings according to identified need;
• provide insights to project planners about implementing the CST project in other schools on a larger scale in a later phase of the project.

Among summative, or checkpoint, purposes, the evaluation would track whether the Child Study Team model:

• in fact resulted in a reduction in Special Education student referrals;
• produced improved student performance in schools that would justify the resources devoted to the program;
• was ready to be implemented in additional schools;
• conformed with legal standards for providing a free and appropriate public education to all students.

Step 3: Analyzing the Audiences and Stakeholders for the Evaluation

A prioritized list was prepared of potential audiences and stakeholders for the Child Study Team project. While all groups appearing on the list could conceivably have been identified as audiences, the project evaluators believed that certain groups would be most likely to review and react to some version of the program evaluation information (audiences) while other groups were less likely to review the evaluation but still had a stake in the process that should be recognized (stakeholders):

Audiences & Stakeholders for the CST Evaluation Project:

• Parents of students referred to the Child Study Team (audience & stakeholders)
• Teachers referring students to the CST (audience & stakeholders)
• Teachers and school support personnel serving on the CST (audience & stakeholders)
• School administrators (audience & stakeholders)
• Board of Education (audience & stakeholders)
• Other schools in the district targeted to implement the CST project in their schools in the future (stakeholders)
• Students being referred to the Child Study Team (stakeholders)

Step 4: Understanding Contextual Factors That Can Positively or Negatively Impact the Evaluation

When the program evaluators thought through possible contextual factors that might impact the Child Study Tam program evaluation, they generated a substantial list:

The district Board of Education and (consequently) the Special Education Department wish to see a decrease in rates of students classified as having school-based disabilities across the district. The CST project is seen by some as a means of slowing down the referral-test-place process that results in students being designated as needing Special Education. Therefore, it is anticipated that the idea of having CSTs in buildings will be strongly supported by the Board of Education (positive impact)

The Board of Education has allocated no funding to the CST project or its evaluation. The few resources being extended to the project are being taken from existing staff and budgets, putting CST into competition with other existing programs (negative impact)

Administrators who oversee elementary education have not been directly involved in the development of the CST project, even though that project attempts to support regular-education teachers in elementary schools. (negative impact)

Some school psychologists in the district have voiced opposition to a process that moves diagnosis of learning disabilities further from a medical model process. (negative impact)

Teachers based at CST schools will be required to put a prereferral intervention into place before they can seek to have a child referred to be evaluated by the Committee on Special Education. Teachers and parents have expressed a concern that this prereferral obligation will substantially slow down the process of designating a student with special needs. Some teachers also feel that they don't have the ability to put effective interventions into place for children with severe behavioral or academic difficulties (negative impact).

Schools select to pilot the first phase of the CST project had administrators who were very interested in seeing the project be successful in their schools. Thus, for pilot schools there is an initial commitment and motivation for the projects to be effective. This ready acceptance of the program may not hold when it is introduced to other schools in the future (positive in the initial stages, perhaps negative later on)

In at least two of the pilot schools implementing CST, pull-out reading remediation programs were recently cut to return reading teachers to the classroom and thus reduce overall class sizes. The positive effect of this move is that class sizes are smaller, while a negative effect is that reading teachers are no longer available as interventionists. (negative impact overall)

Initiatives upon which the CST is being modeled typically require 3 to 5 years to develop before they are considered fully effective. Certain audiences may have the expectation that the program will be successful in pilot phase within a single year. (negative)

Several initiatives have been developed in the district. In many instances, CST fits in with these other initiatives very well (e.g., site-based planning, etc.) (positive). On the other hand, staff may view CST as 'just another fad' (negative).

The CST program requires that teachers use problem-solving approaches, intervention strategies, and assessment techniques with which most staff are unfamiliar. A great deal of inservice will be required for CST to be successful (negative at first, positive as staff competencies increase).

Step 5: Decide on Key Questions to Be Answered in the Program Evaluation

Selecting Crucial Questions: Child Study Team Project Example

In selecting important evaluation questions, the program evaluators of the Child Study Team project considered the central audiences for the evaluation.

A question formulated for the Board of Education was:

• Has there been a change in special education designation rates over the course of four years?

Questions selected for parents included:

• Has there been a change in special education designation rates over the course of four years?
• Are CST procedures consistent with educational law/policy?
• Does CST help children having academic difficulties to learn better in school?
• Does CST help children having behavioral difficulties adapt better to school?
• Is the CST process addressing student difficulties in a timely fashion?

Questions selected for teachers included:

• Does CST help children having academic difficulties to learn better in school?
• Does CST help children having behavioral difficulties adapt better to school?
• Is the CST process addressing student difficulties in a timely fashion?
• Are interventions developed through CST feasible and effective?
• Are teachers supported in attempts to remediate student problems?