Frequently Asked Questions
What
is program evaluation?
Program evaluation is a
process in which information is systematically collected about a program
or some aspect of a program in order to make important decisions.
The program may be defined by:
- What
goes into it (e.g., staff, resources, facility),
- The
process in which programs are delivered (e.g., nature and scope of
professional development activities),
- Product
(what was done and who participated), and
- Outcomes
(e.g., satisfaction, instructional changes, improved student
achievement).
Evaluation may be conducted for several
reasons including:
- Decide
what parts (e.g., which workshops/courses) should be kept and which
ones should be discontinued,
- Find
ways of improving various aspects of Center Programs (e.g., increase
implementation of professional development activities),
- Inform
funders of ways that Teacher Centers improve education.
The reason for collecting information
determines what aspects of the program are investigated and how
information is collected analyzed and communicated.
While the reasons, focus and methods may be approached
differently, program evaluation should be objective, data-based and
guided by standards.
return to
top
Why
should we evaluate Center programs?
There are many reasons
for collecting information about a program that shape the nature of
evaluation and decisions that can be made. Below are some different reasons why Teacher Centers might
evaluate their programs. Depending
on the reason, different aspects of the program become the focus and
different audiences are considered.
|
Reason
|
Object or
focus of the evaluation
|
Audience
|
Measures
|
|
Decide which
professional development activities to
continue and which ones to discontinue.
|
Ways that Center
activities address identified needs of target stakeholders.
The focus is on objectives met.
|
Internal.
Center Director, Policy Board and in some cases school
superintendent or Board of Education)
|
Needs should be
identified.
Assessments,
Surveys covering prioritized Center objectives show programs with
best results.
|
|
Determine ways
that different programs can be improved.
|
Look at the ways
that programs are advertised, delivered and used. The focus is
more on process. What
are obstacles to people using or implementing the program?
|
Internal.
Information is used by those managing and supporting Center
programs (e.g., Center Director, Policy Board, trainers and
possibly school administrators).
|
Participants
might be given an opportunity to describe impediments to accessing
or implementing the program.
|
|
Demonstrate to
supporters or funders that Centers provide a valuable service.
|
Center activities
and resulting outcomes are evaluated. Evaluation objectives need to be important to audiences and
results need to be defensible especially in the context of a
doubting audience (e.g., in the face of budget cuts).
|
External.
This type of evaluation may be used for public relations or
to save a Center from a funder’s chopping block.
|
High quality
measures (reliable, valid, credible) demonstrating desired
outcomes closely related to the program.
|
return
to top
What are some ways to measure the
impact of professional development on teacher practice in the classroom?
- Teachers can be requested to keep journals
during professional development courses, in which they record their
efforts to put their training into practice and reflect on its
impact on student learning or behavior.
- The course instructor may enlist course
participants/teachers to take part in ongoing dialog with the
instructor and each other via electronic mail, 'chat room' or other
remote means of communication. This solution can foster a sense of
community as well as providing documentation of changes in teacher
practice.
- A teacher-training course may be set up to
bring all participants back together several weeks after they have
begun a new program or practice in their classrooms. At the session,
participants can 'debrief' about their experiences and present their
own observations, student projects, etc., as demonstrations of the
effect of their new practices.
- An observer may drop in periodically at the
instructor's invitation to see the teacher use practices taught in
the professional development course and to note student performance.
return to
top
How can we reassure teachers that
evaluation of their teaching practices intended for program evaluation
will not be used to make judgments about their professional competence?
Teachers are often quite willing to document
their classroom practices for purposes of evaluation of professional
development (even to the point of inviting in outside observers).
Teachers deserve to know, however, that any information collected for
program evaluation will not be used to evaluate their abilities as
teachers. If a school administrator might have access to documentation
of teacher classroom practices, teachers must be informed of this
possibility in advance and should have the option of declining to
participate.
Furthermore, if a Center wishes to share program evaluation publicly,
the Center should either first remove any information that would
identify specific teachers or obtain advance permission from teachers
before identifying them and their classroom results.
return to top
When
should we consider measuring student classroom performance to document
the impact of a professional development initiative?
There are no
hard and fast guidelines for determining when a Center should invest the
time and resources to attempt to document the the effects of
professional development on student academic or behavioral performance.
Generally, however, efforts to evaluate professional development efforts
should be selected based upon the scope of these efforts. For example,
it would usually not be worth measuring the classroom impact of single
workshops or informational clinics, as it is unlikely that they would
bring about lasting changes in teacher practice or student learning.
However, a teacher inservice course with a practicum component,
discussion groups, and costly outside presenters may be an excellent
candidate for measuring student classroom performance, as the large
investment of teacher time and Center resources is intended to benefit
students.
return to
top
How
might one evaluate the academic performance of groups of students?
Measures of student
academic performance would be as varied as the academic goals that might
be presented in teachers' professional development courses. The majority
of measures used, though, would typically fall under one or more of the
following categories:
Archival information. There is often a surprising amount of archival
information that can document (directly or indirectly) improvements in
student learning. Here are just a few examples: student grades, teacher
records of completed classroom and homework assignments, rates of
student referral for Special Education services, student attendance,
records of books read by students or signed out from the library.
Group standardized tests. Student performance on group-administered
academic tests can sometimes be useful indicators of improvements in
school skills. (The evaluator should be careful, though, to first
determine that the professional development initiative to be evaluated
actually targets the same skills that group tests measure!)
Direct observation. Observers may visit classrooms at the instructor's
invitation to observe teachers using techniques presented in a course
and also to directly observe student mastery of appropriate information,
concepts, or skills.
Student work products. Students may create work products that
demonstrate their developing abilities. For example, a teacher may take
a course in the use of multi-media technologies as teaching tools.
Later, a group of that teacher's students might post as a work product a
website that incorporates digital-photo, animation, and sound files to
teach science concepts.
return
to top
What
are some ways to measure general classroom 'behavioral climate'?
As with academic assessment, there are
several approaches to assessment of classroom 'behavioral climate.' It
is a good idea to select several methods of assessment from the
categories below to increase the level of confidence that can be placed
in the results.
- Archival information.
Archival indicators of classroom behavioral climate can include
number of teacher disciplinary office referrals, number of student
hours (across the classroom, grade level, school, or district) spent
in in-school or out-of-school suspension), number of parent-school
conferences held because of student behavioral issues, etc.
- Teacher, parent, and student
questionnaires. Questionnaires
about school behavioral climate can be given to teachers, students,
and parents.
- Teacher may be
asked to rate the relative 'disruptiveness' of students in the
classroom, to estimate the number of disciplinary referrals that
they write per week, and other questions of concern to
instructors.
- Students might be
asked how safe they feel in the school environment and to rate
the frequency that they might encounter physical or verbal
harassment.
- Parents may be
asked to rate their level of concern with the overall school
behavioral climate and to estimate the frequency with which
their children may be victimized through physical or verbal
harassment.
- These
questionnaires may be administered prior to, during, and after a
teacher course on classroom management.
- Direct observation.
Observers may visit classrooms at the
instructor's invitation and tally rate of student call-outs,
off-task or out-of-seat behaviors, or other student behaviors that
are of concern. These observations may be made prior to, during, and
after a teacher course on classroom management.
return
to top
How
can we avoid common mistakes evaluators make when evaluating student
classroom performance?
Assessing the impact of professional development
activities on student performance can be a daunting task for any
evaluator. Those who manage successfully to measure student classroom
performance plan their evaluations carefully in advance. Planning
requires that:
- the professional
development activity warrants the additional time and effort of
assessing student academic performance,
- the evaluation
questions to be answered are clear, unambiguous, and realistic,
- a mix of classroom
measures are selected that are likely to provide information
relevant to the evaluation questions, and
- that there are
sufficient resources (including people!) available to collect,
analyze, and report the evaluation information.
return to
top
|
