1998-99
& 1999-00 VERBAL ASSESSMENT REPORT October
1, 2000 Prepared
by Denise
H. Solomon Abstract The
Verbal Assessment Project is designed to provide feedback on the two courses
that comprise the general education communication requirements. Although not
all facets of the general education program have been evaluated, the results
of studies conducted thus far suggest that the requirements are working well.
In particular, preliminary results indicate that students completing the Comm-B
requirement positively evaluate their Comm-B class, have confidence in their
writing, public speaking, and library research skills, and perform well with
respect to a variety of writing performance criteria. A second study focused
on one of the Comm-A courses (Communication Arts 100) revealed significant
decreases in studentsĖ communication, writing, and library anxieties over
the course of the semester. To provide input into general education
instruction, the Verbal Assessment Project has also developed a number of
channels for communicating with the university community. Most notably, the Verbal
Assessment Bulletin disseminates information about assessment activities
in the form of a campus newsletter. The future agenda for the Verbal
Assessment Project includes studies of communication instruction outcomes
among both graduating seniors and students in the Comm-A course, as well as
continued efforts to provide insights and tools related to verbal assessment
to constituencies at the University of Wisconsin. This
document provides a description of verbal assessment at the University of
Wisconsin-Madison. Following a brief overview of the Verbal Assessment
Project, specific assessment activities spanning the 1998-99 and 1999-00
academic years are summarized. Then, the various channels through which
information about verbal assessment has been communicated to the university
community are reviewed. The concluding sections highlight the agenda for the
Verbal Assessment Project in 2001 and directions for future verbal assessment
efforts. The
Verbal Assessment Project: An Overview Since
the fall of 1997, the Verbal Assessment Project has focused on evaluating the
general education communication program. Broadly speaking, the objectives of
general education instruction in communication address two types of student
outcomes. First, students should manifest improved skills in writing,
oral communication, and information literacy. In addition, students should
develop attitudes about the process of writing, speaking, and accessing
information resources that promote the implementation of those skills. Thus,
the mission of the Verbal Assessment Project is to provide insight into the
impact of general education communication courses on the associated abilities,
knowledge, and attitudes of University of Wisconsin students. (Appendix A
includes a summary of the context for verbal assessment at UW-Madison
submitted as part of the 1997-98 Verbal Assessment Report.) Verbal
Assessment Activities: 1998-2000 In
the 1997-98 academic year, the Verbal Assessment Committee addressed the need
to clarify objectives, review assessment alternatives, and develop a
long-range plan for assessment activities. The results of those efforts were
formalized in the 1997-98 Verbal Assessment Report (available upon request).
Although the timing and implementation of specific verbal assessment
activities have been revised along the way, the Verbal Assessment Project has
in large measure attended to the principles outlined in the earlier report.
The following sections summarize the specific assessment studies conducted in
the two years since that report was filed. Communication
B Study The
Communication-B course comprises the second and final campus-wide requirement
in the general education communication program. In contrast to the
Communication-A course, Comm-B classes are offered in a variety of departments
throughout the university. In addition, classes that fulfill this requirement
may take one of two forms: (a) classes that provide advanced instruction in
written or oral communication, or (b) classes focused on conventions of
inquiry particular to specific fields of study (i.e., content courses with
substantial writing components). Although the Comm-B course was originally
conceptualized as a low-enrollment class, Comm-B classes range from small
seminars to large faculty lectures with divided discussion sections. In sum,
Comm-B classes are presented in numerous departments, represent diverse
disciplines, and take a variety of instructional forms. The assessment of the Comm-B course focused on writing
performance and self-reported attitudes about writing, speaking, and library
use among students completing the Comm-B component of the general education
requirements. By focusing on performance and attitudes at the end of a
semester of Comm-B instruction, this assessment effort highlights the outcomes
associated with both the Comm-B course and the two course general education
sequence. In addition, an examination of writing performance and studentsĖ
beliefs and self-perceptions attends to both skills and attitudes as outcomes
relevant to studentsĖ subsequent academic performance. The
study conducted was designed to assess the Comm-B component of the general
education communication program, while being sensitive to the great diversity
of courses that meet this requirement. These goals demanded a research design
that would support conclusions about the Comm-B course, in general,
while at the same time integrating information about a studentĖs particular
Comm-B class and academic background. To these ends, we randomly sampled
sections of Comm-B classes offered in the spring semester of 1999, surveyed a
subset of students enrolled in these classes, surveyed instructors about both
their courses and the students in our sample, gathered descriptive profiles of
students and courses from campus databases, and evaluated final papers that
students submitted as part of their workload in the Comm-B classes. In
drawing the sample of Comm-B classes and students, we were guided to two
goals. First, to provide a foundation for generalizable conclusions about the
Comm-B course, we sought a large sample of classes representing the diversity
of departments involved and capturing various instructional formats. Second,
we focused primarily on students subject to the general education requirements
who were enrolled in their first Comm-B class. Although many students take
more than one Comm-B class, examining outcomes for students at the end of a
first Comm-B class assesses the general education requirements as formally
specified. Classes were randomly sampled from departments across
campus, and a total of 70 classes were visited to solicit participation and
informed consent from students. The primary participants in the study were
students enrolled in their first Comm-B class who consented to participate (N
= 446). To provide a broader basis for assessing studentsĖ perceptions of
the Comm-B course, a smaller number of students who had previously taken a
Comm-B class (N = 53) also completed a subset of procedures. At the end of the semester, student participants were
asked to complete a web-based survey about their Comm-B course. In addition, a
profile of each participant=s
academic background was assembled from student record data retrieved from the
ISIS system. Instructors were also asked to provide final papers written by
students, which were subsequently evaluated by a team of trained raters
according to a variety of writing performance criteria. Further information
about student participants was gathered from questionnaires completed by
instructors. The survey of instructors also solicited information about the
class itself, which was supplemented by details retrieved from the timetable
database. In total, the Comm-B study yielded a wealth of information from a
large and representative sample of courses and students: 369 students
completed surveys, papers were collected for 385 students, 70
different sections from 24 departments were represented, and 58 instructors
completed surveys. Although
the student sample was drawn in spring semester of 1999, assembling the
components of the data file occupied much of the 1999-2000 academic year.
These activities included substantial effort devoted to evaluating the Comm-B
student papers and ongoing attempts to obtain student academic profile data. Evaluating
writing performance as evidenced in the Comm-B student papers required
developing appropriate criteria, assembling a team of paper readers, and
training paper readers to apply the criteria in a consistent fashion. An
initial set of writing performance criteria was developed by a subcommittee of
the Verbal Assessment Committee comprised primarily of the Comm-A course
directors. To evaluate the relevance of these criteria to the Comm-B course,
we asked instructors participating in the Comm-B study to rate the extent to
which they attended to each performance outcome in their class. Then, seven
graduate students who had experience as instructors in the Comm-A course were
hired to evaluate the Comm-B student papers. As part of training efforts, the
paper readers applied the criteria to a small subset of papers and met to
discuss decision rules and to explicate the criteria; over the course of three
training sessions, a detailed coding manual was developed. Once a sufficient
level of inter-rater reliability was obtained, each paper was evaluated by a
subset of four readers. Reliability was assessed periodically to ensure that
raters were maintaining equivalent standards for applying the criteria. A
second component of the study addressed in the 1999-2000 academic year focused
on accessing student record data for academic profile information. At the
direction of Al Cohen of Testing and Evaluation Services, the Division of
Information Technology (DoIT) created a program to retrieve information from
the ISIS system. The implementation of ISIS and competing campus priorities
resulted in substantial delays in this portion of the project. A complete set
of student profile data was generated in June of 2000. Verification of the
data file has continued to identify errors and omissions; therefore, efforts
to secure this component of the Comm-B study are ongoing. Given
the difficulties assembling the student profile data, a full report of the
results of this assessment effort is pending; however, descriptive statistics
for the measures of student outcomes included in the study provide insight
into the overall effectiveness of the Comm-B course. Perceptions of Comm-B
classes solicited by the student surveys revealed that students positively
evaluated both the writing skills they obtained from their classes and the
overall value of the Comm-B course. In addition, students reported relatively
low levels of anxiety and substantial confidence associated with their
writing, public speaking, and library research skills. Moreover, ratings of
writing skills based on papers that students completed at the end of the Comm-B
classes indicated that students, on average, performed well with respect to a
variety of basic writing performance criteria. Although further analyses are
needed to clarify how these outcomes vary as a function of course format,
curricular features, and student characteristics, the preliminary findings
suggest that the Comm-B course in general is functioning effectively. A
full report on the procedures employed and the measures available as of this
writing are provided in Appendix B. Communication
Arts 100 Study The
development of self-report measures to assess attitudes about writing,
speaking, and information literacy in the 1998 Senior Survey (see Appendix D
of the 1997-98 Verbal Assessment Report) provided a tool for examining
attitudinal outcomes associated with particular course experiences. In an
application of these measures, the impact of Communication Arts 100:
Introduction to Speech Composition on communication, writing, and library
anxiety was investigated. The
Comm-A component of the general education communication requirement is
intended to promote general skills and attitudes as a foundation for
subsequent course work and careers after college. As part of this effort, the
Comm-A course incorporates a library training module taught by the Library
User Education Office. By focusing on attitudinal outcomes of in the form of
communication, writing, and library anxiety, this assessment study examined
whether CA100 promotes self-perceptions relevant to students=
academic performance beyond the Comm-A course. The
sample for the study included students (N = 419) enrolled in CA100 in
fall of 1999. Participants completed a survey of library, communication, and
writing anxiety on the first day of class and at the end of the semester. On
both occasions, surveys were completed during class time. For measures of
communication, writing, and library anxiety, students indicated the extent to
which a series of statements applied to them. In total, 11 specific academic anxieties were measured, including anxiety about
(a) communication, in general, (b) dyadic communication, (c) group
communication, (d) public speaking, (e) writing, in general, (f) having
writing evaluated, (g) using the library, in general (h) finding resources in
the library, (i) starting projects in the library, (j) relative library
ability, and (k) asking librarians for help. Across
all 11 measures of communication, writing, and library anxiety, students
reported significantly less anxiety at the end of the semester relative to
their scores on the first day of class. Although the magnitude of decreases in
some academic anxieties varied as a function of students=
demographic characteristics, results indicated reductions in communication,
writing, and library anxiety for all subgroups examined. Predictably,
writing and library anxiety at the beginning of the semester were generally
greater among freshman students, and declined across the sophomore, junior,
and senior ranks. Nonetheless, students in all classes experienced equivalent
decreases in communication, writing, and library anxiety over the course of
the semester. Thus, although some general improvement in academic anxieties
may be likely to occur simply through college experience, this study
highlights the added benefits accruing from enrollment in CA100. An
examination of demographic differences in communication, writing, and library
anxiety at the beginning of the semester also provides insight into the needs
of particular groups of students enrolled in CA100. Specifically, students who
did not have a public speaking course in high school and upper class students
who have perhaps delayed fulfilling the Comm-A requirement reported
significantly higher levels of anxiety about public speaking. A
full report of Communication Arts 100 Study is provided in Appendix C. Senior
Survey Spring
2000 marked the eighth semester for students who matriculated in the fall of
1996 - the year that general education requirements
went into effect. To evaluate attitudes and beliefs relevant to public
speaking, writing, and library use in this population, the Verbal Assessment
Project agenda called for a replication of the 1998 Senior Survey. These data
could also be compared to the baselines established by the 1998 survey to shed
light on the overall impact of the general education communication program on
students= attitudes and beliefs about associated skills. As
in 1998, the Senior Survey was comprised of two primary components assessing
communication and information literacy. The oral and written communication
literacy component of the survey included measures of (a) communication
apprehension, (b) the perceived importance of various communication
competencies, and (c) confidence in abilities to perform facets of oral and
written communication tasks. The information literacy component addressed (a)
confidence in abilities to use library resources, (b) library use, (c) the
utility of any previous information management instruction, and (d) library
anxiety. In a third component of the survey, a number of potential impediments
to student learning were included. The
UW Survey Center was commissioned to implement the survey in April of 2000, to
coincide with the timing of the previous survey conducted in 1998. Due to
problems associated with the implementation of ISIS, the Survey Center was
unable to obtain a random sample of seniors meeting the sampling criteria.
Thus, the replication of the Senior Survey has been postponed until the spring
semester of 2001. Mechanisms
for Communicating with the University Community The
mission of the Verbal Assessment Project includes serving as a source of tools
and information related to communication instruction and verbal assessment.
Accordingly, verbal assessment activities have included providing information
about the Verbal Assessment Project to the university community. These efforts
are summarized in the following sections. Newsletters The
Verbal Assessment Committee created a newsletter to provide information about
verbal assessment activities. Two issues of the Verbal Assessment Bulletin
have been distributed thus far. The first provided information about the
long-range verbal assessment plan, introduced readers to the Comm-B study
underway at the time of publication, addressed the university=s
efforts to meet student demand for seats in Comm-B classes, and summarized
Comm-B instruction provided by the UW Library User Education Office. The
second issue reviewed the procedures of the Comm-B study, included a detailed
summary of the procedures and criteria employed to evaluate writing
performance, reported on Comm-B instructor training provided by the L&S
Program in Writing Across the Curriculum, and presented the results of the
CA100 study of communication, writing, and library anxiety. The
Verbal Assessment Bulletin is circulated to faculty, lecturers, and
instructors instrumental in presenting Comm-A and Comm-B courses. In addition,
copies of the newsletter are sent to a variety of campus administrators. The Verbal
Assessment Bulletin was also shared with representatives of the North
Central Association during their visit to the UW campus as part of the 1999
re-accreditation study. Thus, the Verbal Assessment Bulletin has
developed as an important and ongoing source of information about verbal
assessment efforts. Information
about the Verbal Assessment Project has also been disseminated through
articles in Time to Write, the newsletter of the L&S Program for
Writing Across the Curriculum. In March 1999, Time to Write carried a
lengthy cover story addressing fundamental questions about verbal assessment,
reviewing the general goals and specific agenda for the Verbal Assessment
Project, and addressing the relationship of verbal assessment activities to
the faculty=s
ongoing teaching efforts. A follow-up report in the November 1999 issue
provided a progress report on the Comm-B study. In
sum, these newsletters have disseminated information about the Verbal
Assessment Project to those faculty and instructional staff who are centrally
involved in delivering the general education communication courses on this
campus. TA
Training The
Verbal Assessment Project has provided direct feedback into the training of
teaching assistants in Comm-A and Comm-B courses. Directors of the Comm-A
courses in Communication Arts (Professor Stephen Lucas), English (Professor
Michael Bernard-Donals), and Agricultural Journalism (Professor Marion Brown)
all serve on the Verbal Assessment Committee. Brad Hughes, Director of the
L&S Program for Writing Across the Curriculum, is both a member of the
Verbal Assessment Committee and responsible for Comm-B Instructor Training
Sessions on this campus. The
participation of these individuals on the Verbal Assessment Committee provides
a direct avenue for implementing insights gained through verbal assessment
efforts. For example, preliminary results from the Comm-B study have been used
by Mr. Hughes to supplement training sessions for teaching assistants teaching
their first Comm-B class. In particular, comments from the instructor survey
portion of that study provide insight into the rewards and challenges of
teaching Comm-B that is useful to teachers preparing for their first semester
in the course. In addition, Professor Solomon participated in an English 100
staff meeting focused on the development of criteria for evaluating student
writing; the procedures and results of the student paper evaluations conducted
as part of the Comm-B study were central to this discussion. Through
these channels, the Verbal Assessment Project provides direct and timely
feedback into TA training. Reports
to Administrative Committees Professor
Solomon serves on the L&S General Education Committee, the L&S
Curriculum Committee, and the University Assessment Council. Thus, she is
positioned to provide information derived from the Verbal Assessment Project
as it becomes relevant to the activities of these committees. For example, the
review of additional communication requirements in the College of Letters and
Science (i.e., the ÏComm-C2Ó requirements) conducted by the
L&S Curriculum Committee in 1999 was informed by Professor
Solomon=s
understanding of enrollment and instructional issues in the Comm-B courses. As
a more direct channel for disseminating information, Professor Solomon
presented a summary of the Comm-B assessment efforts to the L&S General
Education Committee in fall semester of 1999. Because Professor Sherry Reames,
chair of the Communication Implementation Committee, also serves on the
L&S General Education Committee, information about the Comm-B study was
indirectly channeled to the committee responsible for overseeing the general
education communication courses. Professor
Solomon also reported directly to the University Academic Planning Council in
December, 2000. This report included a review of the Verbal Assessment Project
goals and long-range agenda, as well as a summary of procedures involved in
implementing the Comm-B assessment study. Information
about verbal assessment activities focused on information literacy has also
been shared with the Library Planning Committee. Abbie Loomis, campus
coordinator for Library User Education, serves on the Verbal Assessment
Committee. Thus, there are direct lines of communication between the unit
primarily responsible for library instruction and the Verbal Assessment
Committee. More specifically, Ms. Loomis has shared results from the CA100
study highlighting students= anxieties about using the library with the
Library Planning Committee. In
sum, the Verbal Assessment Project has a number of avenues for providing
information about general education instruction and outcomes to the relevant
policy making committees at the UW-Madison. Conferences
and Colloquia
Professor
Solomon has been available for conferences or colloquia focused on more
specific aspects of the Verbal Assessment Project. Although the results of the
Comm-B study do not allow an evaluation of individual courses or instructors,
instructors involved in the study can get information about outcomes in their
particular course from Professor Solomon. (For reasons of confidentiality,
information about an individual course is not provided to anyone other than
the specific instructor who participated in the Comm-B study.) More generally,
Professor Solomon has consulted directly with faculty and instructors teaching
Comm-B courses to clarify how their courses might better address the
objectives highlighted by assessment efforts. Addressing
the challenges inherent in assessment has also been the subject of meetings
sponsored by the University Assessment Council. As part of these efforts,
Professor Solomon participated in a colloquium on using web-based surveys for
assessment purposes. Because student participants in the Verbal Assessment
Comm-B study were drawn from a number of courses across campus, the student
survey component of that study was implemented using a web-based survey
supported by Testing and Evaluation Services. This methodology disseminated
the survey quickly to a large number of students, allowed students to complete
the survey at their convenience, and directly translated survey responses into
a usable data file. Difficulties contacting students and alumni to complete
surveys about educational programs are common to all academic units involved
in assessment; therefore, the lessons learned from the Verbal Assessment
Project are widely applicable. Verbal
Assessment Project Agenda: 2001 The
long-range plan proposed in the 1997-98 Verbal Assessment Report called for
assessment efforts focused on three levels: (a) outcomes associated with the
Comm-A course; (b) outcomes associated with the Comm-B course; and (c)
outcomes associated with the cumulative undergraduate education. Assessment
efforts thus far have most directly targeted outcomes associated with the Comm-B
course; therefore, the agenda for the near future involves attending to the
other outcomes levels. In addition, the Verbal Assessment Committee will
strive to improve the dissemination of verbal assessment results and resources
to the university community. In total, the verbal assessment agenda for the
next year includes the following four activities. Senior
Survey The
Senior Survey, originally scheduled for the spring of 2000, will be
implemented in the second semester of the 2000-01 academic year. This survey
will assess attitudes and beliefs relevant to public speaking, writing, and
library use among students who matriculated under the general education
communication requirements. Thus, a comparison of these data to the baselines
established by the 1998 survey will shed light on the overall impact of the
general education communication program on students=
attitudes and beliefs about communication skills. Communication
A Study An assessment study focused on the Comm-A course will
be developed during the 2000-01 academic year for implementation in the fall
of 2001. The implementation of the Comm-A course has been relatively smooth;
therefore, assessment efforts thus far have focused on outcomes associated
with the general education requirements overall and the Comm-B course in
particular. With studies addressing those outcomes completed, the Verbal
Assessment Committee will examine teaching and learning in the context of the
Comm-A course. Verbal
Assessment Bulletin Thus
far, two issues of the Verbal Assessment Bulletin have been produced at
a rate of one per year. Because this newsletter constitutes a primary means
for communicating with faculty, instructors, and administrators involved in
the general education communication program, increasing and regularizing
production of the Verbal Assessment Bulletin is warranted. At present,
two issues of the newsletter are planned for the 2000-01 academic year.
Verbal Assessment Toolkit
Part
of the value of the Verbal Assessment Project is the development of tools that
can be used by other programs, departments, or instructors for assessment
purposes. For example, measures of library anxiety that were developed in the 1998 Senior Survey and refined in
the CA100 study could easily be used to assess the effectiveness of other
programs targeting the development of library research skills. Similarly, the
survey developed as part of the Comm-B study to assess studentsĖ evaluations
of their Comm-B classes could be used by individual instructors in their own
courses. Likewise, the writing performance criteria developed by the paper
readers in the Comm-B study could be implemented as a pedagogical tool in any
class with a substantial writing component. Thus, verbal assessment efforts in
the coming year will include assembling assessment instruments into a toolkit
available to other units on campus. Directions
for Future Efforts Assessment
efforts beyond 2001 will necessarily be informed by the results of the studies
previously conducted or planned. In addition, studies that target particular
communication instruction objectives (e.g., oral communication skills) are
needed to supplement the general picture that should emerge from the research
conducted thus far. Ultimately, the Verbal Assessment Project will evolve into
a balanced mix of studies that combine to support generalizable conclusions
about the implementation of the general education communication program and to
provide specific insight into the effective operation of the courses
comprising the general education communication requirements.
Appendix
A - The Context for Verbal Assessment at UW-Madison Appendix B - Spring 1999 Communication-B
Study Appendix C - Communication Arts |