Anne Goodsell Love, Pat Russo, Vincent Tinto

         During the 1991-92 and 1992-93 academic years, we undertook a study of exemplary collaborative learning programs in higher education as part of the National Center on Postsecondary Teaching, Learning, and Assessment (NCTLA), a federally-funded research consortium devoted to examining factors that influence student learning in higher education. We studied three learning community programs: the Coordinated Studies Program at Seattle Central Community College, Freshman Interest Groups at the University of Washington, and Learning Clusters at LaGuardia Community College. In each case, we collected a variety of data using two survey questionnaires. We also made multiple site visits during which we carried out participant observation of classrooms and held a variety of formal and informal interviews with students.

         In this chapter, we address the following questions about our research:

·      What did we want to find out about collaborative learning programs?

·      How did we go about doing the research?

·      What might we do differently the next time?

·      What tips or advice would we give others who might want to pursue this kind of assessment?

What Did We Want to Find Out about Collaborative Learning Programs?

         At each institution we were interested in student learning and persistence, specifically among first-year students. We wanted to know how participation in learning communities shaped student experiences, especially in the classroom, relative to more traditional curriculum structures. In turn, we wanted to know to what degree and in what manner those experiences with collaborative learning came to influence academic performance and persistence over the course of the first year of college. That is to say, we wanted to understand how participation in a learning community program influenced the process of learning and persistence during the first year of college.

         In carrying out the research, we sought to balance two approaches. One was an approach driven by a conceptual orientation about student involvement and student persistence that was drawn from the theories of Astin (1984, 1992) and Tinto (1975, 1987, 1994). These theories argue that the effect of learning community programs could be understood best through a study of social and academic behaviors of students, especially those behaviors that bring students together with their peers and faculty members both inside and outside the classroom. It was this conceptual orientation that guided the development of our survey questionnaires.

         We balanced this conceptual orientation by also taking an inductive approach, casting our nets widely in an effort to be open to unexpected phenomena. We assumed that Coordinated Studies Programs, Learning Clusters and Freshman Interest Groups were effective ways to respond to the academic and social needs of students. This "optimistic" approach (Bogdan & Taylor, 1990) allowed us to search for the ways in which the programs were positive while at the same time holding the notion of collaborative learning "problematic." Consequently, our findings emerged directly from the data about each program. This freed us from making value judgments (such as which program works best) and allowed us to focus on understanding how each program met the needs of students at each institution and how it shaped student learning and persistence.

         This orientation guided our site observations, analysis of institutional and program documents, and formal and informal student interviews. We wanted to know how students understood their experience as framed by the context of their lives. In other words, we wanted to listen to the students and understand how they made sense of their experiences.
This balancing of a conceptual orientation with an inductive orientation comprised the multi-faceted approach of our research. For example, it was important for us to find out about some often-mentioned outcomes of collaborative learning-student performance as measured by grades, student involvement, and student persistence during subsequent semesters. We also wanted to be able to compare our findings from each institution. The comparative and longitudinal nature of these questions could be answered by the surveys. Since we also were interested in finding out how the learning community programs influenced students, and in what ways, we relied on qualitative inquiry through observation, interviews, and document analysis.

         In order to keep these two distinct foci of inquiry separate we divided responsibilities for each site. Vince developed the conceptual design and conducted survey research at each site, while Anne and Pat developed the qualitative design and conducted observations and interviews at their respective sites. In the following section we provide some specifics about our research design.

         It bears repeating that it was not our intent to carry out an evaluation of program effectiveness or a formative assessment that would provide program participants with information enabling them to improve the program. Though some useful information about possible improvement did arise, it was not our intent to gather such information. Rather, we gathered information about program impact in order to understand how learning communities work. We saw our audience as faculty and administrators at other institutions across the nation who might consider the establishment of learning community programs. We hoped to convey to them our findings in a way which would enable them to understand the potential impact that such programs could bring to their students and campuses. As a corollary, our choice of institutions-urban public two and four-year colleges-was designed to convey a very important argument: if it was possible to construct successful learning community programs in those challenging settings, it was possible to do so anywhere.

How Did We Go about Doing The Research?

         We have taken pains to describe carefully the goals of our study and the audience to which we wished to speak because they governed our decisions about how we would go about doing the research. If we had been interested in an evaluation of program impact and/or in speaking to the needs of specific program faculty and administrators who were concerned about program improvement, our research design would have been different. Invariably, decisions about the questions one seeks to answer and the audiences to which one wishes to convey research findings must precede decisions about research design. In part this is the case because all decisions regarding research design involve a series of choices among alternative courses of action and a range of trade-offs that shape what can be learned from the study. Within the world of limited resources, time, and energy, no one study can answer all possible questions about program functioning, impact, and possible improvement. Thus in designing a study it is necessary to decide what is essential to that study and what must wait for another study.

         The way that we resolved this issue was to develop a multi-person, multi-method design which allowed both qualitative and quantitative methods to balance each other and which provided the people power necessary for the labor intensive nature of the design. We conducted the majority of our research at each site over the course of one academic year. For each site, we carried out both comparative longitudinal survey research and qualitative research (Table I outlines our site visits and our activities).

         For the quantitative design, Vince used survey questionnaires to sample and question first-year students in both learning community program classrooms and comparison classrooms. Two surveys were administered, one at the outset of the first quarter, another later during the first year. As we described earlier, these

TABLE I

enabled us to collect a range of comparative information not only on student attributes but also on student social and academic behaviors over the course of the first year. In the following fall, we collected information from institutional records on student academic performance and continued registration.

         In the survey we asked a variety of questions about student attributes (e.g. gender, social background, employment, and educational plans), and about their patterns of behavior, sense of intellectual and social development, perceptions of the college environment, and re-enrollment plans. The information about personal attributes was merged with existing information from institutional records (e.g. high school grade point average, ability test scores) to form a data set that enabled us to take account of individual attributes in our analysis of program outcomes. Data on student perceptions and behaviors were derived from student responses to a modified version of Pace's Quality of Student Effort Scale (Pace, 1983). These questions, in Likert-scale form, asked individuals to indicate how often they had engaged in various academic and social behaviors such as talking with professors after class, having discussions after class with fellow students about class material, and engaging in various writing activities, library activities, and extracurricular activities. Other questions asked students to assess their own social and intellectual development over the period of the program. In all analyses, descriptive and multivariate, we sought to compare student responses in the learning community programs with student responses in non-learning community programs. We sought to understand how the differences between student responses helped account for differences in subsequent persistence.

         Anne and Pat spent a full week on campus three times over the year. During those visits we collected a variety of data via participant observation, formal and informal interviews, and document analysis. At the beginning of our field visits some of our research questions were: What do Freshman Interest Groups (FIGs), Coordinated Studies Programs (CSPs), or Learning Clusters (LCs) look like? How do students in these programs spend their time? What do students do, and what do they talk about? As the research progressed, we asked more specific research questions based on what we had seen and heard already: What distinctions do students make between learning communities and other experiences? How do students talk about their learning in these programs? To get at these questions, we asked interview questions such as: Tell me about the project you are working on. What advice would you give someone else about succeeding in a CSP? How did the FIG or LC fit in with the rest of your college experience? The intent of the classroom observation and interviews was to determine which aspects of the learning community programs were important to students, and to understand the context of student comments. All three of us maintained phone contact with faculty, staff, and students between visits. The following year we returned to our sites with drafts of our reports to share with faculty and staff. Our intent was to bring the feedback loop full-circle, checking their reactions against our findings.

         The research design called for multiple methods and multiple researchers. This added yet another facet to our approach, one that allowed for richness and depth in our analysis, and one that we discuss more in the next section. (For a description of a multi-site, multi-researcher study that used a single method, see Whitt and Kuh, 1991.) Because we wanted the findings from each part of the research to inform the others, we made our site visits to two of the institutions concurrently. This allowed the three of us to talk on a daily basis, "debriefing" with one another at the end of each day. The sharing of stories, pushing each other's thinking with questions, and problem-solving that occurred each day during our site visits was one of the most exciting and valuable parts of our research process. Some qualitative data analysis occurred during these sessions, allowing us to make informed plans for the following days. This was especially important because our periods of fieldwork were of such short but intense duration. Had we been studying institutions nearer to our own (we were all at Syracuse University at the time), we might have stretched out our observations and interviews over the weeks and months. As it was, we wanted to ensure that we were spending our time in a way that would help us get the broadest range of answers to our questions.

         When we were not at the research sites, we met on a weekly basis to continue sifting through data, examining themes that emerged, and challenging each other's ideas. As themes emerged that seemed to be consistent within each site, information from the other sites repeatedly prompted us to re-examine the data and search for other possible patterns. For example, the boundaries between work, social life, and college were clear for students at LaGuardia Community College and Seattle Central Community College because students commuted to classes. College classes were only one of many important commitments in the students' lives. At the University of Washington, however, the boundaries were not so clear because students lived, worked, socialized, and attended classes on campus. From examples such as these we found that, rather than muddying the waters, these differences between institutions helped to sharpen the picture at each institution. The contrast that emerged was useful for our understanding of the learning community programs because it was so vivid.

         The most important part about how we went about this research, however, was not when we met or how many site visits we made. Rather, it was the truly collegial, collaborative manner in which we did the research. Although we are three very different people coming from different backgrounds, we worked hard to listen to each other, encourage each other, and appreciate the contributions that each of us could make. Vince developed the initial vision, collected resources, and brought experience at carrying out a large-scale, national research project. Throughout the project he kept the larger picture in focus for us. Pat had experience with qualitative research and the ability to see beyond the obvious themes of the data we were collecting. Anne had a background in higher education and collaborative learning literature, and the organization to keep us on track when our enthusiasm carried us astray. We learned first-hand what the students, faculty, and staff at these institutions had been telling us-in collaborative research, as in collaborative learning, the sum of our work was indeed greater than its parts. Sharing our insights together led us on to understandings we might not have made alone.

         A key aspect of our collaborative research project was collaboration. We began working together two months before our first site visit. From the first, we began to negotiate about what we thought we should be doing and what we might find. We discussed survey questions, participant observation strategies, and interview questions. These kinds of negotiations became quite characteristic of our meetings throughout the two-and-a-half years we spent on the project. The probing, challenging, and give-and-take nature of these interactions was essential to our research process. In fact, it was our research process. The continued negotiation led to continued understanding, enabling our research analysis to be a dynamic process, open to new insights and understandings.

         If this description of our research process as a continual negotiation of the meanings we were making of the data sounds adversarial, it was not. It was clear to each of us at the outset that this was to be a team effort. Over the course of our frequent meetings and concurrent site visits we developed respect for each other's abilities, trust in each other's intentions, and the patience to let the data unfold. We came to recognize that it was these factors-frequent meetings, the development of trust and respect, continued negotiation of research strategies and analysis, and the dynamic nature of our discovery-that were the key elements of our collaborative inquiry.

         Our collaborative inquiry was not limited to just the three of us. We found collaboration and negotiation with faculty, staff, and students at all the sites to be essential. Although Vince made initial contacts to gain official permission to conduct our research, we all worked to establish trust and communication patterns that would allow us access to meetings, classes, and ultimately to students' ideas and thoughts. For example, Pat and Anne set out to talk with students only, then quickly realized that we needed to talk with faculty in order to gain their perspective on the learning community activities, gain some historical perspective for each program, and get access to certain classes and meetings. In addition, participants at each institution paved the way for us through the sometimes considerable bureaucracy (getting to and around campuses, finding parking, obtaining a temporary ID). Finally, it was important to explain a research design that was not entirely familiar to research officers at different sites. The qualitative aspect of the design was not a conceptual orientation that is used regularly in offices of institutional research. This process of initiating communication, listening to understand how the participants saw their world, and adjusting our questions to address the key themes that they presented made it clear to the participants that we were interested in the issues that were important to them. In each of the schools we were welcomed wholeheartedly. Our impression was that the participants cared about what they were doing, wanted to share their experiences with us, and were willing to be part of the ongoing conversation about learning communities.

         Although this chapter is more about methods than results, we would like to share our findings briefly. To summarize that research, we found that participation in a learning community enabled students to develop supportive peer groups that helped bond them to the broader social life of the college while at the same time engaging them more fully in the academic experience. Groups that formed for course-related purposes often extended beyond the classroom for informal gatherings and study sessions. Furthermore, students were influenced by participating in a setting in which sources of learning came from a various faculty perspectives not just one. When several instructors were brought together to teach collaboratively, students' learning experiences took on an intellectual richness that traditional courses could not match. Students in learning communities were more socially and academically involved in college life, more positive in their views of the institution, and more likely to persist beyond the first year. Most importantly, these differences held even when student attributes and issues of possible self-selection were taken into account. These impacts of learning communities were as prevalent among underprepared students as for their more prepared peers. Learning communities work for many types of students, including those typically excluded from the mainstream of academic life because of deficient academic preparation.

What Might We Do Differently the Next Time?

         Though we are confident about the validity of our findings, we are aware that the research could have been somewhat enhanced had we done some things differently. However carefully planned, it is invariably the case that the carrying out of a research study leads researchers to the recognition that some aspects of the study would work better if adjusted slightly.

         First, we would have liked to make more site visits or extend the visits we made. Because of the high cost of travel between our institution and the research sites, we were prohibited from making more frequent visits. If we were to do a similar study, we would attempt to make more regular visits. This would allow us to refine our research strategy in a number of ways. First of all, a site visit prior to the development of the survey would have been valuable in allowing us to refine our survey questions. Our observations probably would have led us to gear more of our questions toward students' in-class activities, and the nature and duration of in-class interpersonal relations. Additional site visits also might have enabled us to become less intrusive to the students. Had our presence been less noticeable, our interactions might have become more informal, allowing us to see and hear more than we did. More frequent visits would have allowed us to see more of the students' regular routine throughout the semester, and see any patterns of peer group growth over the semester.

         If we were to make additional site visits, a final visit might have been structured so as to take some of our findings back to the students. Within the qualitative design we frequently checked our thinking by feeding our understanding back to the students with whom we spoke. Yet the value of an additional visit would have been a final or summative confirmation (or disconfirmation) that we had an accurate sense of what went on in the learning community programs. Furthermore, it would have been a small way of giving back to those who had given us our information.

         During the site visits, Pat was able to meet with a few groups of students, which allowed her to hear their reactions to each others' comments. The use of these informal group meetings served as a check that students were not necessarily editing their comments for the benefit of our research. They allowed Pat to hear the kinds of conversations that went on among students. If we were to do it again, we might try to establish or sit in on some of these groups at the other sites, as well.

         Another modification in the research process might have been to use additional resources to more carefully control the sampling of comparison group students and to extend the sampling period so as to permit us to more clearly distinguish program experiences from subsequent program impacts. Regarding the latter, we were obliged to restrict our two-year college sampling frame to two surveys administered at the start and at the end of the Fall quarter. At the largely residential university site we were able to sample at the start of the Fall quarter and in the middle of the Spring quarter. In large measure this was due to the difficulty and costs associated with trying to reach students after the end of the learning community program when they dispersed to many other courses. In the urban community colleges, as contrasted to the public university, it was immeasurably more difficult to reach students outside the walls of the college.

         Regarding sampling, we would have liked to carry out matched stratified sampling of our program and comparison group students so as to enable us to control more clearly for other attributes of students that may also shape learning and persistence. Though we collected data on a range of student attributes and employed a range of statistical controls in our analyses, a more complex-and considerably more expensive and intrusive-sampling plan would have allowed us to explore more carefully the manner in which program experiences shaped outcomes both between and within different types of student groups. In this regard, a final modification that we might have made would have been to examine our data across the lines of race, gender, and class. While we gave attention to these issues in some cases, a more complete examination of data according to race, gender, and class would have provided an even clearer understanding of the impact of learning communities.

         Overall, however, there are few things we would change. The multi-site, multi-method, multi-researcher approach was an essential way to get information that was meaningful to the researchers and the institutions. The willingness to suspend judgment, as well as test assumptions, was essential to the success of our research.

What Tips or Advice Would We Give Others Who Might Want to Pursue Assessment of Learning Communities?

         First, to repeat a point made earlier, it is essential that the assessment design be guided by the research question one seeks to answer and by the audience to which one wishes to speak. A willingness to consider alternative methods or change an existing design in order to address the identified research questions and meet the needs of the appropriate audience is critical.

         Second, we think it essential that assessment be multi-method and, by extension, conducted by a team of researchers. At a very minimum, the use of multiple methods provides considerably greater insight into program functioning and a degree of "cross-triangulation" that enhances the validity of the assessment.

         Third, we cannot overstate the value of participant observation and student interviews in the assessment process. Taking on the persona of someone naive to a campus or program and asking questions that get at underlying assumptions and biases is a sound way to begin a study. Our observations informed the construction of the survey in a variety of ways, shaping questions that were appropriate to each campus. Observations also helped shape the questions that we asked in interviews, so that our questions were based on already collected data rather than on our preconceived ideas of what we might find. Ultimately, it is the views of students, expressed directly or through questionnaires, that underlie the validity of assessment. Unfortunately, too many assessments either ignore student views altogether or distort them through the exclusive use of questionnaires.

         Fourth, we feel it important to maintain an open-ended approach to assessment. Though it is necessary to have a set of clearly stated objectives to guide assessment, it is equally necessary to be open to unexpected findings. Too often our preconceived notions about expected findings blinds us to what is important to program functioning. It follows that no assessment should be so fixed in conception and design that it cannot be modified.

         The notion of preconceived ideas is important to mention because we all use various cognitive schema that allow us to operate in our everyday lives. The assumptions and beliefs we hold about the ways in which our programs work, about the ways in which our institutions function (or malfunction), or about the ways we communicate among our colleagues or with students are important to recognize at the outset of a study. This is another reason why we would urge others to conduct their assessment in diverse teams. Hearing other perspectives allows us to recognize our own assumptions and beliefs, which in turn frees us to look at our research with fresh eyes and ears, and lets the data speak to us in ways that we might not have heard or seen before.

         Finally, we encourage others to make assessment an on-going part of what they do in learning community programs, and not to view it as a special one-time-only event. Assessment can be overwhelming if taken on too ambitiously or all at once. A study of learning communities can take many forms; we chose to combine a few of these ways into a larger research project that met the needs of our research and policy audience. The methods others choose are determined by a variety of factors such as their audience, resources, time demands, and accessibility to different kinds of data. In collaborative assessment, as in collaborative learning, there is no one right solution. Even the best-laid assessment plans involve trade-offs. The aspects that we think are most important for conducting assessment are to get started, make it on-going, make it longitudinal, spend lots of time listening to participants, make it collaborative (negotiating the meaning of the assessment and the data with others), and if possible, conduct it in conjunction with assessments at other sites.

REFERENCES

Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel, 25(4), 297-308.

Astin, A. W. (1992). What Matters in College. San Francisco: Jossey-Bass.

Bogdan, R. C. & Taylor, S. J. (1990). Looking at the bright side: A positive approach to qualitative policy and evaluation research. Qualitative Sociology, 13(2), 183-192.

Pace, R. (1983). College student experience: A questionnaire (2nd Ed.). Los Angeles: University of California Higher Education Research Institute.

Tinto, V. (1975). "Dropout from Higher Education: A Theoretical Synthesis of Recent Research", Review of Educational Research, 65 (Winter): 89-125.

Tinto, V. (1987). Leaving college: Rethinking the causes and cures of student attrition. Chicago: University of Chicago Press.

Tinto, V. (1994). Leaving college: Rethinking the causes and cures of student attrition. Second edition. Chicago: University of Chicago Press.

Tinto, V., Goodsell Love, A., & Russo, P. (1994). Building Learning Communities for New College Students. The Pennsylvania State University: The National Center on Postsecondary Teaching, Learning, and Assessment.

back to the Table of Contents