Development of an observation instrument to measure flourishing learning environments

Abstract

The present study expands the array of learning environment instruments by developing a walkthrough observation instrument that examines the five elements of flourishing classroom learning environments: (a) positive emotion, (b) engagement, (c) meaning, (d) positive relationships, and (e) accomplishment. The Flourishing Classroom Observation Measure was used to observe 202 classrooms from 22 rural schools located in Texas. The findings indicate that these classrooms were not “flourishing” to a great extent on the five key elements. None of the mean values for the five scales were found to be higher than 1.53 on the 3-point rating scale. The findings from this study also indicate that there were few significant differences by content area, grade level, or school.

Background

Improving the teaching and learning in schools is one of the greatest challenges facing educators today. Many educators have found that the current emphasis on high-stakes standardized testing has lowered the quality of classroom instruction in schools [20, 24, 38]. This problem is even more severe in rural settings because of the difficulty that rural schools have in recruiting and retaining high-quality teachers [9, 21]. In addition, many students in rural schools are exposed to deleterious conditions like poverty, poor school facilities, and the lack of up-to-date instructional resources that contribute to a pessimistic and harmful school and classroom environment.

Despite these persistent and severe educational problems related to rural education, there have been few systematic research efforts that have tried to address these concerns. The present study addresses the need to focus on these issues by examining the extent that classroom learning environments in rural schools are flourishing or contributing to students’ wellbeing and academic success.

The present study expands the array of learning instruments that have been widely used over the past several decades by developing a walkthrough observation instrument that examines the five elements of flourishing classroom learning environments. More specifically, the purpose of this study is to report the development and use of the Flourishing Classroom Observation Measure (FCOM). We examined: (a) the reliability, validity, and use of the instrument, (b) findings from the observations of 202 rural school classrooms, and (c) whether there are differences on the five FCOM scales by content area, grade level, and school.

Perspective/theoretical framework

In the last decade, the theoretical and empirical work in the field of positive psychology has begun to have an impact in the areas of psychology and education. The focal topic of positive psychology has become “well-being theory” that includes the following five elements: (a) positive emotion, (b) engagement, (c) meaning, (d) positive relationships, and (e) accomplishment [14, 30–32]. According to [30], (p. 29), “the goal of positive psychology in well-being theory is to measure and to build human flourishing”. He maintains that our goal should be to increase flourishing by increasing emotion, engagement, meaning, positive relationships, and accomplishment.

The wellbeing and flourishing theory has not been directly applied to school settings yet, but many of the five key elements relate to research on classroom learning environments. Several of the most widely used learning environment instruments (e.g., What is Happening in this Class?, My Class Inventory, Classroom Environment Scale) [10–13], for example, include scales such as student cohesiveness, teacher support, satisfaction, and involvement that relate to many of the elements of the flourishing theory. The theoretical and conceptual work of other learning environment researchers like Moos [22], Fraser and Walberg [12], and Wubbels and Levy [40] also relate to flourishing theory.

This line of inquiry is also similar to the growing body of research that focuses on the importance of supportive school relationships and classroom emotional environment [16]. Although socio-emotional behaviors have been widely investigated in terms of classroom climate and learning environment research, the focus on flourishing behaviors or flourishing learning environments is a very recent development.

The most common issue of past classroom environment research has been encapsulated by the investigation of relationships between students’ cognitive and affective learning outcomes and their perceptions of the classroom environment [2, 12, 39]. The present study, however, uses systematic classroom observation rather than student perceptions to assess classroom learning environments in rural schools. Many studies assessing classroom instruction and learning environment have relied on self-report data from administrators, teachers, or students (e.g., [12, 23]). These types of data, however, are often unreliable and tend to be upwardly biased in the direction of over estimating the actual quality of classroom instruction. Few studies have actually gone into rural classrooms to examine the quality of instruction and learning environment. It is especially important to observe classrooms in rural settings because of the historically, pervasive problems of poor instruction and disengagement that commonly occurs in those settings.

Methods

Participants

The participants were 202 PreK-12 classroom teachers in six school districts in south Texas. All of the districts were rural school districts with student populations ranging from approximately 500 to almost 12,000. Two of the school districts had a majority minority student population, while the other four districts served predominantly white student populations.

Instrument

The Flourishing Classroom Observation Measure (FCOM), a high-inference walkthrough instrument, was developed to examine the five elements of wellbeing and flourishing theory: (a) positive emotion, (b) engagement, (c) meaning, (d) positive relationships, and (e) accomplishment. The Flourishing Classroom Observation Measure is considered a walkthrough or walkabout instrument that is designed to obtain multiple snapshots of classroom practices in order to provide a rich data picture [5, 17, 33].

Walkthrough instruments have been widely used in school districts for the past decade [18]. They have been primarily used by school administrators for purposes such as (a) evaluating teachers [5], (b) diagnosing classroom instruction [29], and (c) improving student learning [19]. Most of these walkthroughs are designed to be quite short (3–5 min) [4], but for research purposes we decided to use a larger time period (20 min) based on our previous experience using similar classroom observation instruments [1, 37]. Other recent research also has found that 20 min is an appropriate time period for assessing classroom instruction [8].

The Flourishing Classroom Observation Measure was adapted from the Classroom Observation Measure (COM) [26], which measures the extent to which certain effective instructional strategies are demonstrated during a class period. The COM has been used in a number of studies and found to be reliable and valid [27, 28]. It also has been adapted and used recently in a number of studies [37].

The Flourishing Classroom Observation Measure was used at the end of the classroom walkthrough to rate, on a 3-point scale (1 = not observed at all; 2 = observed to some extent (once or twice); 3 = observed to a great extent (3 or more times), the extent that the five elements of well-being and flourishing theory: (a) emotion, (b) engagement, (c) meaning, (d) positive relationships, and (e) accomplishment were demonstrated during the observation period. Each element had at least three items that were used to measure the element. Some sample items for each of the elements are as follows: (a) Positive emotion—Students displayed positive affect toward teacher (+); Teacher displayed negative affect toward students (-); (b) engagement—Students were engaged in classroom activities (+); Students displayed disruptive behavior (-); (c) Meaning—Teacher related concepts to students’ actual lives (+); (d) Positive Relationships—Teacher appeared to have warm, supportive relationships with students (+); Student displayed positive engagement with peers (+); (e) Accomplishment—Students solved problems using real objects in the classroom (+); Teacher initiated project-based learning activities (+).

Procedures

Trained observers observed 202 classrooms in 22 schools from all six districts for approximately 20 min each. The teachers were aware of the week that the observations were scheduled, but they were not aware of the specific day or time that their class would be observed. Classrooms that were involved in nontraditional instructional contexts (e.g., testing) were avoided and attempts were made to revisit them at other days or times. Experienced classroom observers obtained a 90% coding accuracy with the instrument during initial training sessions. A sample of about 10% of the classrooms was used to determine the inter-rater reliability for the data in the present study. The inter-rater reliability was .84, which indicates a high degree of consistency among observers.

Results

An exploratory factor analyses was conducted with all the observation items on the FCOM. The factor analyses yielded ten factors accounting for more than 65% of the variance. However, upon further examination, the results were not interpretable so we decided to use the five a priori scales that were initially developed based on the flourishing theory. The individual items for each scale were aggregated and averaged to create each element score. The correlations among the elements were moderate, averaging about r = .53, indicating that there was some overlap regarding the discriminant validity of the scales.

Table 1 displays the means, standard deviations, and inter-correlations for the five scales. The means for the five elements ranged from 1.19 to 1.53, indicating that these elements were observed rarely in the 202 classrooms. The most positive element observed was Positive Emotion (M = 1.53), followed by Meaning (M = 1.36) and Accomplishment (M = 1.26). The least positive elements observed were Positive Relationships (M = 1.19) and Engagement (M = 1.20). It should be pointed out that the standard deviations for all the scales are relatively low (SD < .5) indicating that there was not a great deal variance to the extent that these were observed in the classrooms.

Table 1 Means, standard deviations, and inter-correlations for flourishing environments scales

Analysis of variance (ANOVA) was used to examine if there were statistically significant differences on the five elements by (a) district, (b) grade level, and (c) content area. The (ANOVA) results revealed that the only significant difference was for district on the Positive Emotion element, but the Tukey post-hoc test failed to reveal where the significant differences were occurring. No other significant differences were found between districts. The one-way ANOVA revealed significant differences between subject areas for the meaning element. The Tukey post-hoc test showed that “other” subjects were significantly higher than mathematics (p = 0.034). There were no other significant differences between subject areas. The one-way ANOVA revealed no significant difference between grade levels for the five elements.

Overall, the findings of the present study indicate that these rural school classrooms were not “flourishing” to a great extent on the five key elements of (a) positive emotion, (b) engagement, (c) meaning, (d) positive relationships, and (e) accomplishment. None of the mean values for the five elements were found to be higher than 1.53 on the 3-point rating scale. The findings from this study also indicate that there were few significant differences by content area, grade level, or school.

Discussion

The findings from the present study are discouraging since they suggest that nearly all of the classroom learning environments observed are not flourishing and there is little attention provided to socio-emotional learning. The low scores found for positive relationships and engagement in the present study are particularly alarming since both variables have been found to improve achievement outcomes for students [3, 15, 20]. Instruments like the Flourishing Classroom Observation Measure (FCOM) may help educators begin to focus on meaningfully constructed instruction and designing learning environments that build on the strengths of all students. The FCOM may also make educators more aware of the importance of quality of teacher-student relationships.

In the present study, we found the Flourishing Classroom Observation Measure (FCOM) had good inter-rater reliability, but only modest discriminant validity. Furthermore, improvement is required for the construct validity of the instrument. Additional larger-scale studies are needed to further examine the measurement properties of the instrument. We found that it was easy to train observers to use the FCOM and that it was easy to use in the classroom. We also found that the 20-min time frame for observation was ideal for research purposes, which allowed us to capture the quality of the instruction and the classroom environment. However, whether another time frame (e.g., longer or shorter) would provide a better snapshot of the classroom remains to be investigated.

These findings also raise several other important questions that need to be addressed in future studies. Most of these questions center on determining: (a) the extent that these elements relate to student cognitive and affective outcomes, (b) how the observational data relates to students’ perceptions of their classroom learning environment, (c) the factors that constrain teachers from increasing these key elements, and (d) the types of support teachers need to increase the use of these elements in their classrooms. Further research also needs to explore how flourishing learning environments are related to improved student academic outcomes and greater student engagement and motivation to learn. More theoretical, conceptual, and empirical work is also needed on how flourishing theory improves the quality of education for all students. It may be important, for example, to examine how the concept of flourishing learning environments relates to the emergent research on (a) academic optimism [35], (b) resilience [25, 41], (c) grit and perseverance ([6]: [34]) and (d) other socio-emotional outcomes [7].

Conclusion

While success and failure in school are dependent upon a number of influential determinants, it appears that the classroom learning environment may be a contributing factor [36]. The results of the present study are discouraging in that they paint a bleak picture of students from who are from rural schools. The lack of positive relationships, positive emotion, engagement, meaning, and accomplishment observed in these schools and classrooms suggests that these students are seriously at risk of failure. These schools may need to establish rigorous professional development programs to help their teachers improve the quality of their instruction and relationships with students.