Conceptions of assessment and feedback in secondary school

1. Aims, objectives, and research questions

Assessment and feedback are an integral part of the teaching and learning process. They affect not only what is learnt, but how students learn, their motivation, goals, and sense of self (Cowie, 2003). If students have a negative conception of the role of assessment and they misinterpret the meaning of feedback, this can lead to reduced motivation and low self-esteem.

While a learning environment or task may be designed to facilitate student change on a given variable (e.g., feedback to enhance literacy skills), students’ and teachers’ conceptions will influence the way the task or environment are experienced (Trigwell & Prosser, 1991; Fransson, 1977; Kōnings, Brand-Gruwel, & van Merriēnboer, 2005; Vermetten, Vermunt, & Lodewijks, 2002; Meyer & Muller, 1990). Research findings such as these give weight to the claim that what students believe could be the “single most important construct in educational research” (Pajares, 1992 p. 329).

The potentially powerful influence of conceptions on educational outcomes led this project team to investigate what teachers and students think is the purpose of assessment and feedback and how these conceptions influence achievement. Previous research by the research team prepared the way for this project (Brown, 2004; Dixon, 1999; Irving, 2005; Entwistle & Peterson, 2004). In order to get a diverse range of individuals, four schools across a range of deciles and with a varied ethnic and cultural mix were selected to take part. We chose to focus on conceptions of assessment and feedback because assessment and feedback are crucial for promoting learning in schools.

Another important aim of this project was to help teachers to become researchers. We did this by supporting them to develop classroom-based activities that they could use to become aware of their own and their students’ conceptions of assessment and feedback. We also helped the teacher-researchers to use this information to inform their assessment and feedback practices. This project fits well with the following TLRI strategic priorities and research value themes:

• understanding the teaching and learning process
• exploring future possibilities
• reducing inequalities and addressing diversity
• building capacity
• consolidating and building knowledge
• addressing a gap in our knowledge.

The specific project aims and research questions are given below.

Aims of the project

The aims of the project were to:

1. develop ways of identifying students’ and teachers’ conceptions of assessment and feedback
2. develop models which teachers can use to enhance the assessment and feedback process in secondary schools
3. develop research-based evidence for effective assessment and feedback processes which will raise teaching and learning outcomes in mathematics and English in four schools
4. support teachers to become researchers of their own assessment and feedback practices.

Research questions

1. What are students’ and teachers’ conceptions of assessment and feedback?
2. What effect do conceptions of assessment and feedback have on student learning outcomes?
3. What classroom activities can teachers use to identify students’ conceptions of specific assessment and feedback practices?
4. How easy is it to become a teacher-researcher and what factors contribute to the success of programmes like the TLRI?

Project structure

To address the strategic aims and research questions of our TLRI project, the study was divided into three broad research areas which provide the framework for this report:

• action research: enhancing assessment and feedback processes in the classroom
• the teacher-researcher journeys: becoming a teacher-researcher
• identifying students’ and teachers’ conceptions of assessment, feedback, and learning, which led to the development of questionnaires for this purpose; and the effect of the conceptions on student learning outcomes.

2. Design and methodology

A mixed-methods approach was taken to capture student and teacher conceptions of assessment and feedback, and how these have an effect on student achievement. Both qualitative and quantitative techniques have been employed to provide a picture of these conceptions. Each approach informed the other, and enabled us to map some initial findings about New Zealand student and teacher conceptions of assessment and feedback.

Qualitative methods included focus groups, semi-structured interviews, written project notes, anonymous evaluations, brainstorming, transcripts for all team meetings, and selected resources that the teachers used to collect data from their classes—post boxes, concept diamonds, and feedback sheets. In addition, several quantitative data collection processes were employed—class assessment results and questionnaires.

At the start of the project, a series of focus groups with students (Peterson & Irving, in press) and with teachers (Irving & Peterson, in preparation) from participating schools provided rich data that informed the progress of the project, as well as the development of three student questionnaires. These three student questionnaires—conceptions of assessment (Brown, 2006), conceptions of feedback (Irving & Peterson, 2006) and conceptions of learning (Peterson & Irving, 2006)—were based on earlier work, extended through the CAF project and are now being validated through a national sample of secondary students.

A major focus of the research was developing action research projects for each of the teacherresearchers. Each teacher was supported by one of the university research team as their individual mentor. The mentors helped the teachers to formulate questions to direct their research, devise ways of obtaining data to address their questions, and to analyse the data and explain their findings. The first year was focused on building capacity in action research, developing an understanding of the relevant literature, devising tools that could be used to capture what students think with respect to issues in assessment and feedback, and engaging the teachers in examining data and drawing tentative conclusions from their findings. The second year focused on interventions that the teachers could implement in their own classrooms (or, in one case, throughout a department). Each of these interventions was unique to the teacher and school, providing a panoply of small action research experiments.

Student achievement data (using asTTle for either reading or mathematics) have been collected each year to examine the relationship of the research activities to student achievement. Some of these data have been analysed and linked to the conceptions (Brown & Hirschfeld, 2006) but further work is required to fully investigate the relationship of conceptions of assessment, feedback and learning to student achievement.

The project team met once each term over the two years to discuss and share ideas on the teacher action plans, discuss their progress, and provide support. These meetings rotated around the four schools. To facilitate the work of the teacher-researchers in their schools, a person from each school’s senior management was appointed to liaise between the university and the school. In addition, we reported in writing to the school principals each year, and the final meeting in November 2006 was an opportunity to present the project and findings to the principals, liaison teachers, and other guests.

This project was conducted according to the provisions of the University of Auckland Human Participants Ethics Committee (Reference 2004/456). All participants (teachers and students) were provided with participant information sheets and completed consent forms. In addition, the school principals completed a school consent form on behalf of the school. These forms, the meeting and focus group transcripts, and all of the evaluation sheets and questionnaires are held securely at the University of Auckland and will be destroyed after six years.

Teacher action research

Action research is an umbrella term used to describe a form of applied research that seeks to integrate research and practice with the intention of improving practice (Cardno, 2003; Schwandt, 2001). As a particular approach to undertaking research, it has as its focus the details and problems of professional practice. Proponents of action research have argued that through an iterative and cyclic process of problem identification, action, observation, and reflection, practitioners can be assisted to describe, theorise, and change selected aspects of their practice in a supportive manner (Altrichter, Posch, & Somekh, 1993; Cardno, 2003; Elliot, 1991). During the first year of the project, a key task for the teacher-researchers was to develop and implement a data collection tool or instructional activity that could provide them with greater insight into their students’ conceptions of assessment and feedback. This was seen as way in which teacherresearchers could engage in the iterative and cyclic process described above. This experience of engaging in action research formed the basis for the teacher research projects that would be carried out in the second year of the project.

The teacher-researchers engaged in individual projects that addressed a question of interest to them regarding assessment and feedback. Table 1 provides an outline of these projects, including the tools that were used.

Table 1 Topics, aims, and tools used by teacher-researchers in action research

Topic	Project aim	Tool
Close reading	To find out if students understand close reading terminology and to improve students close reading skills	Anonymous post box/drop box
Accuracy in writing	To find out whether grammar, punctuation and spelling (GPS) skills are inhibitory to student writing and improve students GPS skills	Questionnaire
Types of assessment & feedback	To find out what type of assessment and feedback students most prefer and most dislike	Priority diamond
Format of grade	To find out students preferred type of feedback (grade, percentage, NAME)	Trial of different feedback and followed by an evaluation
Creative writing	To find out the effectiveness of creative writing tracking sheet	Tracking sheet and evaluation
Effectiveness of feedback	How do students react to feedback forms on their work	Feedback form and evaluation
Goal setting to improve performance	To encourage students to set goals, show them a pathway to improve and measure the success	asTTle progress reports

Comments and suggestions

A major achievement of the project was to help the teacher-researchers to realise that research in their classrooms did not have to be on a large scale nor address all aspects of a research problem identified in the literature. Rather, they should seek to satisfy their curiosity about an aspect of their role as a teacher or in their teaching, and find ways of involving the students in answering this question. Starter prompts such as “I have often wondered if …” or “why do my students …” or “when I do this, it seems to result in …” can provide the stimulus for an action research exercise. Furthermore, their research does not have to involve complex experimental design. A focused study in a natural setting can provide important answers to how students perceive the environment they share with their teachers. With guidance, what seem like trivial issues can be explored and investigated to inform and potentially change teaching practices.

At the end of the first year of the project, two of our teacher-researchers told us that they were unable to commit the time to the project. Despite our efforts to retain them in the project, they remained resolute and withdrew. These teachers had demanding teaching loads and simply did not have time to take on anything extra.

In Milestone Report 6, we noted how it was important to take into account the realities that teacher-researchers face in meeting their obligations as classroom teachers and as teacherresearchers. We “lost” two of our team, as they felt that they were unable to do justice to their involvement in the project while preparing for and teaching five classes a day along with the associated marking and reporting. In addition, they felt that they were not maintaining a satisfying and satisfactory balance between work and their family life.

The literature on becoming a teacher-researcher is not silent on this issue—it is well documented. The purported benefits (enhanced practice and professionalism, deepening of subject knowledge, the strengthening of teaching and classroom skills, and the personal challenge and refreshment) need to be counterbalanced by the day-to-day challenges that classrooms present.

A recent report to the General Teaching Council for Scotland (Robson & Borthwick, 2004) noted that up to 10 days of class costs had to be provided to ensure that the teacher-researchers could devote the time necessary to their small-scale research projects, and that even with that provision, the team members needed to develop excellent time management skills to maintain their focus.

In discussing the transition from teacher to researcher, Labaree (2003, p. 18) commented:

The job [of the teacher] is to teach the required curriculum to the assigned students at an appropriate level of effectiveness, and this leaves no time for carrying out research. Under these circumstances, teachers can do research only if they add it on top of their existing work, which would place an unfair burden on them because of the heavy load they already bear, or if they do research at the expense of their teaching duties, which would unfairly deprive their students educationally. Realistically, then, moral and occupational constraints limit the time and intellectual effort that teachers can devote to research.

The resolution of this role conflict (Pressick-Kilborn & Sainsbury, 2002) is a critical element in the success of an action-research based project such as this. To this end, in order to help foster closer team relationships and provide further support to the remaining teacher-researchers, we set up an informal mentoring scheme in the second year of the project. Each teacher-researcher was assigned a researcher to work with and develop their action research plans. The mentor kept in semi-regular contact with the teacher-researcher and the pair met at the beginning of each team meeting to discuss their project and for the mentor to give one-on-one feedback and support. This mentoring arrangement worked well for both the researchers and the teacher-researchers.

The teacher-researcher journeys

Teachers talking to teachers can be a powerful vehicle for effecting changes in practices and beliefs (Pennell & Firestone, 1998; Little, 1982), and the journey of each of the teacherresearchers paints a rich and varied picture of their exploration of assessment and feedback. In order to capture this rich picture, a number of data sources were used. These included:

• the audio-taping of each of the group sessions held with the teachers over the course of the two years of the project. Data captured during these sessions provided insight into the nature and scope of the each individual research project (both the trial conducted in the first year and the projects carried out in the second year); methodological issues and challenges experienced by the teacher-researchers; and ways in which challenges and issues were faced and addressed;
• anonymous evaluations completed by teacher-researchers after each of the sessions, which provided further insight to the points noted above, as well as providing feedback on how sessions were meeting the teacher-researcher needs and how subsequent sessions might be structured;
• semi-structured interviews undertaken by the research team at the end of Year 1 to document the aims, methods and outcomes of each individual teacher’s Year 1 trial project and to capture their views of the research process to date;
• semi-structured interviews conducted at the end of the project by an independent researcher to ascertain teachers’ views of the process and their overall involvement in the project.

The three components of Miles and Huberman’s (1994) framework for qualitative data analysis were used to analyse all qualitative data—data reduction, data display, and drawing and verifying conclusions. During data reduction, the data were edited and summarised; coded and recorded; conceptualised and explained. The data were then displayed in chart form. During these two phases, conclusions were drawn and verified with both authors presenting and defending ideas and supporting or challenging those of the other author.

In 2007, we will seek to publish the stories of each of the teacher-researchers and the technical reports from the studies conducted by the university researchers, perhaps in a manner similar to the PEEL project in Australia (Baird & Mitchell, 1993) or as a series of set articles.

Comments and suggestions

We initially encouraged the teachers to keep a diary of their thoughts and feelings about their journeys as teacher-researchers. Although some teachers indicated that they would do this, this was not easy to sustain. However, we found that taping the meetings (especially the teachers’ progress reports), conducting interviews, and using feedback forms at the end of meetings were effective ways of capturing the teachers’ ideas, concerns, and suggestions as their research progressed.

Development of a questionnaire to identify students’ and teachers’ conceptions of assessment and feedback

This type of research is founded on well-established principles of survey opinion research which assumes that participants’ real opinions are adequately captured by the inventory statements and that their responses genuinely indicate their real conceptions. The research depends on sophisticated exploratory and confirmatory factor analytic techniques and assumes that the mean factor or scale scores and factor intercorrelations validly represent the strength and structure of participants’ conceptions. When factorally-confirmed conceptions are mapped to achievement scores with structural equation modelling, it is possible to infer how attitudes are related to performance, provided the model has robust fit to the data and the dataset is sufficiently large to permit such analysis.

The following methodologies were employed.

Questionnaire administration

In the first year of the project, Brown’s (2004, 2006a) Teachers’ Conceptions of Assessment inventory was used to determine teachers’ conceptions of assessment, with results reported in Brown, 2005. In the first and second years, two forms of the Students’ Conceptions of Assessment inventory were administered to ascertain the structure and strength of the students’ conceptions (Brown & Hirschfeld, 2005; Brown, 2006). These two forms built on earlier versions used to study students’ conceptions of assessment related to mathematics (Brown & Hirschfeld, in press a) and reading (Brown & Hirschfeld, in press b) achievement.

Student focus groups

Brown’s survey questionnaire work on students’ conceptions of assessment is promising, and reasonably robust. Further research, grounded in the student voice, was sought to ascertain students’ thoughts on the purpose of assessment, and ensure that the range of factors and items were valid and sufficient. Similarly, little was known about students’ thoughts on the purpose of feedback and therefore in the first year of the project, five focus groups were conducted to identify students’ conceptions of assessment and feedback (Peterson & Irving, in press). The results of these focus groups were used to develop and extend the range of items and factors covered by the research inventories.

Teacher focus groups

Two focus groups were also held with the teacher-researchers to identify their conceptions of assessment and feedback and to find out what they thought the students thought about assessment and feedback (Irving & Peterson, 2007).

Questionnaire modification and design

The student focus group data was then used to modify Brown’s students’ Conception of Assessment inventory for students and to design a new Conception of Feedback questionnaire. A further questionnaire on students’ conceptions of learning was also developed that combined findings from the student focus groups with previous research by Hattie and Purdie (2005). These questionnaires were administered to all students in the teacher-researchers’ target classes at the beginning of the second year of the project. The questionnaire data were then analysed, poor fitting items were either replaced or altered, and the questionnaire was re-administered at the end of the second year.

Development of a nationally representative survey of student conceptions

The sample sizes for the students’ conceptions inventories have been marginal and further development of the instruments is under way with funding from the University of Auckland Faculty of Education Research Committee and the School of Teaching, Learning and Development Research Committee, to generate a national profile of secondary students’ conceptions of assessment, feedback, and learning. This new questionnaire has been administered to a nationally representative sample of more than 800 secondary students. These data from this spin-off project will be analysed in 2007.

Comments and suggestions

Throughout the process of designing and administering the questionnaires on students’ and teachers’ conceptions of assessment, feedback, and learning, we have kept the teachers informed. In particular, we have discussed in our meetings, and received their feedback on, our interpretation of the focus group data and the questionnaire items. We have also given the teachers anonymous feedback on their own questionnaire data and that of their students. We felt it was important to keep this anonymous in order to encourage honest responses from both the teachers and students.

In order not to overburden the teachers, we administered the student questionnaires, giving the teacher a period of relief from classroom teaching. This was warmly welcomed by the teachers.

3. Project findings

Teacher action research

The teacher action research projects covered a variety of topics (see Table 1). These projects led to considerable learning for the teacher-researchers. There has been learning associated with teachers’ understandings of the nature of assessment and feedback as well as professional learning associated with teachers’ understandings of their students’ conceptions. There have been selfreported changes to their classroom practices and improved learning outcomes for students.

A detailed report of each teacher’s action research project and their specific findings is given in Appendix 1. These reports cover the following questions and were written by the teachers with assistance from their mentors.

• What did I want to know or explore or find out more about? What was it about feedback or assessment that I was interested in?
• Why was I interested in this? What motivated my research?
• Who did I do my research with—what students, how many, what teaching?
• What did I find out?
• What does this tell me and other teachers about the problem I was exploring? OR What can other teachers learn from this?
• How much time did all of this take and from where did I get the time?

The teacher-researcher journeys

Teachers cannot be considered a homogeneous group. In any situation where a group of people come together to make changes to their practice through individual and collective endeavour, it must be recognised that “the trajectory for individual teachers … will differ because starting points will vary, as will beliefs, wishes and efforts of those embarking on such changes” (Black et al., 2003, p. 57). Even with a small sample such as the one for this project, it is clearly evident that teachers embarked on the project with different levels of commitment; different understandings about assessment and feedback; and different views of research and insights into their expected role in the research process. While some were motivated volunteers, others were not. Differences among teachers led some to take an active stance to learning to be a teacherresearcher whilst others were more passive in their approach. Some were more receptive to the ideas of others and showed a greater willingness to take advice. Despite these differences, all remained in the project during its first year, albeit with different outcomes achieved by the end of that year.

This raises the question of what enabled teachers to make the transformation from teacher to teacher-researcher?

First and foremost, the investigations that teachers carried out were grounded in their classrooms and in their discipline, and involved their students. Essentially, each classroom become a powerful context for teacher learning (Borko, 2004), and provided the momentum for teachers to persevere with the task at hand. The development of a tool to gain insight into students’ conceptions of assessment and feedback allowed teachers to re-structure a familiar situation (their classrooms); examine that which had previously been hidden (students’ viewpoints); explore students’ perceptions in some depth (their understandings and beliefs about assessment and feedback), and gain new insight (into these understandings and how these may help student learning). Given that the majority of teachers want the best for the students they teach (Guskey, 2002), gaining insight into students’ conceptions appeared to stimulate teachers’ professional curiosity and generate feelings of excitement (Rudduck, 1985). For some of the teachers, selfreported changes in the nature of the interactions between themselves and their students was an additional impetus to continue. This is not surprising given the work of Guskey (2002), who has argued that teachers are motivated by practices that yield improvement in student learning.

Secondly, the importance of the teacher meetings cannot be underplayed in regard to supporting the teachers to make the transformation from teacher to teacher-researcher. They were the critical forums where professional dialogue between all project members was fostered. During these meetings, the teachers engaged in substantive discussion about both assessment and research. Further, the meetings afforded teachers with the opportunity to voice doubts about any professional problems they might have experienced in regard to either the research process itself or any aspect of assessment that warranted attention. Essentially, they provided the teacherresearchers, to varying degrees, with some appreciation of, and opportunities to learn about, a community’s practices, vocabulary, and knowledge (Lave & Wenger, 1991).

The teachers in this project have just begun their journeys as teacher-researchers. As expected, the complex tasks of learning research skills and carrying out this research in the context of their regular, multi-dimensional roles as teachers has proven challenging and demanding. The more experienced researchers in the project team have scaffolded the teachers’ growing research expertise, both individually and within team meeting settings. The team meetings and one-on-one discussions have provided opportunities for “interaction between prior and new knowledge” (Richardson, 1997, p. 3) in ways that have contributed significantly to the teachers’ understandings of research processes and the implications of their findings. There has been a development of a community of practice (Wenger, 1998) where the search for meaning is a shared endeavour.

Even with support available from experienced researchers, the teachers have found the planning and analysis stages of the research project particularly difficult. As with many beginning researchers faced with the complexities of research (Wolcott, 2001), these teacher-researchers have struggled at times to refine their research questions and to design studies to fit these. At all stages, most of the teacher-researchers have expressed some doubt as to their ability to contribute significantly to the project. However, at the end of the first year the teachers in this project had recognised the fruitfulness of this endeavour and were beginning to realise that they had begun the transformation from teacher to teacher-researcher, albeit a considerable challenge (Handscomb & Macbeth, 2004). The teachers have moved along this pathway at very different rates. Some were slower to start; others were enthusiastically planning and generating data from an early stage. Some addressed the tyranny of time and multiple professional demands more easily than others. Some required considerable support in analysing and interpreting the data; others handled the data with confidence and were quick to see the implications of the findings for their practice. All had realised the power of research to inform their practice.

Overall, the main findings from the analysis of the journey to becoming a teacher-researcher are listed below:

• Teacher-researchers’ initial understandings of the research process were restricted and limited with regard to knowledge of research designs; how to establish a workable research question, and how to generate and analyse data. Also their understandings about research-related issues such as validity, reliability, adequacy of data, and researcher bias were negligible.
• Teacher-researchers needed a lot of support during team meetings and individually from the university team to assist them to work through these issues. In two instances, these were not resolved.
• Without support from the university researchers, it is doubtful that any of the teacher-researchers would have planned and implemented a piece of small scale research.
• Engagement in academic reading related to research, assessment, and feedback, although considered important, was challenging for the teacher-researchers. Nevertheless, several of the teacher-researchers used something serendipitous from one of the provided readings as the basis of their research.

The teachers also perceived multiple benefits from being involved in the study, as listed below:

• Teachers perceived that their understandings about the processes and purposes of assessment and feedback had increased.
• Teachers perceived that they had adopted a greater variety of teaching strategies to elicit student understandings, not only of assessment and feedback, but also in regard to curriculum understandings.
• Teachers perceived their use of formative assessment and feedback to students was stronger than before their participation in the project.
• Teacher felt that they engaged with students differently—in a more in-depth manner.
• Teacher believed that they viewed students differently. Student responses had shown teachers that they took the acts of assessment and feedback far more seriously than teachers had given them credit for.
• Some teachers felt they had increased expectations of their students—they now asked students to become more reflective about assessment results and feedback suggestions, and asked them to take more responsibility for their learning.
• Most teachers felt they were more knowledgeable about their students—they gained greater insight into students’ understandings of assessment and feedback and they had gained insight students’ learning needs.

The questionnaire development

The CAF project led to or extended the development of three questionnaires:

• conceptions of feedback;
• conceptions of learning;
• students’ conceptions of assessment.

These questionnaires have the potential to provide teachers with a simple way of identifying their students’ conceptions of assessment, feedback, and learning, so that any less effective conceptions can be identified.

Our focus on New Zealand secondary students’ conceptions of assessment and feedback was particularly timely, as New Zealand’s ethnic and cultural mix is changing. While the proportion of those who identified themselves as New Zealand European increased by 3 percent over the past decade, there has been a 21 percent increase in Maori, a 39 percent increase in those claiming Pacific Island ethnic identity and 142 percent increase in those identifying as Asian (Statistics New Zealand, 2004). This change in New Zealand’s ethnic and cultural landscape means students are entering our secondary schools with potentially more diverse conceptions of what assessment, feedback, and learning are, and these views are likely to affect the way they engage with learning and instruction.

We outline below the findings from our conceptions questionnaires.

Web editor note: Questionnaires can be accessed in the report appendices

Students’ conceptions of feedback

The five student focus groups sought to tap student beliefs and understandings around three key aspects of feedback (in parallel with assessment)—definition, purpose, and personal impact/response. The students felt that feedback was the link between assessment and learning, and that it did not stand distinct from either. In addition, they were clear about what counted as feedback, and what did not. The nature of the feedback they received also had an effect on their views of the assessment, especially whether the assessment could be considered as irrelevant.

The data from the focus groups was used to develop 55 items for a conceptions of feedback instrument (CoF-I), which was piloted with 256 students. Exploratory and confirmatory factor analysis of data from the pilot study revealed the presence of six different components: “feedback comes from teachers”; “feedback motivates me”; “feedback provides information”; “feedback is about standards”; “qualities of good feedback”; and “help seeking”. The fit of the model was marginally acceptable (χ² = 1139.54 ; df = 587 ; p = .000 ; TLI = .78 ; CFI = .80 ; RMSEA = .067). In addition, Samejima’s Graded Response model was used to select items with optimal measurement characteristics for a second administration with a larger sample.

In a second survey to provide data for a more robust structure, the 47-item CoF-II has been administered to over 800 students from a nationally representative sample of New Zealand schools. Furthermore, the CoF-II responses of a further 250 students together with standardised measures of their learning (asTTle scores in reading or mathematics) will enable us to use structural equation modelling to determine the effect of differing conceptions on student learning outcomes. This second study has been made possible through additional funding from the University of Auckland Faculty of Education.

Students’ conceptions of learning

The CAF project did not set out to identify students’ conceptions of learning. However, analysis of the five secondary student focus groups on their conceptions of assessment and feedback revealed that the students saw learning as an integral part of these processes: learning is assessed, which leads to feedback, which leads to more learning which is ultimately re-assessed. In other words, students’ comments about learning and their understanding of what learning means were inextricably woven into their discussions on assessment and feedback in these focus groups. While some of the students’ conceptions of learning were similar to those identified in the Purdie and Hattie (2002) Conceptions of Learning inventory, there were also important differences. These differences laid the groundwork for our study and the development of a new questionnaire.

A conceptions of learning questionnaire was developed by combining 43 new items extracted from our focus groups with 15 items from Purdie and Hattie’s (2002) Conceptions of Learning inventory. The pilot questionnaire was administered to 236 secondary students in the CAF project. The pilot data were analysed using Multilog in order to remove items with poor discrimination. The data were then analysed with maximum likelihood factor analysis with oblique rotation and confirmed with measurement and structural equation models.

The pilot questionnaire found seven inter-correlated conceptions of learning among 236 secondary school students. The fit of the model to the data was marginal (χ² = 1819.24; df = 810; p = .000; TLI = .72; CFI = .73; RMSEA = .077). As for Purdie and Hattie (2002), the conceptions of learning identified included knowledge gain; using information; understanding; and personal and community growth. New factors were also identified, namely learning as an ongoing continuous process; learning requires effort; and learning as developing knowledge objects. This new model of students’ conceptions of learning is promising. However, a larger sample is needed to provide more robust findings in this area. A subsequent grant from the University of Auckland Faculty of Education has allowed a nationally representative follow-up survey of more than 800 secondary students on the CoL-II. These data are yet to be analysed.

Conceptions of assessment

Both teachers and students in our project have been surveyed using the appropriate conceptions of assessment inventory. Brown (2004) had previously surveyed primary sector teachers, and this project enabled him to extend the process of validation, and draw some comparisons between the primary and secondary sectors. Similarly, previous surveys of high school students (Brown & Hirschfeld, in press a; in press b) were amplified through this project.

Inventory of teachers’ conceptions of assessment

The results from the participating teachers at the start of the project exhibited what may be taken as a sector effect—that is, primary and secondary teachers did not conceive of assessment in identical fashions. The CAF teachers, like their primary teacher colleagues (Brown, 2004), had very similar levels of agreement for “improvement” and “irrelevance” conceptions of assessment. Where they differed was around accountability uses of assessment. The secondary teachers tended to agree that assessment was more about student accountability rather than school or teacher accountability, while primary teachers conceived of assessment as making themselves accountable rather than their students. This result may not be surprising and seems in keeping with the view that, in secondary schools, high stakes assessment increasingly places a substantial portion of the responsibility for learning in the hands of the students. What is surprising is that the difference between the primary and secondary sectors was noticeable only in the area of accountability; this is highly suggestive for future research programmes. It may be that every teacher can agree with “improvement’ and tends to disagree with “irrelevance” conceptions of assessment—what they seem to differ on is the locus of accountability—the teacher or the student.

Students’ conceptions of assessment inventory

Brown’s student version of the Conceptions of Assessment inventory (Brown & Hirschfeld, 2005) was slightly modified based on the findings from the five CAF project focus groups. This revised version was then administered to 236 Year 9 and 10 secondary students in the second year of the CAF project (Brown, 2006). Six inter-correlated conceptions of assessment were found. Their conceptions were classified according to who used assessment (i.e., the students personally, their teacher, or the public) and the effect of assessment (i.e., irrelevant, fun, or making accountable). Students agreed between moderately and mostly with the conceptions that assessment makes students accountable, that students use assessments, and that teachers use assessments. They were in moderate agreement with the conception of public or future use of assessment, slight agreement that assessment was fun, and rejected the conception that assessment was irrelevant. These values were largely consistent with those found in the smaller study in the first year of the project.

Students’ conceptions were largely independent of the types of assessments that students associated with the term “assessment”, though there was a weak association with the interactive assessment type and that assessment is fun. This may suggest that students perceive beneficial formative assessment activities such as self-evaluation, peer-assessment, and assessment conversations as fun activities which they tend to disagree with and which earlier analyses (Brown & Hirschfeld 2005; in press a; in press b) have indicated are associated with lower levels of achievement. In other words, highly interactive, formative assessment practices were seen as fun but reduced student achievement, perhaps because students did not respond actively to, or were unable to access, the information available through such interactions.

Student outcomes data were not available in the second year of the study, but they were available in the first year of the study (Brown & Hirschfeld, 2005). We argued that students’ own responses to the conceptions of assessment inventory indicated that students who were more self-regulating (i.e., used assessment to take responsibility for their own learning) generally achieved more on asTTle educational measures. These results suggested that students who see assessment as a constructive force for personal responsibility gained higher scores, while those who sought to “blame” schools or teachers for assessment results, those who did not take assessment seriously, or who ignored assessment received lower scores.

The revised version of the student conceptions of assessment questionnaire has since been administered to more than 800 secondary students. The data from this nationally representative survey will be analysed in 2007. It is anticipated that this large sample will allow the factor structure of the student conceptions of assessment questionnaire to be confirmed.

4. Limitations of the project

Teacher action research

Each of the teacher-researchers was free to explore a question of their own interest and the individualised nature of these projects therefore limits our ability to generalise across a variety of contexts (e.g., school types, curriculum subjects, levels of teaching experience, assessment practices, year level, and so on). The action research projects would need to be repeated in a range of other schools, contexts, and with more students, before the findings could be generalised.

The teachers involved in the study came from schools representing a wide range of decile rankings (9, 4, 4, and 1). Teachers’ perceptions of the differences in school context and student population demographics may have been a barrier in regard to using some of the ideas presented by other teachers, although on several occasions one teacher would remark to another that they had “tried that idea and it was good”.

This study has also highlighted how the TLRI values of creating a teacher-researcher partnership, building capacity, and having an effect on teacher practice is not an easy process. As noted in Section 3, teacher-researchers’ understanding of the research process is often restricted and they need considerable support to assist them in their development as action researchers. This support needs to come not only from experienced researchers, but also from the school (principal and colleagues) and their students. In particular, teachers need to be given the time to participate in such initiatives and the flexibility to try new initiatives within their classrooms. Without this support, participation and commitment to projects such as this is difficult.

As one teacher said:

As a fulltime teacher and assistant dean with three young children and a husband who is often away … getting to school each morning was a minor miracle in itself. During the day, I teach four periods out of five, spend the other periods sorting out issues that can range from lost shoes to sexual abuse. After any number of meetings after school, I then pick up my own three children and take them to their activities, help with homework, cook dinner and on the story goes. Anyway, the point is that I do not have time to do much else and whatever I choose needs to be meaningful, manageable, and relevant to me in as many spheres of my life as possible.

… I believe that an important part of my success is that I had the support from the English department and school. I am lucky that the school shares my philosophy about teaching and without this support it would have been more difficult for me to continue with the action research.

Teacher-researcher journeys

Our documentation of the process by which the teachers learnt to become researchers is restricted to eight teachers. The teachers came from schools representing a range of deciles. While this variety provides a good range, more teachers representing each decile would have been preferable. The teachers were also drawn to the project for different reasons. Our initial collaboration was with schools’ senior managers. They recruited the teachers into the project. In several instances, teachers joined the project with a limited understanding of the nature of their involvement. Others joined because there were no other teachers available in their department. These factors, along with personal factors, did affect commitment to the project and the subsequent outcomes. Future studies should, where possible, consider controlling these factors to enable their findings to be generalised.

Drawing two teachers from each secondary school but from different subject areas was not necessarily helpful. We believe it would have been more supportive and beneficial if two teachers from the same subject area had been drawn from each school.

The questionnaire development

Web editor note: Questionnaires can be accessed in the report appendices

Focus groups

We conducted five focus groups with Year 9 and 10 students to assess their conceptions of assessment and feedback. In order to see if students in different years have similar conceptions, ideally we would have run focus groups with students in Years 11 to 13.

Some of the students in the focus group knew each other from either their mathematics or their English class. Focus group experts believe that focus groups are best conducted when the participants do not know each other (Krueger & Casey, 2000) so that they are not discouraged from giving their opinions or disagreeing with friends. While the students in our focus groups were not necessarily friends, half of them at least knew each other by name, and this familiarity may restrict the discussion.

Teachers’ conceptions of assessment inventory

The sample in this study was highly unrepresentative of New Zealand secondary teachers. The effect of the qualifications or certification pressure of secondary school on teachers’ conceptions of assessment is not unexpected. A separate study in Queensland using the same instrument found that secondary teachers there were more committed to the student accountability conception of assessment (Brown, Lake, & Matters, in press). However, these results require confirmation from a large and representative sample of New Zealand secondary teachers, which is being planned by Brown in 2007 with support from the University of Auckland New Staff Research Fund.

Students’ conceptions of assessment, conceptions of feedback, and conceptions of learning

These studies have limitations due to the nature of the sample and the instrument.

The sample size in all the CAF studies was small and unrepresentative of New Zealand secondary students, as they are drawn from just eight classes in four urban schools. Sample sizes of less than 250 obtained from the teacher-researchers’ classes are marginal for the type of factor and structural analysis conducted on questionnaires of this length. Furthermore, the small sample size prevented the firm identification of the sub-factors. This concern is being addressed in our nationally representative survey that uses all three modified instruments with more than 800 students.

Good survey design requires about five items per aggregated factor to reduce chance effects. The questionnaires do not have sufficient items for all factors (e.g., “assessment is fun” currently has only two items).

Given the small sample sizes, the findings from the questionnaires are tentative but extremely intriguing and potentially powerful. In addition to more items and more participants, the strength of these findings would be enhanced through multi-method studies (e.g., think-aloud and observations of practice) to determine the relationship of conceptions to practices and outcomes.

5. Building capacity and capability

Teacher action research

Research has shown that students seem to learn more when they know that teachers understand their perceptions (Bishop, Berryman, Tiakiwai, & Richardson, 2003). Certainly the development of tools and teachers’ engagement in a wide range of projects has provided them with deeper insight into students’ perceptions of assessment and feedback.

The collaborative development of instructional activities by the teacher-researchers and the university researchers to identify students’ different conceptions of assessment resulted in improved communication between teachers and students—they served to open up communication pathways. For two of the teacher-researchers, this extended beyond the sphere of this project to include other teachers and researchers—they presented findings from their studies to national conferences and schools’ professional development meetings (Hellyer, 2006; McKay, 2005).

The teacher-researchers’ journeys

As Black et al., (2003) noted in regard to the teachers they worked with who were implementing a range of formative assessment strategies and practices, each followed “different trajectories of change—so that not only their starting points, but also the routes they travelled were different.” (p. 83). This was the case in this project. Teachers’ engagement in small-scale research projects, grounded in their personal interests and needs, enabled them to gain insights into not only their students’ conceptions but also their own conceptions and how these played out in practice. These insights provided the impetus for self-reported changes to practices in a number of instances.

The questionnaire development

It is presumed that the production of robust measures of conceptions of assessment with New Zealand norms, and understanding of how those conceptions relate to practices and outcomes, will have benefit for New Zealand teachers. The Teachers’ Conceptions of Assessment inventory is already available for use from Dr Brown’s website (http://www.arts.auckland.ac.nz/staff/ index.cfm?S=STAFF_gbro008) and it is anticipated that the student instrument will be released in the same way once the nationally representative survey is completed. It is expected that once we understand how conceptions relate to outcomes, innovations in professional development can be experimented with.

Research Team

Researchers, University of Auckland
Dr Gavin Brown
Mrs Helen Dixon
Dr Mavis Haigh
Dr S. Earl Irving (Principal Investigator)
Dr Elizabeth Peterson (Principal Investigator)

Teacher-researchers
James Bashford, Pakuranga College
Catherine Hellyer, Pakuranga College
Amanda McKay, Waitakere College/Henderson High School
Harold Merriman, Papatoetoe High School
Darryn Rae, Papatoetoe High School
Rebecca Sharkey, Waitakere College

Reports from Auckland UniServices Limited should only be used for the purposes for which they were commissioned. If it is proposed to use a report prepared by Auckland UniServices Limited for a different purpose or in a different context from that intended at the time of commissioning the work, then UniServices should be consulted to verify whether the report is being correctly interpreted. In particular it is requested that, where quoted, conclusions given in UniServices reports should be stated in full.

6. Publications from the project

Brown, G. T. L. (2005). An exploration of secondary school teachers’ conceptions of assessment. In B. Kozuh, T. Beran, A. Kozlowska, & P. Bayliss (Eds.), Measurement and Assessment in Educational and Social Research (pp. 39–48). Krakow, Poland: Oficyna Wydawnicza AFM.

Brown, G. T. L. (2005). An exploration of secondary school teachers’ conceptions of assessment: A TLRI study. Conceptions of Assessment and Feedback Project Report No. 1. Auckland: University of Auckland

Brown, G. T. L., & Hirschfeld, G. H. F. (2005). Secondary school students’ conceptions of assessment. Conceptions of Assessment and Feedback Project Report No. 4. Auckland: University of Auckland.

Brown, G. T. L. (2006). Secondary school students’ conceptions of assessment: A Survey of four schools. Conceptions of Assessment and Feedback Project Report No. 5. Auckland: University of Auckland.

Brown, G. T. L., Irving, S. E., Peterson, E. R., & Hirschfeld, G. H. F. (2007, August). Students’ conceptions of assessment: Studies of New Zealand secondary students within the Conceptions of Assessment and Feedback Project. Paper to be presented at the 12th Biennial EARLI Conference, Budapest.

Dixon, H., & Haigh, M. (2006). Becoming and being a teacher-researcher. Paper presented at the Asia Pacific Education Research Association International Conference, Hong Kong, November 2006.

Haigh, M., & Dixon, H. (2006), Becoming and being a teacher-researcher. Conceptions of Assessment and Feedback (CAF) Project Research Report No. 6. Auckland: University of Auckland.

Hellyer, C. (2006). Close reading and asTTle. Paper presented at Assessment for Learning Conference, Palmerston North, October.

Haigh, M., & Dixon, H (2006). Becoming and being a teacher-researcher. Paper presented at the annual conference of the New Zealand Association for Research in Education (NZARE), Rotorua, December.

Irving, S. E., & Peterson, E. R. (2007). Secondary school teachers’ conceptions of assessment and feedback. Manuscript in preparation.

Irving, S. E., Peterson, E. R., & Brown, G. T. L. (2007, August). Student conceptions of feedback: A study of New Zealand secondary students within the Conceptions of Assessment and Feedback project. Paper to be presented at the 12th Biennial EARLI Conference, Budapest.

McKay, A. (2005). “What’s important and what helps me make progress”: Students’ views on Assessment. Paper presented at the annual conference of the New Zealand Association for Research in Education (NZARE), Dunedin, December.

Peterson, E. R., & Barron, K. A. (2005). A sticky situation: A case for Post-it® notes: “I’ve been using stickies a lot (in my classroom) ever since you guys came into my life.” Conceptions of Assessment and Feedback (CAF) Project Research Report No. 2. Auckland: University of Auckland.

Peterson, E. R., & Barron, E. R. (2006). A sticky methodology: Using sticky notes to help facilitate focus groups. In B. Kozuh, R. Kahn, A. Kozlowska, P. Krope (Ed.), Description and explanation in educational and social research (pp. 89–100). Los Angeles, CA: UCLA Press.

Peterson, E. R., & Irving, S. E. (in press). Secondary school students’ conceptions of assessment and feedback. Learning and Instruction.

Peterson, E. R., & Irving, S. E. (2006). Secondary school students’ conceptions of assessment and Feedback. Conceptions of Assessment and Feedback (CAF) Project Research Report No. 3. Auckland: University of Auckland.

Peterson, E. R., Irving, S. E., & Brown, G. T. L. (2007, August). Students’ conceptions of learning: Studies of New Zealand secondary students within the Conceptions of Assessment and Feedback Project. Paper to be presented at the 12th Biennial EARLI Conference, Budapest.

References

Altrichter, H., Posch, P., & Somekh, B. (2005). Teachers investigate their work: An introduction to the methods of action research. London: Routledge.

Baird, J. R., & Mitchell, I. J. (1993). Improving the quality of teaching and learning: An Australian case study: The PEEL Project. Melbourne, Vic: Monash University Printing Services.

Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2003). Assessment for learning: Putting it into practice. Maidenhead, UK: Open University Press.

Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33 (8), 3–15.

Brown, G. T. L. (2006a). Teachers’ conceptions of assessment: Validation of an abridged instrument. Psychological Reports, 99, 166–170.

Brown, G. T. L. (2004). Teachers’ conceptions of assessment: Implications for policy and professional development. Assessment in Education: Policy, Principles and Practice, 11 (3), 301–318.

Brown, G. T. L., & Hirschfeld, G. H. F. (in press a). Students’ conceptions of assessment and mathematics: Self-regulation raises achievement. Australian Journal of Educational and Developmental Psychology.

Brown, G. T. L., & Hirschfeld, G. H. F. (in press b). Students’ conceptions of assessment: Links to outcomes. Assessment in Education: Principles, Policy and Practice.

Brown, G. T. L., Lake, R., Matters, G. (in press). Queensland Teachers’ Conceptions of Assessment. Educational Assessment.

Brown, G. T. L., & Hirschfeld, G. H. F. (2006). Students’ conceptions of assessment and educational assessment outcomes: Understanding testing from the students’ perspective. Poster presented at the 5th annual conference of the International Test Commission (ITC), Brussels, Belgium, July.

Cardno, C. (2003). Action research: A developmental approach. Wellington: New Zealand Council for Educational Research

Dixon, H. (1999). The effect of policy on practice: An analysis of teachers’ perceptions of school-based assessment practice. Unpublished masters thesis, Massey University, Albany, New Zealand.

Elliot, J. (1991). Action research for educational change. Philadelphia: Open University Press.

Entwistle, N. J., & E. R. Peterson. (2004). Conceptions of learning and knowledge in higher education: Relationships with study behaviour and influences of learning environments. International Journal of Educational Research, 41 (6), 407–428.

Fransson, A. (1977). On qualitative differences in learning. IV – Effects of motivation and test anxiety on process and outcome. British Journal of Educational Psychology, 47, 244–257.

Guskey, T. (2002). Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8 (3/4), 381–391.

Handscomb, G., & Macbeth, J. (2004). Professional development through teacher inquiry. Professional Development Today, 7 (2), 6–12.

Hattie, J., & Purdie, N. (2005). Assessing students’ conceptions of learning. Australian Journal of Educational & Developmental Psychology, 2, 17–32.

Irving, S. E. (2005). The development and validation of a student evaluation instrument to identify highly accomplished mathematics teachers. Unpublished doctoral dissertation in education, University of Auckland.

Kōnings, K. D., Brand-Gruwel, S., & van Merriënboer, J. J. G. (2005). Towards more powerful learning environments through combining the perspectives of designers, teachers, and students. British Journal of Educational Psychology, 75, 645–660.

Krueger, R., & Casey, M. (2000). Focus groups: A practical guide for applied research (3rd ed.). Newbury Park, CA: Sage.

Labaree, D. F. (2003). The peculiar problems of preparing educational researchers. Educational Researcher 32 (4), 13–22.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.

Little, J. W. (1982). Norms of collegiality and experimentation: Workplace conditions for school success. American Educational Research Journal, 19, 325–340.

Meyer, J. H. F., & Muller, M. W. (1990). Evaluating the quality of student learning: An unfolding analysis of the association between perceptions of learning context and approaches to studying at the individual level. Studies in Higher Education, 15, 131–155.

Miles, M., & Huberman, A. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.

Pajares, F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of Educational Research, 62, 307–332.

Pennell, J. R., & Firestone, W. A. (1998). Teacher-to-teacher professional development through state-sponsored networks. Phi Delta Kappan, 79 (5), 354–357.

Pressick-Kilborn, K., & Sainsbury, E. (2002). Research in your own backyard: Recognising assumptions and redefining identities. Paper presented at the annual conference of the Australian Association for Research in Education, Brisbane, December.

Richardson, V. (1997). Constructivist teaching and teacher education: Theory and practice. In V. Richardson (Ed.), Constructivist teacher education: Building new understandings (pp. 3–14). London: Falmer Press.

Robson, D., & Borthwick, G. (2004). Experiences of the GTCS teacher researcher programme. Paper presented at the British Educational Research Association, Manchester, September.

Rudduck, J. (1985). Teacher research and research-based teacher education. Journal of Education for Teaching, 11 (3), 281–289.

Schwandt, T. A. (2001). Dictionary of qualitative inquiry. Thousand Oaks, CA: Sage Publications

Trigwell, K., & Prosser, M. (1991). Improving the quality of student learning: The influence of learning context and student approaches to learning on learning outcomes. Higher Education, 22, 251–266.

Vermetten, Y. J., Vermunt, J. D., & Lodewijks, H. G. (2002). Powerful learning environments? How university students differ in their response to instructional measures. Learning and Instruction, 12, 263–284.

Wolcott, H. (2001). Writing up qualitative research (2nd ed.). California: Sage.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge University Press.

The appendices for this online version of the report have been removed. However, you can access them here

Acknowledgements

The Conceptions of Assessment and Feedback (CAF) research team would like to thank the Teaching and Learning Research Initiative (TLRI) for their funding and support of this project and the New Zealand Council for Educational Research which administers the TLRI. We have had a great deal of encouragement from both Christina Smits and Rosemary Hipkins. We would also like to thank NZCER/TLRI for the additional funding we received in support of two of our teacher-researchers who presented their CAF work at conferences.

The financial and administrative support of UniServices has been invaluable. In particular, we would like to thank Sandie Gusscott, Deborah Mabee and Louise Shambrook for their patience in answering and sorting out all our niggly questions about budgets and administration.

Lastly, we thank and acknowledge the support of the principals, school liaison people, teachers and students that took part in this project without whom this project would not have been possible. The teacher-researchers have given generously of their time, enthusiasm and persistence – thanks for seeing the project through. We have greatly enjoyed working with you.

This project has truly been a team effort and one we believe that we should all be proud of.