An Assessment of the Portfolio Method of Academic Writing Evaluation of EFL Students - Research

Academic Writing Evaluation of EFL Students

Introduction

Academic writing assessment is a regular feature of class evaluation and standardized testing for both ESL/EFL students and native speakers of English. Arguably, strong academic writing ability is among the most important skills that a student can have or acquire, because effective communication in any academic field is based on the capacity to use high level structures and vocabulary in writing and comprehending academic texts. Biber and Gray (2010), through textual analyses of academic writing, determined that academic writing is not actually more complex than conversational English, as is sometimes assumed. Rather, academic language is more "compressed" than conversational English, and the ideas expressed may be abstract and therefore less explicit in meaning. As a result, academic texts are "... efficient for expert readers, who can quickly extract large amounts of information from relatively short, condensed texts" (Biber and Gray). A higher level of proficiency is needed to decode and utilize these structures.

For all of these aforementioned reasons, proficiency in academic writing is an important requirement and predictor of academic success among all students, and may pose a special challenge to ESL/EFL students. The assessment of academic writing and comprehension skills through standardized testing, notably the TOEFL (Test of English as a Foreign Language) and the IELTS (International English Language Testing System), are used internationally as well as in the UK as an index for proficiency in academic English. However, ample research suggests that the analytical assessment of a single, short piece of writing may not accurately assess students' ability or provide a foundation for improving skills. Song and August (2002), for example, found that the testing and assessment of timed writings may "unfairly penalize" non-native English speakers. Weigle (2002) describes a number of methods of assessing students writing, which include direct and indirect testing and testing of timed writings in addition to the assessment of a portfolio of written work. This latter method, known as the portfolio method of assessment, grades students' written English based upon a range of work written and revised over a period of time, with an opportunity for feedback and dialog with the instructor. This 'portfolio' method of assessment, according to some scholars, offers a more comprehensive view of students' proficiency in written English. Moreover, students may benefit from the opportunity for feedback and continuous dialogue afforded by this method (Sommers 1991, Kibler 2010). However, despite the promise of the portfolio method of assessment to offer more accurate evaluation and an enrichment of skills, studies of the efficacy of the portfolio method in assessing ESL/EFL students' academic writing are sparse and incomplete.

In response to the current lack of research that measures the efficacy of the portfolio method of assessing written English skill among ESL/EFL students, this proposed study will address the following:

Research Questions

- Is the portfolio method of assessment associated with higher scores or inflated scoring for ESL students?

- Does the portfolio method of assessment, in a program which includes regular feedback on writing assignments, enable ESL students to build their academic writing skills more than other methods?

- Does the analytical method of assessment, applied to individual, unrelated writing assignments or tests, compare favourably to the holistic and portfolio methods?

- Does the holistic method of assessment, applied to individual, unrelated writing assignments or tests, compare favourably to the analytical and portfolio methods?

- Which of the three methods leads to the most improved outcome on a standardized writing test?

Literature Review

The portfolio method of assessing students' academic writing is described by several authors as having distinct advantages in comparison to other methods (Weigle 2002). Baker (1993) finds that over 90% of students who were assessed using the portfolio method reacted favourably to it. Song and August (2002), in a qualitative study, indicate that non-native writers tend to score better on a portfolio assessment than during timed writing, as used in standardized testing, which is graded analytically. Moreover, there is some indication that students, particularly non-native English speakers, may benefit significantly from the increased opportunity for interaction, error correction and feedback afforded by the portfolio method, which emphasizes the process of writing and revising rather than focusing exclusively on the finished product. However, as Hamp-Lyons & Condon (2000) point out, there have been relatively few studies of the use of portfolio assessment of academic writing among EFL/ESL students, despite the advantages that the method may have for these students.

Despite the relative absence of studied dealing directly with the use of portfolio assessment among the ESL/ EFL student population, a number of articles do shed light on the needs of the ESL student in the classroom and how academic writing among this student group can be assessed and improved. Flowerdew (2008) finds that these students gain more acceptance based on their expertise in multi-disciplinary subjects. It can be inferred from this study that a method such as the portfolio method, which examines students' ability to contribute meaningfully to academic dialogue rather than judging their abilities based on single, timed writings, may more aptly measure their true ability. Cargill (2006) describes and assesses the effectiveness of a set of educational workshops which took place in China from 2001 - 2003, emphasizing and reinforcing the value of ongoing dialog and collaboration with a team of students and practicing scientists, indicating that multi-level collaboration, feedback and dialog increases student confidence. This endorsement of multi-level collaboration is reiterated by Kibler (2010), who analyzes interaction between students and teachers which "... blur traditional boundaries between 'expert' and 'novice' writers" and allows for L1 use in the classroom setting. Kibler (2010) notes that these interactions are very beneficial to bilingual students, who take on the role of both expert and novice in various situations, based on their own individual areas of proficiency. Although these studies do not deal directly with the portfolio method of academic writing assessment, they introduce and engage with pedagogical concepts, such as collaboration and open dialogue, which are clearly conducive to the portfolio method.

The portfolio method allows for direct feedback and dialogue between students and teachers. Drafts may be shown to the teacher and to peers, and students are encouraged to revise their work, incorporating feedback. Several studies indicate that error correction and feedback has a positive effect on ESL/EFL students' learning, although this point remains somewhat controversial. Bithener et al (2005) indicate that error correction and feedback can improve accuracy, noting, however, that the correct use of new linguistic forms can vary across samples of writing from a single student's body of work. On the other hand, Truscott (2007) argues that error correction has a very small effect, if any. Bithener (2008), in turn, responds to the work of Truscott (1996, 2007) and others and demonstrates that correction has been proven effective and students score better in a post-test after having been corrected. Imprecise assessment of means of correction has yielded inconsistent results in the past, according to Bithener (2008). Guenette (2007) offers another perspective on this debate, citing the influence of inconsistent research methodologies and influence of external variables on the outcome. Storch (2005) finds that pairs of students consistently produce better work than individual students, on the basis of "task fulfillment, grammatical accuracy, and complexity" (p. 153). Clearly, additional research is required with regard to the role of feedback and error correction in ESL/EFL students' learning.

Despite the positive effects of dialogue and collaboration, the suitability of the portfolio method of assessment of ESL/EFL students has been called into question. Hirvela and Sweetland (2005), in a qualitative study assessing and characterizing the reactions of two Asian non-native English speakers to a portfolio task and reflection, notes that these individuals failed to grasp and fully appreciate the intent of the exercise, therefore failing to fully benefit from it. However, as this study is based upon a very small sample group of two students, the value of the portfolio method certainly warrants further consideration.

Proposed Methodology

The participants in the proposed study will be a group of approximately 30 ESL/EFL students enrolled in a post-secondary English writing class. All of the students, by virtue of their having gained admission to the class and program, are identified as having scored 85 or above on the TOEFL exam, indicating an above average to high level of proficiency in academic English.

The students will be divided into three groups and will participate in separate tutorials according to their group allocation. The tutorial teacher will be responsible for setting assignments for the students and will also be involved in the assessment of the assignments. The first group (Group A) will be assigned timed, in-class writing tasks which will be assessed according to a holistic method, with a grade being given based upon the assessor's general impression of the students' proficiency and success in accomplishing the assignment. The second group (Group B) will likewise be assigned timed, in-class writings, and will be assessed analytically, with a clear break-down of criteria to be met for each assignment. The third group (Group C) will be assigned a portfolio of writing, to be completed during the course of the term. The group will engage in constant dialogue and receive feedback from the teacher and peers.

At the end of the term, all students will be assigned a timed writing, which will be analytically assessed according to a set of criteria. The average scores of the students in the three groups will be compared. The scores on the term assignments will likewise be compared in order to ascertain whether there are differences in success rate between the three groups. The grading of the final timed writing assignment will be undertaken by a team of two assessors who are not acquainted with the students and who were not involved in previous assessments. The students' names will not be visible on this final assignment.

Rationale

This methodology represents the first step in a comparative assessment of evaluation methods, their relative accuracy, and their ramifications with regard to ESL/EFL students' academic success and development. The study is not meant to be absolutely conclusive - that is, it will not absolutely indicate whether one method of assessment is superior to others - but will offer insight into the relative effects of the various assessment methods for ESL students. In particular, the method allows the researcher to assess the effects of the portfolio method of assessment as it relates to ESL/EFL students. At the same time, it allows for a comparison of the effects and efficacy of the portfolio method in comparison to other, more common methods of assessment. As significant literature indicates the value of feedback and dialogue in the learning process, especially for non-native English speakers, this research, specifically aimed at establishing the value of the portfolio method for this student group, may help establish a foundation for further research on this method. The use of a relatively large sample group of students (as opposed to Hirvela and Sweetland's qualitative study, which assessed the experience of only two students, for example) is justified in this case because there is a need to eliminate the influence of individual differences between students and to produce a reliable result based on representative sampling.

Bibliography

Baker NW (1993) The effect of portfolio-based instruction on composition students' final examination scores, course grades, and attitudes toward writing. Research in the Teaching of English 27(2): 155-174.

Biber D & Graya B (2010) Challenging stereotypes about academic writing: Complexity, elaboration, explicitness. Journal of English for Academic Purposes 9 (1): 2-20.

Bithener J (2008) Evidence in support of written corrective feedback. Journal of Second Language Writing 17 (2): 102-118.

Bithener J, Young S & Cameron D (2005) The effect of different types of corrective feedback on ESL student writing. Journal of Second Language Writing 14 (3): 191- 205

Cargill M (2006) Developing Chinese scientists' skills for publishing in English: Evaluating collaborating-colleague workshops based on genre analysis. Journal of English for Academic Purposes 5 (3), 207-221.

Flowerdew J (2008) Scholarly writers who use English as an Additional Language: What can Goffman's "Stigma" tell us? Journal of English for Academic Purposes 7 (2): 77 - 86.

Guenette D (2007) Is feedback pedagogically correct?. Research design issues in studies of feedback on writing. Journal of Second Language Writing 16 (1): 40-53.

Hirvela A & Sweetland YL (2005) Two case studies of L2 writers' experiences across learning-directed portfolio contexts. Assessing Writing 10: 192-213.

Kibler A (2010) Writing through two languages: First language expertise in a language minority classroom. Journal of Second Language Writing 19 (3): 121 - 142.

Lee, I (2010) Writing teacher education and teacher learning: Testimonies of four EFL teachers. Journal of Second Language Writing 19 (3): 143-157.

Sommers N (1991) Bringing practice in line with theory: using portfolio grading in the composition classroom. In P.Belanoff & M.Dickson (eds.), Portfolios: Process and Product. Portsmouth: Boynton Cook, pp.153-164.

Song B & August B (2002) Using portfolios to assess the writing of ESL students: a powerful alternative? Journal of Second Language Writing 11: 49-72.

Storch N (2005) Collaborative writing: Product, process, and students' reflections. Journal of Second Language Writing 14 (3): 153-173

Truscott J (2007) The effect of error correction on learners' ability to write accurately. Journal of Second Language Writing 16 (4): 255 - 272.

Weigle SC (2002). Assessing writing. Cambridge language assessment series. Cambridge: Cambridge University Press.

Wette R (2010) Evaluating student learning in a university-level EAP unit on writing using sources. Journal of Second Language Writing 19 (3): 158- 177.