Getting Inside “Measuring Second Language Pragmatic Competence”

This month we published Measuring Second Language Pragmatic Competence by Rod Ellis, Carsten Roever, Natsuko Shintani and Yan Zhu. In this post Rod introduces us to the book.

There is a background to the writing of any book which often goes unstated. In my blog post I’d like to take you inside the book by telling you a bit about why and how it got to be written. I will explain who the authors are, what motivated the writing of the book, what research it was based on and what I see as the uses of the book.

How did I come to work with my co-authors on this book?

The real starting point was a research study I completed more than fifteen years ago investigating the measurement of second language (L2) grammatical knowledge. This research drew the distinction between implicit and explicit knowledge of an L2 and involved the development of a set of tests designed to provide separate measures of these two types of grammatical knowledge. The results of this research project were reported in a book called  Implicit and Explicit Knowledge in Second Language Learning, Testing and Teaching published by Multilingual Matters in 2009. My idea was to extend this earlier research by investigating whether it was also possible to develop tests that would provide separate measures of implicit and explicit L2 pragmatic knowledge (i.e. the knowledge we use to communicate smoothly and appropriately).

I decided therefore to put in an application for a Discovery Research Grant from the Australian Research Council, drawing on my experience of the earlier project. My own expertise lay in second language acquisition research but I recognized that I also needed expertise in L2 pragmatics. I approached Carsten Roever, who had published widely in L2 pragmatics, to see if he would be interested in joining in the project. Carsten was an ideal co-researcher because he also had experience in language assessment. Together we developed a research proposal, which was successful. We now had the funds we needed for the project.

Our original aim was to investigate different populations of L2 learners – both EFL and ESL. After designing a battery of tests, we started by collecting data from L2 university learners in China and Japan. To help in this we were joined by two other researchers – Yan Zhu, a professor at Fudan University in Shanghai, and Natsuko Shintani, a professor at Kansai University in Osaka. Unfortunately restrictions arising from the Covid epidemic in Australia made it impossible to continue by collecting data from ESL learners.

The research team, then, comprised myself, Carsten, Yan Zhu and Natsuko. This team has remained with the project throughout and many of the chapters in the book involved all four authors.

Why was it important for us to take a psycholinguistic perspective?

By ‘psycholinguistic perspective’ I mean an approach that is founded on the psychological underpinning of linguistic rules and processes. In the case of language testing, this involves considering the mental processes involved when completing tests. While there are a number of different psycholinguistic perspectives, the one we drew on was based on the implicit/ explicit distinction. From this theoretical perspective, when learners complete a test they can draw on their implicit knowledge (i.e. knowledge that they have no conscious awareness of and can be accessed automatically) or their explicit knowledge (i.e. knowledge they are conscious of and that is only accessible slowly after reflection). The implicit/explicit distinction is a fundamental distinction in current thinking theorizing about an L2 but, surprisingly, it has not figured in research on the testing of L2 pragmatic competence. In fact, many of the popular ways of testing L2 pragmatics really only measure explicit processing and thus tell us nothing about learners’ ability to process their pragmatic knowledge implicitly even though, arguably, it is implicit abilities that are essential in everyday communicative situations.

All research needs to focus on how it can build on and extend existing research. By adopting a psycholinguistic perspective, we hoped to fill a gap in current research on the measurement of pragmatic competence.

How did we develop the tests that featured in the book?

The development of any test requires making decisions about what to test and how to test it. Our starting point was to decide what aspects of pragmatics to measure. A study of the L2 pragmatics literature led us to identify three aspects: metapragmatic awareness, implicature, and conversational structure. In deciding how to assess these aspects, we tried to design tests that would tap either learners’ implicit pragmatic processing abilities (i.e. responses made rapidly and without time for reflection) or explicit processing abilities (i.e. responses made without time pressure and that favoured reflection).  We ended up with two tests involving explicit processing, two tests more likely to favour implicit processing, and a test of the ability to comprehend irony where we were unsure of the kind of processing it would most likely elicit. These tests also differed in terms of whether they involve comprehension/perception or production.

The initial tests were designed by myself and Carsten Roever, piloted on a small group of ESL learners, revised, and then administered with the help of Yan Zhu and Natsuko Shintani in China and Japan respectively. One of our aims was to carry out a careful evaluation of the tests and there are chapters in the book that report the results of the evaluation of each test. Not unexpectedly, we did not find that all the tests worked as well as we had hoped! Subsequently, we redesigned some of the tests and used them in a final study where we investigated the effects of instruction on the development of the two types of knowledge.

What do we hope readers will take away from the book?

We envisage that the main readers of our book will be second language acquisition researchers interested in pragmatics. An abiding issue in SLA research is how learning is measured and we hoped that the tests we developed would help in designing studies that investigate how different conditions of learning impact on the kind of the learning that results. Two chapters in our book illustrate how the tests can be used in this way. In one chapter, we report a comparative study of L2 learners who completed a one-year study abroad and learners who received language instruction while staying in their home country. In another study, we used the tests to investigate the effects of different kinds of instruction on the acquisition of implicit and explicit L2 pragmatic knowledge.

We did not write the book for teachers. But the perspective that informs the book is one that teachers need to consider. Any time teachers set their students a test they need to consider what kind of knowledge the test is measuring.

What are we working on next?

The tests provide the tools we need to investigate the pragmatic competence of L2 learners and how learning conditions shape the kind of learning that take place. We are already embarking on two other studies investigating this. In one study (involving myself and Scott Aubrey), we are comparing the pragmatic competence of students in Hong Kong and Shanghai universities to see if there are differences in these groups of learners. In the other study with Natsuko Shintani, we are investigating how the interactive experiences of Japanese students on a study abroad shape the kind of pragmatic abilities they develop.

Rod Ellis

For more information about this book please see our website.

If you found this interesting you might also like Assessing Speaking in Context edited by M. Rafael Salaberry and Alfred Rue Burch.