Tagging the Bard: Evaluating the accuracy of a modern POS tagger on Early Modern English corpora

Rayson, P, Archer, DE, Baron, A, Culpeper, J and Smith, N (2007) Tagging the Bard: Evaluating the accuracy of a modern POS tagger on Early Modern English corpora. In: Corpus Linguistics Conference 2007 (CL2007), 27 July 2007 - 30 July 2007, University of Birmingham, UK.

Preview

Available under License In Copyright. Download (146kB) | Preview

Abstract

In this paper we focus on automatic part-of-speech (POS) annotation, in the context of historical English texts. Techniques that were originally developed for modern English have been applied to numerous other languages over recent years. Despite this diversification, it is still almost invariably the case that the texts being analysed are from contemporary rather than historical sources. Although there is some recognition among historical linguists of the advantages of annotation for the retrieval of lexical, grammatical and other linguistic phenomena, the implementation of such forms of annotation by automatic methods is problematic. For example, changes in grammar over time will lead to a mismatch between probabilistic language models derived from, say, Present-day English and Middle English. Similarly, variability and changes in spelling can cause problems for POS taggers with fixed lexicons and rule-bases. To determine the extent of the problem, and develop possible solutions, we decided to evaluate the accuracy of existing POS taggers, trained on modern English, when they are applied to Early Modern English (EModE) datasets. We focus here on the CLAWS POS tagger, a hybrid rule-based and statistical tool for English, and use as experimental data the Shakespeare First Folio and the Lampeter Corpus. First, using a manually post-edited test set, we evaluate the accuracy of CLAWS when no modifications are made either to its grammatical model or to its lexicon. We then compare this output with CLAWS’ performance when using a pre-processor that detects spelling variants and matches them to modern equivalents. This experiment highlights (i) the extent to which the handling of orthographic variants is sufficient for the tagging accuracy of EModE data to approximate to the levels attained on modern-day text(s), and (ii) in turn, whether revisions to the lexical resources and language models of POS taggers need to be made.