Assignment 2 : Part of Speech Tagging

Due Wednesday, October 3

In this  assignment you will evaluate part of speech tags and attempt to come up with a more accurate POS tagging for a corpus.

Data

The data you will use for this exercise consist of the 9/11 report with part of speech tags assigned by a Brill-like (transformation-based) parser. The tags used are the Penn tagset; the list of all tags is available at http://americannationalcorpus.org/FirstRelease/gatetags.txt.

Code

Several modules and examples of nltk code are available at http://www.cs.vassar.edu/~cs395/code.

Exercises

1. Correcting POS tags

The part of speech tags in the 9/11 report were automatically generated , and some of the tags are incorrect. We are going to try to correct at least some of the errors.

Download report911_tagged from the course data directory. In nltk, create a reader for the 9/11 corpus using the following code (replace [path-to-your-data] as appropriate):

    >>> report911 = nltk.corpus.TaggedCorpusReader("[path-to-your-data]911_tagged",'.*',".txt")

Now you can see the tagged text with

    >>> report911.tagged_words()
        [('WE', 'NNP'), ('HAVE', 'NNP'), ('SOME', 'NNP'), ...]

Now, try to improve the tagging. Here is a set of steps you can follow, but you can use any method you'd like. For example, you might try to use probabilities or other information to improve a tagger's accuracy, use nltk's HMM tagger, use the regExp tagger, etc. The more inventive (and successful), the better your grade will be!
  1. Using the examples from class and information in the nltk tagging tutorial, create a set of taggers including default, unigram, bigram, and trigram.
  2. Write a routine "performance" to run the taggers in sequence and output the accuracy for each tagger as it goes along. For training data, use the 9/11 tagged corpus (despite its not being a true "gold standard"). Run this with the taggers you created.
  3. Write a routine that will take a tagger and a tagged corpus as input, compare tags assigned by the tagger and the tags already in the corpus, and output the 100 most frequent cases where the two taggings differ. Run this using your most accurate tagger. 
  4. Examine the comparison and find tags that were incorrectly assigned in the original 9/11 tagged corpus. Note that this may occur even if the two taggings agree. If necessary, re-run the tag comparison routine from step 3, printing a different subset of the compared tags (e.g., the 101:200 most frequent, etc.) to see more comparisons. 
  5. Correct the most frequently mis-tagged words in the 9/11 tagged corpus. You can do this using the regular expression routines included in nltk.
  6. After correcting the tags, re-run the routines created in steps 2 and 3. Does the output differ?
  7. Repeat steps 2-5 as many times as you can. Try to increase the accuracy of your best tagger as much as possible. 

2. Cross-validation

Write a routine that does cross-validation of the 9/11 tagged corpus. The routine should take as input two sub-corpora, one for training and one for testing, and output the accuracy of the test tagging. Run this routine repeatedly, using 10 different 90% / 10% subsets for training and testing respectively. Average the accuracy for each run and print the final result.

3. Training on another corpus

Train your taggers using the Penn Treebank, which is also tagged with the Penn tagset. For example, this code uses 100000 sentences from the treebank for training and applies it to the 9/11 report:

from nltk.corpus import treebank
>>> unigram_tagger = nltk.UnigramTagger(treebank.tagged_sents() [:100000])
>>> unigram_tagger.tag(report911.words())

Generate the accuracies for your taggers using the performance routine. How do the results differ from the accuracy when the 9/11 report is used to train?

Repeat step 4 from above using the corpus tagged using the Penn Treebank for training. How do the results compare?

Resources

Code

There are several modules in the course code directory that you can use, either directly or as models for your code. They include:
bigramscomputes bigrams for a corpus
buildStoplistbuildStoplist : builds a list of stop words (words not to be included in an analysis)
compareModals Comparison of modals in parts of the Brown corpus (from class slides)
conditional_probcomputes a conditional probability for a POS tag for each word
countTagscomputes the tag most often assigned to words in a tagged corpus
find_namesfinds proper names (sort of) (from class slides)
freqdistcomputes a frequency distribution for words in genesis and prints the token with greatest number of occurrences (from class slides)
init reads the plain text 911 report file
storyGentry to generate a story (from class slides)
wordcountscounts words in the text
wordcounts-textdoes the same as wordcounts but prints each word and frequency

Links

In addition to the nltk tutorial, here are some links to resources that describe the various nltk modules etc.:

nltk api              : an api for the modules in nltk. Check out the index, which I find most useful
nltk source        : the source of nltk modules, just  in case it helps

Word counts for the 911 items:

chapter-1.txt        19260
chapter-10.txt        6068
chapter-11.txt        9414
chapter-12.txt       15623
chapter-13.1.txt     10689
chapter-13.2.txt     14348
chapter-13.3.txt     19349
chapter-13.4.txt     34343
chapter-13.5.txt     37985
chapter-2.txt        10539
chapter-3.txt        33835
chapter-4.txt        17089
chapter-5.txt        13005
chapter-6.txt        19382
chapter-7.txt        17278
chapter-8.txt        11326
chapter-9.txt        19748
preface.txt           1253
TOTAL               310534

What to hand in, and how

To submit your assignment, send it by email to ide@cs.vassar.edu with the subject line CS395 ASSIGNMENT 2.

In addition to the program, submit a summary of your results (a paragraph or so).