assn1

Assignment 1 : Regular Expressions

Due Wednesday, September 19

In this assignment you will write one or several regular expressions, run your expressions on data, and write a short report about the results. There are several suggested tasks below. You only need do one task. But don't be limited by the list below. I encourage you to come up with your own task(s). The exact assignment is up to your own interests and creativity!

Data

Several texts are available at http://www.cs.vassar.edu/~cs395/data. There are two sets, one consisting of plain text, and one consisting of texts from the American National Corpus in UTF-16. You are also free to find data on the web, use your own email, or any other large corpus you find.

Python scripts

The Python scripts demonstrated in class are available at http://www.cs.vassar.edu/~cs395/code. regexs.py and regexcount.py should be used for plain text files; regexs-utf16.py and regexcount-utf16.py should be used for data encoded in UTF-16. You can also develop your own Python scripts, if you prefer and are able.

Here are some examples of what to type at the Unix prompt in order to run the provided modules:

python regexs.py 'that that' [name of file]	prints all lines containing 'that that' in the file
python regexs-utf16.py '\bdis(\w+)\b' *.txt	prints all lines containing text matching the regular expression in all files with extension ".txt" in the current directory
python regexcount.py '\bdis(\w+)\b' *.txt	prints prints all lines containing text matching the regular expression in all files with extension ".txt" in the current directory, with a count of the number of times it appeared at the bottom

Example Tasks

Write a regular expression for monetary amounts (in dollars, francs, marks, lira, etc, using digits and spelled-out numbers, etc). Run this on all the Wall Street Journal text, and report some patterns you notice. Is there a pattern to where digits versus spellings are used? Which monetary unit tends to cooccur with the largest numeric amounts?
Write a regular expression for people names that is triggered by honorifics (Mr, Ms, Senator, etc). Develop and debug it on newspaper text. How many names does it find? Who appears the most frequently? How many strings does it find that are errors of some sort---either not actually people names, or incorrect boundaries? Now run your regular expression on some different genres---say, Jane Austin's books, or Shakespeare's plays. How many errors does it make there?
Write a regular expression for variants of George Bush's name (Bush, G.W. Bush, George Bush, President Bush, etc.). Develop and debug it on the New York Times data. How many strings does it find that are errors of some sort---either not actually a reference to the current president, or incorrect boundaries? Run your regular expression on the Slate data--does it find all references to Bush using your regular expression?
Write several regular expressions that detect certain morphological patterns. Look for patterns of repeated consonants, or vowels. Look for repeated syllables. Make a list of morpheme suffixes, and find the words that contain the longest string of them.
Write a regular expression to detect uses of phrases like "pick...up", where additional words may be interjected between "pick" and "up". What are the word sequences that are interjected? Do the results differ for texts from different genres, or different time periods?
Use the Wall Street Journal file that contains parts-of-speech (wsj15-18.pos), and look for certain patterns in parts-of-speech. What parts of speech are most likely after the phrase "that that"? What three-word part-of-speech sequences are most likely after a verb? What nouns tend to be modified by "strong" versus "powerful"?
Look for uses of "he" and "she," "his" and "her," and look for patterns in the words around them. Are certain verbs or adjectives more likely applied to men than women? What about differences among Jane Austin, Shakespeare, and Dickens?

Or, consider some more involved (and fun!) variants:

Write your own version of ELIZA. Use a series of regular expression substitutions to create a system that engages in a typed conversation. You won't be able to do this using regexs.py and regexcount.py. You'll need to write your own Python program.
Write a program that takes English words as input, and attempts to perform a morphological parse. Rather than implementing a finite-state transducer, you might do this with a cascade of regular expressions and if-then-else's.
Generate a frequency dictionary for words in two sets of texts from different genres and/or time periods. Eliminate "stop" words from the list (e.g., "the", "a", "of", etc.). Are the high frequency words different between the two sets of texts? If so, what is your hypothesis about why this is the case?

What to hand in, and how

To submit your assignment, send it by email to ide@cs.vassar.edu with the subject line CS395 ASSIGNMENT 1.

In addition to writing your regular expression or program, write a short report about your motivations for the task you chose, your experiences in implementing it, your experiences in running it, and your findings. Feel free to suggest additional things you might like to do that would build on what you've done so far. The report should be clear and well-written, but it needn't be long--about one page is fine. Also, no need for fancy formatting--in fact, you can just type this report as the body of your email. Your regular expression or program can also be included in the bodyof the email or included as an attachment.