CS395: Assignment 4
***
Due Wednesday, November
14 ***
In this assignment
you will experiment with different features and
learning
algorithms for classifying newsgroup articles into different topical
categories.
You will use the Weka
toolkit for running experiments. Weka is a
powerful tool with a huge number of features and capabilities. These
include tools to help you analyze which items were incorrectly labeled,
and
which features seem particularly important. You can download Weka
from http://www.cs.waikato.ac.nz/ml/weka/
or use it on the Computer
Science lab machines (type weka-chooser at the prompt).
You will use Python for
processing the data files to select
features of interest, writing these features out to files that can be
input to Weka. I am supplying you with some code (weka.py)
that will get you started.
Note that Weka also
works from within the GATE
program, which is also on our machines (type gate to
invoke). The nice part of this is that you can annotate your text in
GATE and send it directly into Weka. This is useful when you use GATE
for annotating, since massaging the data to serve as input to Weka is
then unnecessary. There is a fairly steep learning curve for using GATE
with Weka though, so it might be worth it only if you intend to use
GATE in your final project.
Evaluation
Procedure
We want to
realistically simulate the testing process.
Therefore, start by training and testing on two different groups of
newgroups.
You are to first do training only on the training
set. Experiment as much as you want with the classifiers and feature
selectors.
Before running it on the test set you must freeze
your code and decide which Weka parameters you are going to use in
advance. Write these down and take screen shots of the output on the
training
set (using cross validation or dividing the training set into training
and development sets).
Once everything is set,
run your best model and parameters on
the test
set. Do this only once and save the
results of
testing (see Using
Weka for a description of how to do this) to hand in.
Datasets
Use data from the subset
of the 20 newsgroups collection that I've put in a zip
file (8.3M compressed). It is also available in a
gzip'd
tar
file (5.2M compressed).
- Diverse set. First, everyone should
train and test comparing these two newsgroups, which are
intended to be quite different from one another and hence easier to get
good
scores on:
- rec.motorcycles, sci.space
- Homogeneous set. Second, choose from
one of the following two sets of newsgroups:
- rec.autos, rec.motorcycles, rec.sport.baseball,
rec.sport.hockey
- sci.crypt, sci.electronics, sci.med.original,
sci.space
Getting
Started
It's always a good idea
to take a look at the data before you get
started, so
look at some of the plain text of the different groups to get a feeling
for how
long they are, what format they take, what kind of language they use,
and so on.
The code in weka.py
creates features that consist of all the
words in the text minus the stopwords, which you can (optionally)
remove. These features are weighted by their TF values. You should
consider these features and
their weights as a baseline to compare different
features/weighting approaches against. To compare the two approaches,
you can run the same learning algorithms using different feature sets
in
order to compare them. You can also use different learning algorithms
for the different feature sets, as some learning algorithms may do
better with reduced features or weighted features than others. You
can also experiment with the feature selection methods in Weka rather
than doing it yourself in the code.
You may want to
experiment with changing the parameters of some of the
Weka learning programs on the training set.
You can also use their feature analysis programs to see which features
are doing
well. This may give you ideas about which features will yield
better
results.
For
the adventurous
Besides just using
words, you can create more complex features. One possibility is to use nltk to extract any
of a variety of
linguistic features (e.g., root form (lemma), part of speech, noun
and/or verb chunks) and utilize this information to create features.
For example, you could consider only heads (head noun or head verb) of
chunks;
create complex features from patterns based on part of speech (e.g., a
given word or POS only when there is a certain word or part
of speech within a window of +-n words).
Another possibility is to use words with specific semantic properties
(I have some word lists you could use for things such as
positive/negative opinion, subjectivity, overstatement, hostility...and
many more)
and use these as features, or in combination with other
features. Careful, though: if you are attempting to classify
by topic, you should look at the data carefully to determine whether or
not this will contribute to determining ttopic, and not something else.
Ideas
for Feature Weighting
- Give more weight to features from the subject line.
- Use tf.idf weighting on words.
- Some of the articles make use of technical terminology,
which may include
noun compounds, numbers (or alpha-numeric terms) and/or abbreviations.
It may
be a good idea to give these features more weight.
Assignment Submission
You must try at least 2
different additional types of features and at
least 2
different classifiers in your experiments. You must experiment on the
diverse
and on one homogenous document set. Indicate which homogenous set you
used -- rec or sci.
Turn in a description of
which features/feature weighting/classifiers
you tried
and the accuracy scores for how well the best ones worked. Contrast how
things
worked with the diverse set vs. the homogenous set of newsgroups.
Describe the results of your experiments:
Which features helped/hurt -- why?
Did you use Weka for feature selection? Did it help?
Which classifiers helped/hurt -- which setting variations did you try?
Did you try binary or multi-way classification?
How were the results for the two different collections different or
similar?
How do you think the results could be further improved?
Turn in the results file(s) that shows the results of
running your best classifier(s) on the training set just before
freezing your approach. Then run your settings on the test set and save the results of that as well. Turn
in results for training and testing on both the diverse set and
the homogenous set of your choosing.
Your goal is to produce the best possible classifier. There will be fabulous prizes for the winner!
Attach everything to
an email and send it to ide [at] cs.vassar.edu.
Resources