Sentiment, measured

Double negatives beat both models: “wasn’t terrible” and “not a bad watch” read as negative, because “terrible” and “bad” are among the strongest negative words in training and the not_ feature has too few examples to flip them. Model A counts every token, so nine exclamation marks in “no longer working!!!!!!!!!” are nine votes for positive; model B only notes that “!” is present and gets it right. Model A also learned that “Hitchcock” signals a positive review.

1. Both models wrong, both confident · IMDb

   The soundtrack wasn't terrible, either.

   Gold

   positive

   P(positive), A

   0.004 wrong

   P(positive), B

   0.002 wrong

2. Both models wrong, both confident · IMDb

   Predictable, but not a bad watch.

   Gold

   positive

   P(positive), A

   0.002 wrong

   P(positive), B

   0.007 wrong

3. Negation: A wrong, B right · Amazon

   I purchased this and within 2 days it was no longer working!!!!!!!!!

   Gold

   negative

   P(positive), A

   0.994 wrong

   P(positive), B

   0.286 right

4. Negation: A wrong, B right · Amazon

   Not a good bargain.

   Gold

   negative

   P(positive), A

   0.814 wrong

   P(positive), B

   0.304 right

5. Negation: B still wrong · Yelp

   If you're not familiar, check it out.

   Gold

   positive

   P(positive), A

   0.048 wrong

   P(positive), B

   0.016 wrong

6. Negation: B still wrong · Yelp

   The goat taco didn't skimp on the meat and wow what FLAVOR!

   Gold

   positive

   P(positive), A

   0.013 wrong

   P(positive), B

   0.021 wrong

7. Only B wrong · Yelp

   Ryan's Bar is definitely one Edinburgh establishment I won't be revisiting.

   Gold

   negative

   P(positive), A

   0.470 right

   P(positive), B

   0.918 wrong

8. Only A wrong · IMDb

   There are the usual Hitchcock logic flaws.

   Gold

   negative

   P(positive), A

   0.913 wrong

   P(positive), B

   0.417 right

• Data. 3,000 labelled sentences, 1,000 each from Amazon, Yelp and IMDb, half positive. 21 exact repeats (ignoring case and spacing) were dropped first, leaving 2,979, so no sentence can appear in both training and test.
• Split. Seeded (331) and stratified by source and label: 2,084 train, 299 validation, 596 test.
• Features. Lowercase regex tokens; contractions split so “don’t” keeps its “n’t”. Model A counts unigrams. Model B uses unigram and bigram presence plus a not_ copy of each word between a negation and the next punctuation mark.
• Training. A: multinomial Naive Bayes, Laplace smoothing (alpha 1), not tuned. B: L2 logistic regression, full-batch gradient descent with Nesterov momentum, 1,000 steps; the L2 strength (3 × 10⁻⁵) was picked from 7 values by validation log loss. The test set played no part in fitting or tuning.
• Uncertainty. 1,000 bootstrap resamples of the test set; intervals are the 2.5th to 97.5th percentiles. F1 for A: 0.757 to 0.828; B: 0.764 to 0.835; B minus A: −0.021 to +0.038 (B ahead in 70% of resamples).

• Small sample. Each source contributes about 200 test sentences, so a slice accuracy moves by half a point with every sentence. Read slice gaps as hints, not findings.
• One split, one seed. The bootstrap covers test-set sampling only. A different split or seed would also move the numbers.
• Easier than real use. The dataset’s authors kept sentences with a clear polarity and no neutral ones. Chat messages, sarcasm and mixed feelings are harder, and these models have never seen them.
• Domain shift is not tested. Training and test sentences come from the same three sites. Training on two sources and testing on the third would be the next experiment.
• Examples are quoted, not written. The failure examples are verbatim test sentences from the public dataset, reviews posted on Amazon, Yelp and IMDb; any names in them come from that text.
• Validation reads higher than test. At the L2 strength it picked, model B scored F1 0.852 on the 299 validation sentences and 0.803 on the test set. That setting was chosen on the validation set, so its score there is optimistic; the test number is the one to quote.
• Scores are not calibrated. Neither model’s P(positive) was checked against observed frequencies. Naive Bayes pushes more of its scores to the extremes, so the same slider move changes fewer of its calls.