All-but-the-Top: Simple and Effective Postprocessing for Word Representations

Code

• github.com/woctezuma/steam-descriptions
  tf★ 0
• github.com/nlpAThits/WOMBAT
  none★ 0
• github.com/lgalke/vec4ir
  tf★ 0
• github.com/s1998/All-but-the-top
  tf★ 0

Abstract

Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a very simple, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations even stronger. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and text classification) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

Tasks

Benchmark Results

Reproductions