A Structured Self-attentive Sentence Embedding
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos santos, Mo Yu, Bing Xiang, Bo-Wen Zhou, Yoshua Bengio
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/facebookresearch/pytextpytorch★ 6,305
- github.com/prakashpandey9/Text-Classification-Pytorchpytorch★ 818
- github.com/hantek/SelfAttentiveSentEmbedpytorch★ 53
- github.com/SeoroMin/transformer_pytorch_ver2pytorch★ 2
- github.com/text-machine-lab/transformerpypytorch★ 2
- github.com/PrashantRanjan09/Structured-Self-Attentive-Sentence-Embeddingpytorch★ 0
- github.com/TieDanCuihua/A-STRUCTURED-SELF-ATTENTIVE-SENTENCE-EMBEDDINGtf★ 0
- github.com/ShuvenduBikash/transformer_spelling_correctorpytorch★ 0
- github.com/leejieun51/transformerpytorch★ 0
- github.com/kaushalshetty/Structured-Self-Attentionpytorch★ 0
Abstract
This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.