Discrete Wavelet Transform for Efficient Word Embeddings and Sentence Encoding

Abstract

Wavelets have emerged as a cutting edge technology in a number of fields. Concrete results of their application in image and signal processing suggest that wavelets can be effectively applied to Natural Language Processing (NLP) tasks to capture a variety of linguistic properties. In this paper, we leverage the power of applying Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We evaluate, intrinsically and extrinsically, how wavelets can effectively be used to consolidate important information in a word vector while reducing its dimensionality. We demonstrate the effectiveness of using the DWT based embeddings, together with Discrete Cosine Transform (DCT), to compresses a sentence with a dense amount of information in a fixed size vector based on locally varying word features. We show the efficacy of the proposed paradigm on downstream applications yielding comparable and even superior (in some tasks) results to other state of the art spectral models.

Tasks

Reproductions