SOTAVerified

Lexical Normalization

Lexical normalization is the task of translating/transforming a non standard text to a standard register.

Example:

new pix comming tomoroe
new pictures coming tomorrow

Datasets usually consists of tweets, since these naturally contain a fair amount of these phenomena.

For lexical normalization, only replacements on the word-level are annotated. Some corpora include annotation for 1-N and N-1 replacements. However, word insertion/deletion and reordering is not part of the task.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 26–47 of 47 papers

TitleStatusHype
An In-depth Analysis of the Effect of Lexical Normalization on the Dependency Parsing of Social Media—0
Normalization of Indonesian-English Code-Mixed Twitter Data—0
Lexical Normalization of User-Generated Medical Text—0
MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization ToolCode0
Adapting Sequence to Sequence models for Text Normalization in Social MediaCode0
Modeling Input Uncertainty in Neural Network Dependency ParsingCode0
Noise-Robust Morphological Disambiguation for Dialectal Arabic—0
A Taxonomy for In-depth Evaluation of Normalization for User Generated Content—0
Handling Normalization Issues for Part-of-Speech Tagging of Online Conversational Text—0
MoNoise: Modeling Noise Using a Modular Normalization SystemCode0
The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions—0
NCSU-SAS-Ning: Candidate Generation and Feature Engineering for Supervised Lexical Normalization—0
IHS\_RD: Lexical Normalization for English Tweets—0
NCSU\_SAS\_SAM: Deep Encoding and Reconstruction for Normalization of Noisy Text—0
Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition—0
USZEGED: Correction Type-sensitive Normalization of English Tweets Using Efficiently Indexed n-gram Statistics—0
Tweet Normalization with Syllables—0
Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization—0
TweetNorm\_es: an annotated corpus for Spanish microtext normalization—0
Towards Shared Datasets for Normalization Research—0
A Large Corpus of Product Reviews in Portuguese: Tackling Out-Of-Vocabulary Words—0
A Log-Linear Model for Unsupervised Text Normalization—0
Show:102550
← PrevPage 2 of 2Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1MoNoiseAccuracy87.63—Unverified
2Syllable basedAccuracy86.08—Unverified
3TextNormAccuracy83.94—Unverified
4unLOLAccuracy82.06—Unverified