SOTAVerified

Lexical Normalization

Lexical normalization is the task of translating/transforming a non standard text to a standard register.

Example:

new pix comming tomoroe
new pictures coming tomorrow

Datasets usually consists of tweets, since these naturally contain a fair amount of these phenomena.

For lexical normalization, only replacements on the word-level are annotated. Some corpora include annotation for 1-N and N-1 replacements. However, word insertion/deletion and reordering is not part of the task.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–25 of 47 papers

TitleStatusHype
ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media TextCode1
ÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5Code1
Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data?—0
A Character-level Ngram-based MT Approach for Lexical Normalization in Social Media—0
NCSU\_SAS\_SAM: Deep Encoding and Reconstruction for Normalization of Noisy Text—0
Noise-Robust Morphological Disambiguation for Dialectal Arabic—0
Normalization of Indonesian-English Code-Mixed Twitter Data—0
Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing—0
Sequence-to-Sequence Lexical Normalization with Multilingual Transformers—0
Sesame Street to Mount Sinai: BERT-constrained character-level Moses models for multilingual lexical normalization—0
Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition—0
Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization—0
The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions—0
Towards Shared Datasets for Normalization Research—0
To What Extent Does Lexical Normalization Help English-as-a-Second Language Learners to Read Noisy English Texts?—0
Tweet Normalization with Syllables—0
TweetNorm\_es: an annotated corpus for Spanish microtext normalization—0
USZEGED: Correction Type-sensitive Normalization of English Tweets Using Efficiently Indexed n-gram Statistics—0
A Large Corpus of Product Reviews in Portuguese: Tackling Out-Of-Vocabulary Words—0
A Log-Linear Model for Unsupervised Text Normalization—0
An In-depth Analysis of the Effect of Lexical Normalization on the Dependency Parsing of Social Media—0
A Taxonomy for In-depth Evaluation of Normalization for User Generated Content—0
A Text Editing Approach to Joint Japanese Word Segmentation, POS Tagging, and Lexical Normalization—0
A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Media—0
CL-MoNoise: Cross-lingual Lexical Normalization—0
Show:102550
← PrevPage 1 of 2Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1MoNoiseAccuracy87.63—Unverified
2Syllable basedAccuracy86.08—Unverified
3TextNormAccuracy83.94—Unverified
4unLOLAccuracy82.06—Unverified