SOTAVerified

Lexical Normalization

Lexical normalization is the task of translating/transforming a non standard text to a standard register.

Example:

new pix comming tomoroe
new pictures coming tomorrow

Datasets usually consists of tweets, since these naturally contain a fair amount of these phenomena.

For lexical normalization, only replacements on the word-level are annotated. Some corpora include annotation for 1-N and N-1 replacements. However, word insertion/deletion and reordering is not part of the task.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–47 of 47 papers

TitleStatusHype
ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical NormalizationCode0
A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Media—0
ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media TextCode1
Automatic Textual Normalization for Hate Speech DetectionCode0
Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media DataCode0
A Character-level Ngram-based MT Approach for Lexical Normalization in Social Media—0
CL-MoNoise: Cross-lingual Lexical Normalization—0
MultiLexNorm: A Shared Task on Multilingual Lexical NormalizationCode0
Sesame Street to Mount Sinai: BERT-constrained character-level Moses models for multilingual lexical normalization—0
Multilingual Sequence Labeling Approach to solve Lexical Normalization—0
A Text Editing Approach to Joint Japanese Word Segmentation, POS Tagging, and Lexical Normalization—0
To What Extent Does Lexical Normalization Help English-as-a-Second Language Learners to Read Noisy English Texts?—0
ÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5Code1
Contrastive String Representation Learning using Synthetic Data—0
Sequence-to-Sequence Lexical Normalization with Multilingual Transformers—0
DaN+: Danish Nested Named Entities and Lexical NormalizationCode0
User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical NormalizationCode0
Lexical Normalization for Code-switched Data and its Effect on POS TaggingCode0
Lexical Normalization for Code-switched Data and its Effect on POS-tagging—0
Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data?—0
Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing—0
A Clustering Framework for Lexical Normalization of Roman UrduCode0
Adapting Deep Learning for Sentiment Classification of Code-Switched Informal Short TextCode0
A Multi-cascaded Deep Model for Bilingual SMS ClassificationCode0
Enhancing BERT for Lexical Normalization—0
An In-depth Analysis of the Effect of Lexical Normalization on the Dependency Parsing of Social Media—0
Normalization of Indonesian-English Code-Mixed Twitter Data—0
Lexical Normalization of User-Generated Medical Text—0
MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization ToolCode0
Adapting Sequence to Sequence models for Text Normalization in Social MediaCode0
Modeling Input Uncertainty in Neural Network Dependency ParsingCode0
Noise-Robust Morphological Disambiguation for Dialectal Arabic—0
A Taxonomy for In-depth Evaluation of Normalization for User Generated Content—0
Handling Normalization Issues for Part-of-Speech Tagging of Online Conversational Text—0
MoNoise: Modeling Noise Using a Modular Normalization SystemCode0
The Denoised Web Treebank: Evaluating Dependency Parsing under Noisy Input Conditions—0
NCSU-SAS-Ning: Candidate Generation and Feature Engineering for Supervised Lexical Normalization—0
IHS\_RD: Lexical Normalization for English Tweets—0
NCSU\_SAS\_SAM: Deep Encoding and Reconstruction for Normalization of Noisy Text—0
Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition—0
USZEGED: Correction Type-sensitive Normalization of English Tweets Using Efficiently Indexed n-gram Statistics—0
Tweet Normalization with Syllables—0
Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization—0
TweetNorm\_es: an annotated corpus for Spanish microtext normalization—0
Towards Shared Datasets for Normalization Research—0
A Large Corpus of Product Reviews in Portuguese: Tackling Out-Of-Vocabulary Words—0
A Log-Linear Model for Unsupervised Text Normalization—0
Show:102550

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1MoNoiseAccuracy87.63—Unverified
2Syllable basedAccuracy86.08—Unverified
3TextNormAccuracy83.94—Unverified
4unLOLAccuracy82.06—Unverified