Analysis and Mitigation of Dataset Artifacts in OpenAI GPT-3

Abstract

With the recent release of public beta, we took full advantage of OpenAI's Models-as-a-Service (MaaS) offering of GPT-3 to analyze and mitigate dataset artifacts in a model that has one of the highest number of parameters. Recent studies in dataset artifacts and adversarial attacks suggest that state of the art (SoTA) NLP models are susceptible to spurious correlations in training datasets. We decided to investigate GPT-3 on dataset artifacts taking advantage of its large scale and task-agnostic pre-training. We began by verifying few-shot capabilities of GPT-3 in order to lay the groundwork for analysis. Furthermore, we employed our approach to fine-tuning for the natural language inference (NLI) task. Using SNLI as a baseline, we carried out several experiments with Adversarial NLI (ANLI) to evaluate the performance and robustness of GPT-3. Our findings suggest that using adversarial datasets could mitigate dataset artifacts in GPT-3 at a negligible overall performance cost.

Tasks

Reproductions