MOROCCO: Model Resource Comparison Framework
Valentin Malykh, Alexander Kukushkin, Ekaterina Artemova, Vladislav Mikhailov, Maria Tikhonova, Tatiana Shavrina
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/sberbank-ai/ru-gptsOfficialIn paperpytorch★ 2,092
- github.com/RussianNLP/MOROCCOOfficialIn paperpytorch★ 24
- github.com/ai-forever/ru-gptspytorch★ 2,091
Abstract
The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environments is often prohibitively expensive. We tackle this problem by evaluating not only the standard quality metrics on downstream tasks but also the memory footprint and inference time. We present MOROCCO, a framework to compare language models compatible with jiant environment which supports over 50 NLU tasks, including SuperGLUE benchmark and multiple probing suites. We demonstrate its applicability for two GLUE-like suites in different languages.