9 upvotes
New metric shows lexical overlap predicts zero-shot performance
What is The Lexical Imprint of Pre-training Data?
The paper introduces unigram cross-entropy, a simple, model-free score that measures how well the word distribution of a pre-training corpus matches a downstream benchmark. It shows that this lexical alignment alone explains a large chunk of zero-shot accuracy.
Who it is for
Researchers and dataset curators can use UCE to gauge whether a new pre-training set will transfer well before training costly models.