The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models

For the ESM-2 family of protein language models, researchers tested how filtering viral proteins from pretraining data affects downstream tasks. Filtered models show clear declines in masked language modeling, mutation effect prediction and long-range contact prediction on viral proteins, with the largest drops on human-infecting virus proteins, while performance on non-viral proteins is largely preserved. The negative effect grows as models scale from 8M to 3B parameters.