Filtering Viral Proteins From Pretraining Data Degrades Protein Language Model Performance
Start here
Researchers examined what happens when viral proteins are removed from the pretraining data of protein language models, using the ESM-2 family. The filtered models showed clear declines on viral-protein tasks: masked language modeling, mutation effect prediction and long-range contact prediction all dropped, with the largest losses on proteins from human-infecting viruses. Performance on non-viral proteins was largely retained, suggesting the damage is concentrated rather than general. The effect also grew with model scale, worsening as models increased from 8M to 3B parameters. The work raises a practical tension for teams building models intended for therapeutic and vaccine applications, where excluding viral data for safety or licensing reasons may cost capability exactly where it is needed. How to mitigate the trade-off is not addressed.
Written by a model from the reports · updated 3 hours ago
Coverage timeline
Follow the coverage to see the different angles.
- arxiv.orgThe Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models
For the ESM-2 family of protein language models, researchers tested how filtering viral proteins from pretraining data affects downstream tasks. Filtered models show clear declines in masked language modeling, mutation effect prediction and long-range contact prediction on viral proteins, with the largest drops on human-infecting virus proteins, while performance on non-viral proteins is largely preserved. The negative effect grows as models scale from 8M to 3B parameters.
Coverage and heat for this story
Reports so far 1·Heat now 5·Peak heat 10 (Oct 19 20:00)·Reports, 24h 0
Two panels, same clock (pointer stays in sync): top is cumulative editorial reports — only rises; last point = All reports; slope = new reports that hour. Bottom is Live-board heatIndex — 7-day participants with a 24-hour half-life, so it can fall when talk cools.