We combine a new protein language model (AminoBERT) with an improved version of our end-to-end differentiable machinery (RGN2) to directly generate 3D coordinates. On orphan proteins, RGN2 outperforms all major methods, including #AlphaFold , RoseTTAFold, and trRosetta. (2/4)

On designed proteins RGN2 is close but not yet best accuracy-wise. However, it is orders of magnitude faster; a useful property for exploring new protein sequences. (3/4)

Comments on manuscript are most welcome of course. For future work, we look to combine ideas from AF2 with language models, without sacrificing speed. (4/4)

@MoAlQuraishi @mrprotein24 @NazimBouatta @SurgeBiswas @charochereau @geochurch Is there code/model parameters available?

@ThomasJAGraham @mrprotein24 @NazimBouatta @SurgeBiswas @charochereau @geochurch Not yet, but will soon!

@MoAlQuraishi @mrprotein24 @NazimBouatta @SurgeBiswas @charochereau @geochurch Do you find that weight decay / dropout helps on the pretraining? Most models transformers I see don't use those.

@MoAlQuraishi @mrprotein24 @NazimBouatta @SurgeBiswas @charochereau @geochurch Very interesting. (And very interesting too that this is the second article on this topic this week - the other one by Burkard Rost and col)

@MoAlQuraishi @mrprotein24 @NazimBouatta @SurgeBiswas @charochereau @geochurch Is the code available? I am interested in the representations learned by the model.