Mingxuan Li

Mingxuan Li

MLE | AI4Science | Drug Discovery

Literature Notes - ESM3

1 minute read

Published:

  • ESM3 pushes the boundaries of protein design by integrating sequence, structure, and function into a single unified framework.
  • It scales up to 98 billion parameters and leverages multimodal inputs, enabling generative design and evolutionary insights.
  • Simulates evolutionary processes, producing proteins equivalent to 500 million years of natural evolution within days.

Multimodal Protein Modeling

  • Utilizes three key modalities:
    • Sequence: Captures patterns in amino acid arrangements.
    • Structure: Predicts 3D protein configurations.
    • Function: Designs biologically relevant proteins based on evolutionary goals.
  • Embeds modalities into a shared latent space, allowing flexible and efficient protein generation.

Revolutionary Protein Design

  • Demonstrates its generative power with esmGFP, a novel fluorescent protein sharing only 58% sequence identity with its closest relative.
  • Empowers programmable protein creation by combining structural motifs and novel scaffolds.
  • Aligns generative outputs with user-specified functional goals using fine-tuning mechanisms.

Scalability and Performance

  • Trained on 3.15 billion sequences and 236 million structures, with a total of 771 billion tokens.
  • Largest version contains 216 transformer layers, outperforming smaller models on generative and predictive tasks.
  • Scaling laws of AI emphasize that larger models achieve superior generalization and innovation.

Graph: Impact of ESM3 on Protein Design

Real-World Applications

  • Biotechnology: Synthesis of novel enzymes, biosensors, and therapeutics.
  • Sustainability: Engineering proteins for eco-friendly materials and energy solutions.
  • Basic Science: Mapping evolutionary trajectories and discovering novel protein functionalities.

Closing Remarks

ESM3 redefines synthetic biology by accelerating protein design and evolutionary studies. Its multimodal capabilities and massive scalability open new horizons in biotechnology, offering tools to tackle humanity’s biggest challenges, from medicine to sustainability.

You May Also Enjoy

Literature Notes - Geneformer

3 minute read

Published:

  • Traditional methods for analyzing single-cell transcriptomics rely on cell-type-specific annotations or perturbation-based studies, which can be resource-intensive and dataset-specific.
  • Geneformer, a transformer-based foundation model, learns transcriptional dynamics directly from large-scale unlabelled single-cell transcriptomes, enabling context-aware gene mapping across diverse cellular states and conditions.
  • Pretrained on Genecorpus-30M (29.9 million single-cell transcriptomes), Geneformer provides a robust framework for a wide range of downstream applications, from dosage sensitivity prediction to in silico gene perturbation analysis.

Literature Notes - ESM2

less than 1 minute read

Published:

  • Traditional protein prediction models rely heavily on evolutionary information, often requiring computationally expensive multiple sequence alignment (MSA) inputs.
  • ESM2, powered by large language models (LLMs), learns evolutionary patterns directly from raw protein sequences, eliminating MSA requirements and simplifying the computational pipeline.
  • Achieves a ~60x faster inference compared to prior state-of-the-art methods, facilitating studies on vast metagenomic datasets (e.g., MGnify90).

Literature Notes - Physics-informed machine learning

1 minute read

Published:

The text discusses the advancements in Physics-Informed Neural Networks (PINNs) facilitated by their integration with modern machine learning libraries such as TensorFlow, PyTorch, and specialized frameworks like DeepXDE and SimNet. PINNs excel in incorporating physical laws into machine learning models, making them particularly adept at handling complex, multidimensional scientific problems.

Literature Notes - MolFormer

1 minute read

Published:

The study highlights the capabilities of the Molecule Transformer (M-Transformer), an unsupervised, pretrained molecular language model that excels in predicting molecular properties from SMILES sequences. This model surpasses traditional graph-based models in various benchmarks, efficiently utilizes computational resources by reducing GPU usage by a factor of 60, and accurately captures interatomic relationships. Further exploration into expanding its applicability beyond small organic molecules is recommended.