-
- CodonTransformer utilizes the BigBird Transformer architecture, an advanced neural network model capable of efficiently processing long sequences. A novel sequence representation strategy, Shared Token Representation and Encoding with Aligned Multi-masking (STREAM), combines organism encoding with tokenized amino acid-codon pairs to achieve context-awareness.
+
+
+
+
+
Model
+
+
+ CodonTransformer addresses the challenge of codon optimization by translating protein sequences into optimized
+ codon sequences
+ using the encoder-only BigBird Transformer architecture. We frame this task as a Masked Language Modeling
+ (MLM) problem,
+ where the model predicts codons by unmasking tokens from [aminoacid_UNK] to [aminoacid_codon].
+ Our innovative STREAM training strategy allows the model to learn codon usage patterns by unmasking multiple
+ mask tokens
+ while organism specific embeddings are added to the sequence to contextualize predictions.
+
+
+ The training process involves two stages: pretraining on over one million DNA-protein pairs from 164 diverse
+ organisms
+ to capture universal codon usage patterns, followed by fine-tuning on a curated subset of highly optimized
+ genes
+ specific to target organisms. This dual training strategy enables CodonTransformer to generate DNA sequences
+ with
+ natural-like codon distributions tailored to each host, effectively optimizing gene expression across multiple
+ species.
+
+
- This innovative approach allows the model to capture both global and local codon usage patterns, enabling it to generate DNA sequences that are finely tuned to the host organism's preferences.
-
-
Key Features
-
- - Encoder-Only: Bidirectional design allows uniform optimization across the entire sequence.
- - STREAM Encoding: Integrates organism, amino acid, and codon information into shared tokens.
- - Masked Language Modeling: Learning complex patterns by predicting masked codons.
-
-
-
CodonTransformer Outperforms Existing Tools
-
-
- Through extensive testing and benchmarking, CodonTransformer demonstrates superior performance in generating natural-like codon distributions and minimizing negative cis-regulatory elements compared to existing codon optimization tools.
-
+
+
+
+
+
CodonTransformer Outperforms Existing Tools
+
+
+ CodonTransformer demonstrates superior performance in generating
+ natural-like codon distributions and minimizing negative cis-regulatory elements compared to existing codon
+ optimization tools.
Here are the benchmarking and evaluation results of CodonTransformer:
+
+
-
+
+
+
+
+
Learning Codon Patterns Across Organisms
+
+
+ CodonTransformer effectively learned codon usage patterns across multiple species, as shown by high codon
+ similarity indices (CSI) when generating DNA sequences for various organisms. The model adapts to the specific
+ codon preferences of each host, ensuring optimal expression.
+
+
+
+
+
+
+
+
+
Fig. 2: CodonTransformer learned codon patterns across organisms.
+
Codon usage index (CSI) for all and the top 10% CSI original genes and generated DNA sequences for all
+ original proteins by CodonTransformer (base and fine-tuned models) for 9 out of 15 genomes used for
+ fine-tuning in this study. See Supplementary Figs. 2-16 for all 15 genomes and additional metrics of GC
+ content codon and distribution frequency (CDF). Source data for Fig. 2 and Supplementary Figs. 2-16 is
+ available at https://zenodo.org/records/13262517.
-
-
-
Learning Codon Patterns Across Organisms
-
-
- CodonTransformer effectively learned codon usage patterns across multiple species, as shown by high codon similarity indices (CSI) when generating DNA sequences for various organisms. The model adapts to the specific codon preferences of each host, ensuring optimal expression.
-
+
-
-
+
+
+
+
+
Generating Natural-Like Codon Distributions
+
+
+ The model produces DNA sequences with codon usage patterns closely resembling those found in nature, avoiding
+ clusters of rare or highly frequent codons that can negatively affect protein folding and expression. This is
+ visualized using %MinMax profiles and Dynamic Time Warping (DTW) distance metrics.
+
+
+
+
+
+
+
+
+
Fig. 3: CodonTransformer generates natural-like codon distributions.
+
a. Schematic representation of %MinMax and dynamic time warping (DTW). %Minmax represents
+ the proportion of common and rare codons in a sliding window of 18 codons. DTW algorithm computes the minimal
+ distance between two %MinMax profiles by finding the matching positions (Methods). b. %MinMax
+ profiles for sequences generated by different models for genes yahG (E. coli), SER33 (S. cerevisiae),
+ AT4G12540 (A. ²thaliana), Csad (M. musculus), ZBTB7C (H. sapiens). c. DTW distances between
+ %MinMax profiles of model-generated sequences and their genomic counterparts for 50 random genes selected
+ among the top 10% codon similarity index (CSI). For each organism, the gene for which the %MinMax profiles are
+ represented above (b) is highlighted in grey. d. Mean and standard deviation of normalized
+ DTW distances by sequence length between sequences for the 5 organisms (for organism-specific DTW distances,
+ see Supplementary Figs. 17). Data underlying this figure is provided in Supplementary Data 1.
+
+
+
+
+
+
Benchmarking with Real World Proteins
+
+
+ When benchmarked against proteins of biotechnological interest, CodonTransformer consistently generates
+ sequences with minimized negative cis-regulatory elements, outperforming other tools. This enhances the
+ potential for successful heterologous expression.
+
+
+
+
+
+
+
+
+
Fig. 4: Model benchmark with proteins of biotechnological interest.
+
Mean and standard deviation of Jaccard index (a), sequence similarity (b),
+ and dynamic time warping (c) distance between corresponding sequences for the 52 benchmark
+ proteins across the 5 organisms (for organism-specific results, see Supplementary Figs. 19, 20, and 21,
+ respectively). (d), Number of negative cis-elements in the sequences generated by different
+ tools (✕ shows the mean). Data underlying this figure is provided in Supplementary Data 2.
-
Fig. 2: CodonTransformer learned codon patterns across organisms.
-
Codon usage index (CSI) for all and the top 10% CSI original genes and generated DNA sequences for all original proteins by CodonTransformer (base and fine-tuned models) for 9 out of 15 genomes used for fine-tuning in this study. See Supplementary Figs. 2-16 for all 15 genomes and additional metrics of GC content codon and distribution frequency (CDF). Source data for Fig. 2 and Supplementary Figs. 2-16 is available at https://zenodo.org/records/13262517.
+
-
+
+
+
Getting Started
-
-
-
Generating Natural-Like Codon Distributions
-
-
- The model produces DNA sequences with codon usage patterns closely resembling those found in nature, avoiding clusters of rare or highly frequent codons that can negatively affect protein folding and expression. This is visualized using %MinMax profiles and Dynamic Time Warping (DTW) distance metrics.
+
+
+ Along with open-source the data and model, we also provide a comprehensive Python package for
+ codon optimization. The CodonTransformer package has 5 modules:
-
-
-
-
-
-
-
Fig. 3: CodonTransformer generates natural-like codon distributions.
-
a. Schematic representation of %MinMax and dynamic time warping (DTW). %Minmax represents the proportion of common and rare codons in a sliding window of 18 codons. DTW algorithm computes the minimal distance between two %MinMax profiles by finding the matching positions (Methods). b. %MinMax profiles for sequences generated by different models for genes yahG (E. coli), SER33 (S. cerevisiae), AT4G12540 (A. ²thaliana), Csad (M. musculus), ZBTB7C (H. sapiens). c. DTW distances between %MinMax profiles of model-generated sequences and their genomic counterparts for 50 random genes selected among the top 10% codon similarity index (CSI). For each organism, the gene for which the %MinMax profiles are represented above (b) is highlighted in grey. d. Mean and standard deviation of normalized DTW distances by sequence length between sequences for the 5 organisms (for organism-specific DTW distances, see Supplementary Figs. 17). Data underlying this figure is provided in Supplementary Data 1.
-
-
+
- 
-
-
- 2
- INSERM
- 
-
-
- 3
- Vector Institute
- 
-
- 
- 
-