AI Engineering Transition Path

Research papers for software engineers to transition to AI Engineering

Tokenization

• Byte-pair Encoding
• Byte Latent Transformer: Patches Scale Better Than Tokens

Vectorization

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• IMAGEBIND: One Embedding Space To Bind Them All
• SONAR: Sentence-Level Multimodal and Language-Agnostic Representations
• FAISS library
• Facebook Large Concept Models

Infrastructure

• TensorFlow
• Deepseek filesystem
• Milvus DB
• Billion Scale Similarity Search : FAISS
• Ray

Core Architecture

• Attention is All You Need
• FlashAttention
• Multi Query Attention
• Grouped Query Attention
• Google Titans outperform Transformers
• VideoRoPE: Rotary Position Embedding

Mixture of Experts

• Sparsely-Gated Mixture-of-Experts Layer
• GShard
• Switch Transformers

RLHF

• Deep Reinforcement Learning with Human Feedback
• Fine-Tuning Language Models with RHLF
• Training language models with RHLF

Chain of Thought

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Chain of thought
• Demystifying Long Chain-of-Thought Reasoning in LLMs

Reasoning

• Transformer Reasoning Capabilities
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Scale model test times is better than scaling parameters
• Training Large Language Models to Reason in a Continuous Latent Space
• DeepSeek R1
• A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods
• Latent Reasoning: A Recurrent Depth Approach
• Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo

Optimizations

• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• ByteDance 1.58
• Transformer Square
• Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
• 1b outperforms 405b
• Speculative Decoding

Distillation

• Distilling the Knowledge in a Neural Network
• BYOL - Distilled Architecture
• DINO

SSMs

• RWKV: Reinventing RNNs for the Transformer Era
• Mamba
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• Distilling Transformers to SSMs
• LoLCATs: On Low-Rank Linearizing of Large Language Models
• Think Slow, Fast

Competition Models

• Google Math Olympiad 2
• Competitive Programming with Large Reasoning Models
• Google Math Olympiad 1

Hype Makers

• Can AI be made to think critically
• Evolving Deeper LLM Thinking
• LLMs Can Easily Learn to Reason from Demonstrations Structure

Hype Breakers

• Separating communication from intelligence
• Language is not intelligence

Image Transformers

• Image is 16x16 word
• CLIP
• deepseek image generation

Video Transformers

• ViViT: A Video Vision Transformer
• Joint Embedding abstractions with self-supervised video masks
• Facebook VideoJAM ai gen

Case Studies

• Automated Unit Test Improvement using Large Language Models at Meta
• Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering
• OpenAI o1 System Card
• LLM-powered bug catchers
• Chain-of-Retrieval Augmented Generation
• Swiggy Search
• Swarm by OpenAI
• Netflix Foundation Models
• Model Context Protocol
• uber queryGPT

More Resources

I manage my lists here: https://interviewready.io/resources/