1. Terms
1.1. GPT: Generative Pre-trained Transformer
1.2. LLMs: Large Language Models
1.3. Parameters (Weights)
1.3.1. Quantization
1.3.1.1. Double Precision F64
1.3.1.2. Full Precision F32
1.3.1.3. Half Precision F16
1.3.1.4. 8 bit Precision
1.3.1.5. 4 bit Precision
1.4. Tokenization
1.5. Context
1.5.1. Prompt & Answer
1.6. Tensor
1.7. Training vs Inference
1.7.1. In-context Learning: Capability of learning during Inference
1.7.1.1. Zero-Shot: 0 example given
1.7.1.2. One-Shot: 1 example given
1.7.1.3. Few-Shot: 2–5 examples given
1.7.1.4. Many-Shot: Many examples given
2. Challenges
2.1. Hallucination
2.1.1. Solution: Interrogate
2.1.2. Solution: Trigger online search
2.2. Math: Number comparison
2.3. Spelling
2.4. Dataset Quality
2.4.1. Clock drawing
2.4.2. Left-handed writer
2.4.3. Hands drawing
2.5. Context: Invisible Prompt Injection
2.6. Bias
2.7. Ask of Origin
2.7.1. Hard-code
2.7.2. System message
2.8. *World-wide Concerns by United Nations
2.8.1. Inequality between countries
2.8.2. Reduction in low-wage advantage of developing countries
2.8.3. Unemployment
3. Recognition Path
3.1. Setup AI Model
3.1.1. Fixed number of dimensions to represent Token (E.g: 12288 dimensions)
3.1.2. Fixed number of attention heads for the task AI Model does (E.g: 96 heads)
3.1.3. Fixed 3 roles within Each head: - Q with a 2D matrix of random weights (E.g: 12288 rows x 128 columns) - K with a 2D matrix of random weights (E.g: 12288 rows x 128 columns) - V with a 2D matrix of random weights (E.g: 12288 rows x 128 columns) 128 is what each head gets to process = 12288 dimensions / 96 heads so when concatenating we have a square shape 12288 x 12288
3.1.4. Fixed number of layers for understanding depth (E.g: 96 layers aka 96 Transformer blocks)
3.2. Raw Data Input
3.3. Tokenization to assign ID to words/subwords
3.4. Random mapping each ID to a vector in multi-dimensional space
3.5. Collection of contexts for each ID shifts the ID vector to a fitting direction/distance
3.6. Association forming for each ID
4. AGI Path
4.1. Early AGI: Learn without retraining
4.2. Advanced AGI: Understanding the world
4.2.1. Common Sense
4.2.2. Creativity
4.2.3. Emotional Intelligence
4.2.4. Interaction via Robot
4.3. AGI: Beyond human intelligence
5. Main Players
5.1. Autoregressive
5.1.1. OpenAI - GPT o4 (Reasoning)/4.1 **#Codex CLI**
5.1.2. Google - Gemini 2.5 Pro/Gemma 3 **#Firebase Studio**
5.1.3. Anthropic (Amazon) - Claude Sonnet 3.7 **#Claude Code**
5.1.4. xAI - Grok 3
5.1.5. Meta (Facebook) - Llama 4
5.1.6. Microsoft - Phi 4/MAI-DS-R1
5.1.7. Mistral AI (France/Europe) - Mistral Small 3/Mixtral
5.1.8. Deepseek - Deepseek R1 (Reasoning)/V3
5.1.9. Alibaba - QwQ (Reasoning)/Qwen 3
5.2. Diffusion
5.2.1. Features
5.2.1.1. Text-to-Image
5.2.1.1.1. Stable Diffusion
5.2.1.1.2. DALL-E 3
5.2.1.1.3. Imagen
5.2.1.1.4. Midjourney
5.2.1.2. Text-to-Video
5.2.1.2.1. Sora
5.2.1.2.2. Make-A-Video
5.2.1.2.3. CogVideo
5.2.1.2.4. Runway
5.2.1.3. Text-to-Audio
5.2.1.3.1. AudioLDM
5.2.1.3.2. Riffusion
5.2.2. General Flow
5.2.2.1. UNet
5.2.2.1.1. Mechanism
5.2.2.2. VAE
5.2.2.3. CLIP
5.2.2.4. Miscellaneous
6. Applications
6.1. Out-of-the-box
6.1.1. Internet Search / Deep Search
6.1.2. Think Mode / Reason Mode
6.1.3. Analyzing / Summarizing / Translation
6.1.4. Voice / Video Conversation
6.1.5. Image creation / Image Quiz
6.1.6. Custom GPTs
6.2. Book reading / Podcast creation / Flash cards app / Diagram
6.3. Tailored LLMs
6.3.1. SFT **Online LLMs' instances**
6.3.2. SFT & RL **Local LLMs**
6.4. MCP App Development (Model Context Protocol)
6.4.1. Bring Tools to Agents we don't control
6.4.2. Bring Tools to Agents we can't develop (Suitable for Non-developers)
6.5. AI Agent
6.5.1. Autogen (Microsoft)
6.5.2. CrewAI (LangChain-based)
6.5.3. LangChain & LangGraph (LangChain)
6.5.4. OpenAI SDK (OpenAI)
6.5.4.1. Swarm (OpenAI)
6.5.5. Google ADK (Google)
6.5.5.1. Agent2Agent Protocol
6.6. AI Workflows
6.6.1. Prompt Chaining
6.6.2. Prompt Routing
6.6.3. Parallel Prompting
6.6.4. Evaluator & Optimizer
6.7. Robot
7. Training Phases (for Autoregressive Models)
7.1. Pre-training
7.1.1. Data Collection
7.1.1.1. Download
7.1.1.2. Filter
7.1.1.3. Remove Duplicates
7.1.1.4. Keep in text format
7.1.2. Embedding Layer
7.1.2.1. Embedding
7.1.2.1.1. Tokenization
7.1.2.1.2. Mapping Tokens to Vectors
7.1.2.1.3. Convert Vectors to Tensors
7.1.2.2. Vectorization: Refine Vectors
7.1.2.2.1. Positional Encoding/Embedding: Add position info about where the token is in the context
7.1.2.2.2. Segment Embedding: Distinguish sentences from each other (Optional)
7.1.3. Transformer Layers
7.1.3.1. Transformer
7.1.3.1.1. Encoder
7.1.3.1.2. Decoder
7.1.3.2. Vectorization: Refine Vectors
7.1.3.2.1. Contextual Adjustment: Add more relevant context to each token
7.1.3.2.2. Sequence Representation: Create new intermediate Vector to represent meaning of entire sentence
7.1.4. Output Layer
7.1.4.1. For Training
7.1.4.2. For Inference
7.1.4.2.1. Output single vector representing the whole sequence of tokens (aka Answer)
7.2. Post-training (Supervised Finetuning)
7.2.1. Methods
7.2.1.1. Full Finetuning
7.2.1.1.1. Epoch
7.2.1.1.2. Batch size
7.2.1.1.3. Learning Rate
7.2.1.2. Partial Finetuning
7.2.1.2.1. Same as Full Finetuning but freeze some layers
7.2.1.3. Adapter-based: Adding new features
7.2.1.4. Prompt Finetuning: Giving hints
7.2.1.5. Transfer Learning
7.2.1.5.1. Feature Extraction: New Model with extracted functions
7.2.1.5.2. Distillation: New Model optimized for performance/speed
7.2.2. Output: Weights Adjustments
7.2.2.1. Backpropagation
7.2.2.2. Loss Functions
7.2.2.2.1. Cross-Entropy Loss
7.2.2.2.2. Masked Language Modeling (MLM) Loss
7.2.2.2.3. Contrastive Loss
7.2.2.2.4. Triplet Loss
7.2.2.2.5. Mean Square Error (MSE) Loss
7.2.2.2.6. Custom Loss
7.3. Post-training (Reinforcement Learning)
7.3.1. Verifiable Domain vs Unverifiable Domain
7.3.2. Chain of thought (Aha Moment)
7.3.2.1. Tree of Thoughts (ToT)
7.3.2.2. Graph of Thoughts (GoT)
7.3.2.3. Chain of Draft (CoD) Prompting
7.3.3. Methods
7.3.3.1. Reinforcement Learning from AI Feedback (RLAIF)
7.3.3.1.1. Constitutional AI Model (Anthropic ONLY)
7.3.3.2. Reinforcement Learning from Human Feedback (RLHF) # aka Proximal Policy Optimization (PPO)
7.3.3.2.1. Models
7.3.3.2.2. Generalised Advantage Estimator (GAE)
7.3.3.3. Direct Preference Optimization (DPO)
7.3.3.4. Group Relative Policy Optimization (GRPO)
7.3.3.4.1. Reward Models
7.3.3.4.2. Rejection Sampling