Notes

Speculative decoding is a clever inference optimization technique that exploits a fundamental asymmetry in how LLMs work: verifying a token is much cheaper than generating one. The Basic Setup You run two models simultaneously — a small, fast “draft” model and your large “target” model. The draft model generates several tokens ahead in a single pass (typically 4–8 tokens). The large model then verifies all of those candidate tokens in parallel in one forward pass. If the draft tokens match what the large model would have produced, you accept them all at once. If a token diverges, you reject it (and everything after it) and fall back to the large model’s output for that position. ...

Model weights are the learned numbers inside the neural network. During training, the model adjusts billions of numeric parameters so that, given some input text, it becomes better at predicting the next token. Those parameters are the weights. Short Intuition A useful way to think about it: The model architecture is the blueprint. The weights are the filled-in values that make the blueprint useful. Without weights, the model is just an empty structure. What Weights Do Weights control how information flows through the network. ...

GGUF (GPT-Generated Unified Format) is a binary file format for storing and distributing large language models, designed specifically for efficient local inference. Background Introduced by the llama.cpp project in 2023 as a replacement for the older GGML format. The name reflects its origins but it’s now used broadly across many model families beyond GPT. Key Characteristics Self-contained — A single .gguf file bundles everything needed to run a model: weights, tokenizer vocabulary, metadata, and architecture config. No separate config files needed. ...

Prompt bias is a type of AI bias that comes from how a question or instruction is written, not just from the model itself. In simple terms: The wording, framing, or assumptions in a prompt can push an AI toward a particular answer—even if that answer isn’t neutral or fully accurate. What Prompt Bias Looks Like Here’s a quick comparison: Neutral prompt: “What are the effects of remote work on productivity?” ...

What primacy bias is Primacy bias is the tendency of an AI model to give disproportionate weight to items that appear earlier in a list or prompt. When a model is asked to choose from many options, options shown first can become over-represented in the final answer even when later options are equally or more appropriate. In practical terms, this means that a selector prompt like: style-a style-b style-c … can systematically prefer style-a more often than expected if the candidates are always presented in the same order. ...

Slack MCP Ideas: Using Slack MCP monitor for automation opportunities within the org. Using Slack MCP identify duplicated efforts in the org.

ELO scoring for AI models works the same way it does in chess — it’s a method for ranking competitors based on head-to-head outcomes, where your rating rises or falls depending on whether you beat or lose to opponents of known strength. How it works The core idea: Every model starts with a baseline rating. When two models are compared, the system predicts the expected outcome based on the rating gap. If the actual result matches the prediction, ratings barely move. If an underdog wins, ratings shift dramatically. ...

Distillation in AI (also called knowledge distillation) is a model compression technique where a smaller “student” model is trained to mimic the behavior of a larger, more capable “teacher” model. How it works Instead of training the student on hard labels (e.g., “this image is a cat”), the student learns from the teacher’s soft outputs — the probability distribution the teacher assigns across all classes. These soft outputs carry richer information. For example, knowing a model thinks an image is 70% cat, 20% leopard, and 10% tiger tells the student more about the underlying structure than just “cat.” ...

Paper: Training-Free Group Relative Policy Optimization By: Youtu-Agent Team Publication date: October 9, 2025 The Problem Fine-tuning LLMs with reinforcement learning (RL) to improve agent performance in specialized domains is expensive, data-hungry, prone to overfitting, and kills cross-domain generalization. Most RL approaches are limited to sub-32B models due to compute constraints. The Core Idea Instead of updating model parameters (gradient-based RL), Training-Free GRPO updates model context — building an evolving library of experiential knowledge that gets injected into the prompt. The model weights stay frozen throughout. ...

Attention in AI Attention is a mechanism that allows a model to focus on the most relevant parts of its input when producing an output — much like how humans pay more attention to certain words or objects in a scene than others. The Core Idea Instead of treating all parts of the input equally, attention assigns weights to different elements, so the model can dynamically decide what’s important for each step of its task. ...

The Transformer architecture, introduced in the 2017 paper “Attention Is All You Need” by Vaswani et al., revolutionized AI by replacing recurrent networks with a purely attention-based design. Here’s a breakdown of how it works: Core Idea: Self-Attention Instead of processing sequences step-by-step (like RNNs), Transformers process all tokens in parallel and learn relationships between every pair of tokens simultaneously. This is done via self-attention. For each token, three vectors are computed: ...

RNN stands for Recurrent Neural Network — a type of neural network designed to work with sequential data. Unlike standard feedforward networks, RNNs have a “memory” mechanism: they pass information from one step to the next, making them well-suited for tasks where order and context matter, like text, speech, or time-series data. The key idea is that at each step, the network takes both the current input and a hidden state from the previous step, producing an output and an updated hidden state. ...

RLHF and DPO: Aligning AI to Human Preferences Both techniques address the same core problem: after pre-training on raw text, a language model needs to be steered toward responses that are helpful, safe, and aligned with what humans actually want. They’re two different approaches to the same goal. RLHF — Reinforcement Learning from Human Feedback The idea: Train a separate model to predict what humans prefer, then use that model as a reward signal to fine-tune the LLM via RL. ...

Instruction tuning is a fine-tuning technique where a pre-trained language model is further trained on a dataset of (instruction, response) pairs to make it better at following natural language instructions. How it works A base language model trained on raw text is good at predicting the next token, but not necessarily at being helpful. Instruction tuning bridges that gap by showing the model thousands to millions of examples like: Instruction: “Summarize this article in 3 bullet points.” Response: “• Point 1 …” The model learns to map user intent → useful output. ...

Perplexity in Language Models Perplexity measures how well a probability model predicts a sample of text. Intuitively, it captures how “surprised” or “perplexed” a model is when it encounters new text — a lower perplexity means the model found the text more predictable, i.e., it’s a better model. The Core Idea A language model assigns a probability to every sequence of words. Given a test sentence, the model predicts the probability of each next word given all preceding words: ...

Model quantization is the process of reducing the numerical precision of a neural network’s weights (and sometimes activations) to make models smaller and faster, with acceptable loss in accuracy. The core idea Neural networks store parameters as floating-point numbers — typically 32-bit floats (float32). Quantization maps these to lower-precision representations like 16-bit floats, 8-bit integers, or even 4-bit integers. Fewer bits per number means less memory and faster arithmetic. Common precision levels Format Bits Typical use float32 32 Training baseline bfloat16 / float16 16 Training & inference on GPUs int8 8 Efficient inference int4 / int3 / int2 4 or less Aggressive compression (LLMs) How it works Post-training quantization (PTQ) takes a trained model and converts its weights after the fact. It’s fast and simple but can hurt accuracy at very low bit depths. ...

To login g c l o u d a u t h l o g i n Application Default Login g c l o u d a u t h a p p l i c a t i o n - d e f a u l t l o g i n Project Set g c l o u d c o n f i g s e t p r o j e c t b u d d y h q - p r d GKE cluster context set dev: ...

Use this command: k u b e c t l p o r t - f o r w a r d s v c / t e m p o r a l - w e b 8 0 8 0 : 8 0 8 0 The first port is the localhost port, the second the service port.