AI, ML & NLP Foundations | Open Agent School
https://www.openagentschool.org/ai-ml-foundations
Understand the language. See how it works. Know when it matters.
Public authored learning content. Interactive experiments require JavaScript but no login or API key.
Synthetic teaching calculations are not LLM training, production evidence, or independently verified competency.
Related destinations retain their own access requirements. Optional AI tutoring uses learner-selected providers.

LESSON: Dot product
URL: https://www.openagentschool.org/ai-ml-foundations/dot-product
Also called: inner product
Summary: Multiply matching features, then add: a small operation behind model scores and retrieval.
In everyday words: A dot product takes two equal-length lists of numbers, multiplies matching positions, and adds the results. It returns one number, not a new list. Both direction and size affect that number.
Why it matters: A model can score an item by pairing its features with learned importance weights. Understanding each contribution makes an otherwise mysterious score inspectable.
Scenario - A tiny help-desk score: In this synthetic example, a ticket has features [2, 1] for urgency and customer impact. Weights [3, -1] give 2 times 3 plus 1 times -1: a score of 5. This is a ranking rule, not a probability.
Analogy - A shopping receipt: Multiply each quantity by its price and add the line totals. A negative weight is like a discount that subtracts from the bill.
Where the analogy stops: Model features need not be quantities you can buy, and a large score does not prove that an item is useful.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/dot-product.webp
Picture description: Groceries paired with blank price tags, a clipped discount coupon, and an unprinted receipt on a checkout counter.
Picture caption: A receipt adds quantity-times-price contributions; a discount subtracts. Model scores are not prices, probabilities, or proof of usefulness.
What you see: Input quantities. What it represents: Each grocery quantity stands for one input feature.
What you see: Matching weights. What it represents: Pair each quantity with its own price, like matching components in a dot product.
What you see: Negative contribution. What it represents: A discount subtracts from the total, as a negative weighted contribution can.
What you see: One combined score. What it represents: The receipt represents the sum of all matched contributions, without normalization.
Predict: If you double one whole vector while keeping the other fixed, what happens to the dot product?
Try: Change a vector component and trace its matching product before inspecting the total. Compare the original vector with a positively scaled copy.
Explain: Doubling a whole vector doubles every contribution and therefore the total. Changing only one coordinate changes only its matching product; negative weights can make an increase lower the score.
Optional math: s = sum_i(x_i * w_i)
s: the resulting scalar score
x_i: input feature at position i
w_i: weight at the same position i
sum_i: add the products over all positions i
For [2, 1] and [3, -1], s = 6 - 1 = 5. There is no division by vector lengths: this is not normalized cosine similarity.
Common misconception: Dot product always lies between -1 and 1.
Correction: An unnormalized dot product is unbounded. Only when both vectors have unit length does it equal their cosine similarity.
Check your understanding: What is the dot product of [2, 1] and [3, -1]?
5 - Yes. Matching products are 6 and -1; their sum is 5.
7 - The second weight is negative. Add -1, not +1, to the first product.
Always a value between -1 and 1 - That range belongs to cosine similarity, not an unnormalized dot product.
Use it with purpose: Should larger feature magnitudes automatically improve your retrieval score?
Guidance: Use a dot product when magnitude is intentionally meaningful. If you want direction-only comparison, consider cosine similarity and evaluate the retrieval results rather than assuming either metric is universally best.
Where to go next:
Retrieval engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Vector scoring choices change which evidence an agent retrieves.
Agentic RAG (pattern): https://www.openagentschool.org/patterns/agentic-rag - Inspect how a retrieval score helps select evidence before an agent writes an answer.
References:
Dive into Deep Learning: Linear Algebra: https://d2l.ai/chapter_preliminaries/linear-algebra.html

LESSON: Matrix multiplication
URL: https://www.openagentschool.org/ai-ml-foundations/matrix-multiplication
Also called: matrix product
Summary: Turn many row-by-column dot products into a table of model outputs.
In everyday words: Matrix multiplication combines two rectangular tables using dot products: each output cell comes from one row of the first table and one column of the second. The first table must have as many columns as the second has rows.
Why it matters: Neural networks process many examples and many output features at once. Matrix multiplication expresses that shared work without writing a separate scoring rule for every example.
Scenario - Two tickets, two scores: A synthetic feature table X has rows [1, 2] and [3, 4]. A weight table W has rows [2, 0] and [0, 1]. XW has rows [2, 2] and [6, 4]: the first feature doubles while the second stays unchanged.
Analogy - A recipe production sheet: One table lists batches of ingredients; another lists how each ingredient contributes to each product. Row-by-column totals tell you the output for each batch.
Where the analogy stops: Matrix entries can be negative, and multiplication order matters. This is not ordinary element-by-element multiplication.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/matrix-multiplication.webp
Picture description: Bakery ingredient batches, an unprinted production grid with ingredient samples, and finished pastry trays share one work counter.
Picture caption: One ingredient batch combines with a product recipe to make an output. Matrix multiplication uses row-by-column sums, not cellwise pairing.
What you see: Input rows. What it represents: Each ingredient batch stands for a row of input features.
What you see: Recipe columns. What it represents: Each product recipe gathers contributions across ingredients, like a weight-matrix column.
What you see: Output table. What it represents: Batch-by-product totals stand for the output matrix, whose shape follows the outer dimensions.
Predict: If you double every entry of the input matrix while keeping the weight matrix fixed, what happens to each output cell?
Try: Change the input matrix scale. Pick one output cell and follow its input row and weight column through the matching products.
Explain: Every product in that cell scales by the same factor, so their sum does too. Doubling the input matrix doubles every output entry, including negative entries; the row-by-column rule still determines each value.
Optional math: Y_ij = sum_k(X_ik * W_kj)
Y_ij: output at example row i and output column j
X_ik: input feature k for example i
W_kj: weight connecting input k to output j
sum_k: add over all shared feature positions k
If X is a 2-by-3 table and W is 3-by-4, Y is 2-by-4. The shared size 3 is the number of products inside each dot product.
Common misconception: AB and BA always produce the same result.
Correction: Order generally changes the result, and the reversed product may not even have compatible dimensions.
Check your understanding: A 2-by-3 input table multiplies a 3-by-4 weight table. What shape is the output?
3 by 3 - The shared dimension is summed over; it does not become the output shape.
2 by 4 - Yes. The two example rows and four output columns remain.
2 by 3 - The output columns come from the weight table, which has four columns.
Use it with purpose: A model reports an incompatible-shape error. What do you inspect first?
Guidance: Write the meaning and size of each axis. Check that input-feature columns match weight rows before changing batch size or transposing a table; a transpose can hide a meaning error.
Where to go next:
What is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Large language models repeatedly transform batches of numerical representations with matrix operations.
References:
Dive into Deep Learning: Linear Algebra: https://d2l.ai/chapter_preliminaries/linear-algebra.html

LESSON: Linear layer
URL: https://www.openagentschool.org/ai-ml-foundations/linear-layer
Also called: dense layer, affine layer, fully connected layer
Summary: Learn how weights mix features and bias shifts the result before any activation.
In everyday words: A linear layer mixes input features with weights and adds a bias for each output. Libraries usually call this a linear layer, although adding a nonzero bias makes the transformation affine: it can shift as well as scale and mix.
Why it matters: This is a common building block inside models. Separating the weighted contribution from the bias helps you explain why a score changes even when an input is zero.
Scenario - A synthetic delivery estimate: A tiny model predicts minutes from distance using weight 2 and bias 5. A distance of 3 gives 11 minutes. The 5-minute bias represents a fixed preparation allowance in this toy rule, not a discovered causal law.
Analogy - A taxi fare: Distance-dependent charges resemble weights; the starting fare resembles bias. Changing the starting fare moves every estimate by the same amount.
Where the analogy stops: Real fares and delivery times can have thresholds and interactions that a single linear layer cannot express.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/linear-layer.webp
Picture description: Inside a parked taxi, base-fare tokens sit beside a cord-marked road map and a dish representing the combined fare.
Picture caption: A starting fare resembles bias; distance-based charges resemble weighted input. Real fares may contain thresholds a single affine rule cannot capture.
What you see: Starting bias. What it represents: The base fare contributes even before the journey distance is added.
What you see: Weighted input. What it represents: Distance contributes according to a rate, like an input multiplied by a weight.
What you see: Combined output. What it represents: Adding the base allowance to the variable charge illustrates an affine output.
Predict: Will changing only the bias alter the slope or shift the output?
Try: Hold the weights and input fixed, change the bias, and compare the weighted sum with the final output.
Explain: The bias adds the same offset regardless of the input. Changing a weight instead changes how sensitive the output is to its corresponding feature.
Optional math: y = W x + b
y: output vector
W: weight matrix with one row per output
x: input column vector
b: bias vector with one value per output
For one input and output, y = 2 * 3 + 5 = 11. No activation function is included in this equation.
Common misconception: A bias-free weighted sum and a biased linear layer are exactly the same mathematical transformation.
Correction: A strictly linear map sends zero to zero. An affine layer can send zero to its bias; this distinction matters when interpreting a model.
Check your understanding: With weight 2 and bias 5, what does an input of 0 produce?
0 - The weighted part is zero, but the layer still adds its bias.
2 - The weight multiplies the input; it is not added by itself.
5 - Yes. The zero input contributes nothing, leaving the bias of 5.
Use it with purpose: A target rises sharply only after two features occur together. Is one linear layer enough?
Guidance: Use it as an interpretable baseline, then test nonlinear features or a nonlinear network if the baseline misses the interaction. More layers without nonlinear activations do not solve that representational limit.
Where to go next:
Atomic LLM training (concept): https://www.openagentschool.org/concepts/atomic-llm-training - Identify which trainable numbers belong to weights and which belong to biases.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: ReLU
URL: https://www.openagentschool.org/ai-ml-foundations/relu
Also called: rectified linear unit
Summary: See the simple bend that keeps positive signals and clips negative ones to zero.
In everyday words: ReLU, the rectified linear unit, returns zero for a negative input and returns the input itself for a positive input. This bend makes a network capable of more than one straight-line transformation.
Why it matters: Nonlinear activations allow layers to compose flexible functions. ReLU is simple to compute, but a unit that stays negative can stop receiving useful gradient through this activation.
Scenario - An issue detector: A synthetic detector combines two signals into a pre-activation score. Scores -2, 0, and 3 become 0, 0, and 3 after ReLU. These are internal signals, not percentages of certainty.
Analogy - A one-way overflow marker: Only a level above a reference line is recorded; below the line the marker reads zero. Once above the line, each extra unit is preserved.
Where the analogy stops: A real valve can leak or have inertia. ReLU is an exact mathematical rule, and negative evidence may be useful elsewhere in a network.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/relu.webp
Picture description: Independent clear vessels contrast water below an overflow notch with water above a matching notch, aligned by a shared reference strip.
Picture caption: Below the reference, ReLU returns zero; above it, ReLU preserves positive excess height. The exact rule is max(0, input), not a claim about hydraulic flow.
What you see: Below the reference. What it represents: A negative relative height stands for input that ReLU maps to zero.
What you see: Zero boundary. What it represents: The reference edge represents the point where positive output begins.
What you see: Positive excess. What it represents: Height above the reference illustrates the positive part that ReLU preserves.
Predict: Does moving an input from a more negative value to a less negative value change ReLU output?
Try: Compare inputs below zero, at zero, and above zero. Watch for the point where output starts tracking input.
Explain: All negative inputs produce zero. Above zero the output rises one-for-one. The derivative at exactly zero is undefined mathematically; software uses a chosen convention.
Optional math: ReLU(z) = max(0, z)
z: the pre-activation input
ReLU(z): the activated output
max: choose the larger of the two arguments
ReLU(-2) = 0 and ReLU(3) = 3. Its local slope is 0 below zero and 1 above zero.
Common misconception: ReLU outputs are probabilities because they cannot be negative.
Correction: ReLU has no upper bound. An output of 3 is valid, so it is not a probability transformation.
Check your understanding: An input changes from -4 to -1. What happens to its ReLU output?
It stays at 0. - Yes. Both inputs are on the clipped, negative side.
It increases by 3. - That would be true for an identity function, but ReLU clips both inputs to zero.
It becomes -1. - ReLU chooses the larger of zero and the input, so -1 is replaced with zero.
Use it with purpose: A hidden unit produces zero on every training example. What is worth checking?
Guidance: Inspect its pre-activation values, initialization, and learning rate. Persistent negative inputs can leave a ReLU inactive; do not interpret that alone as proof the input feature is irrelevant.
Where to go next:
Agent learning (concept): https://www.openagentschool.org/concepts/agent-learning - Understand why a model update may fail to move a persistently inactive internal unit.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Sigmoid
URL: https://www.openagentschool.org/ai-ml-foundations/sigmoid
Also called: logistic function
Summary: Map a real-valued score into 0 to 1 without confusing that number with proven reliability.
In everyday words: The logistic sigmoid smoothly maps any finite real score into a value strictly between 0 and 1. It is often used to parameterize the probability of one outcome in a binary model.
Why it matters: A bounded output can support a yes/no decision, but the transformation alone does not make the model calibrated. Calibration asks whether predicted probabilities match observed frequencies on suitable data.
Scenario - A synthetic spam score: A toy classifier emits a raw score of 0 for an ambiguous message. Sigmoid maps it to 0.5. Raising the raw score raises the output, but whether messages scored near 0.8 are spam about 80% of the time requires a separate evaluation.
Analogy - A dimmer with end stops: The output moves smoothly between two limits. Near the middle, a small input change is noticeable; near either end, the same change has less effect.
Where the analogy stops: A dimmer setting is not a statement about how often a model is right. The analogy describes shape, not statistical reliability.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/sigmoid.webp
Picture description: A plain rotary dimmer low on a workbench connects by one visible cable to an illuminated shaded lamp.
Picture caption: A dimmer suggests a smoothly bounded output. Sigmoid is not a calibrated-confidence guarantee, and a real dimmer need not follow its curve.
What you see: Input setting. What it represents: Turning the control stands for changing the model's raw score.
What you see: Output, not certainty. What it represents: Lamp brightness represents an output level, not evidence that a prediction is correct.
Predict: Will the same score increase change the output equally near the center and far from it?
Try: Compare score changes around zero with changes at a large positive or negative score.
Explain: The curve is steepest near zero and flattens toward the ends. This saturation also reduces the local gradient; a high output is not independently verified confidence.
Optional math: p = 1 / (1 + exp(-z))
p: sigmoid output, often used as a binary probability estimate
z: raw real-valued score, also called a logit
exp: the exponential function, raising Euler's number e to its argument
At z = 0, exp(0) = 1, so p = 0.5. The formula specifies an output shape, not a calibration guarantee.
Common misconception: A sigmoid output of 0.9 proves the classifier is 90% likely to be correct.
Correction: It is a model estimate for the named outcome. Reliability must be checked against held-out outcomes, including relevant user groups and changed conditions.
Check your understanding: What does sigmoid do to a raw score of 0?
Returns 0 - The denominator is 1 + exp(0) = 2, not infinity.
Returns 0.5 - Yes. Zero is the center of the logistic curve, not a zero probability.
Proves the model is calibrated - A mathematical transformation cannot substitute for comparing predictions with observed outcomes.
Use it with purpose: Can you route high-stakes requests automatically whenever this output exceeds 0.9?
Guidance: First check calibration and the consequences of errors on representative held-out data. Choose a threshold and human-review policy from those costs, not merely from a visually reassuring number.
Where to go next:
Uncertainty and calibration (concept): https://www.openagentschool.org/concepts/agent-uncertainty-calibration - Separate a model probability estimate from evidence that the estimate is trustworthy.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html
scikit-learn: Model Evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

LESSON: Tanh
URL: https://www.openagentschool.org/ai-ml-foundations/tanh
Also called: hyperbolic tangent
Summary: Explore an activation that preserves sign while compressing values toward -1 and 1.
In everyday words: Tanh is a smooth function that maps finite real inputs into values strictly between -1 and 1. Negative inputs remain negative, zero remains zero, and positive inputs remain positive.
Why it matters: Some model states need a bounded signed signal rather than a positive-only output. Tanh provides that representation, although its flattened ends can make gradient-based learning slower.
Scenario - A signed feedback signal: In a synthetic feedback model, negative scores oppose a proposed action and positive scores support it. Tanh maps a zero score to 0 and a score of 1 to about 0.762, keeping the sign while limiting the size.
Analogy - A spring-loaded control stick: The center is neutral; movements left or right express opposite directions within a bounded range.
Where the analogy stops: A physical stick hits a hard stop. Tanh approaches its limits smoothly, and its sign is a model representation rather than a moral judgment.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/tanh.webp
Picture description: A spring-centered control stick rests between physical left and right travel stops on an industrial test bench.
Picture caption: The centered stick pictures a bounded signed signal. Tanh approaches its limits smoothly; a physical stick reaches hard stops.
What you see: Negative direction. What it represents: Movement to one side represents a negative signed activation, not a bad outcome.
What you see: Neutral center. What it represents: The centered pivot represents tanh mapping zero input to zero output.
What you see: Positive direction. What it represents: Movement to the opposite side represents a positive activation within a bounded range.
Predict: Will doubling a large positive input double the output?
Try: Compare equal input changes near zero and farther from zero, and compare a positive input with its negative.
Explain: Tanh saturates: large inputs produce progressively smaller output changes. Opposite inputs produce equal-magnitude outputs with opposite signs.
Optional math: tanh(z) = (exp(z) - exp(-z)) / (exp(z) + exp(-z))
z: real-valued input score
tanh(z): bounded, signed output
exp: the exponential function with base e, Euler's number
At zero, the numerator is 1 - 1 = 0 and the denominator is 2. The output is zero, unlike sigmoid's midpoint of 0.5.
Common misconception: Tanh and sigmoid have the same output range.
Correction: Tanh ranges from -1 to 1; logistic sigmoid ranges from 0 to 1. Neither reaches its limiting endpoints for finite real inputs in exact arithmetic.
Check your understanding: If tanh(1) is about 0.762, what is tanh(-1)?
About 0.762 - Tanh preserves the sign of its input; the negative input cannot give this positive value.
0 - Clipping negative values to zero is ReLU behavior, not tanh behavior.
About -0.762 - Yes. Tanh is symmetric through the origin: negating input negates output.
Use it with purpose: You want a bounded internal signal that distinguishes opposition from support. Which property matters?
Guidance: A zero-centered signed range can make tanh appropriate. Check saturation and learning behavior; do not choose it as a probability output without a separate, appropriate probability model.
Where to go next:
Atomic LLM training (concept): https://www.openagentschool.org/concepts/atomic-llm-training - Compare how activation choices shape internal states and the gradients used to train them.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Softmax
URL: https://www.openagentschool.org/ai-ml-foundations/softmax
Also called: normalized exponential
Summary: Turn competing scores into shares that sum to one, while keeping calibration separate.
In everyday words: Softmax transforms a list of real scores into positive shares that sum to 1. Raising one score relative to the others increases its share and reduces the total share available to the other options.
Why it matters: Models use softmax for mutually exclusive classes, token choices, and attention weights. The same normalization can represent very different things; an attention share is not a probability that a claim is true.
Scenario - Three synthetic routing options: A toy router gives billing, delivery, and returns equal scores. Softmax assigns each 1/3. If billing receives a larger score while the others stay fixed, billing gets more than 1/3 and the other two get less.
Analogy - Dividing a fixed pie: The shares compete for one whole pie. Giving one option a larger share leaves less for the rest.
Where the analogy stops: The pie says how a model allocates weight, not whether the selected option is correct. Separate examples need not use the same set of valid choices.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/softmax.webp
Picture description: A complete round pie is divided into unequal wedges, with its intact outer rim visible on a serving board.
Picture caption: Softmax divides one whole among competing options. Larger shares leave less for others, but a share is not proof that an option is correct.
What you see: Larger share. What it represents: Raising one relative score gives its option more of the normalized total.
What you see: Competing shares. What it represents: Other options share the remaining probability mass.
What you see: One fixed whole. What it represents: The complete pie represents normalized shares summing to one.
Predict: What changes if you raise one score, and what changes if you add the same amount to every score?
Try: Compare a single-score change with a shared shift of all scores. Check the total of the output shares.
Explain: Only relative score differences matter. A shared shift cancels during normalization, while a single-score increase redistributes the shares. Their sum remains 1.
Optional math: p_i = exp(z_i) / sum_j(exp(z_j))
p_i: normalized share for option i
z_i: raw score for option i
z_j: raw score for each option j in the denominator
exp: the exponential function with base e
sum_j: add across all available options j
Three zero scores give three exponentials of 1, so each share is 1/3. Implementations subtract the maximum score before exponentiating for numerical stability; the mathematical result is unchanged.
Common misconception: Softmax makes its largest output calibrated confidence.
Correction: Softmax guarantees normalized shares, not calibrated confidence. A wrong model can assign almost all its mass to a wrong answer.
Check your understanding: What happens when you add 10 to every score before softmax?
All output shares stay the same. - Yes. The common exponential factor cancels from numerator and denominator.
Every output share increases. - All shares cannot increase together because their sum must remain 1.
Every share becomes equal. - Adding a common offset preserves score differences; unequal scores remain unequal.
Use it with purpose: A document may belong to several independent categories. Should one softmax select all its labels?
Guidance: A single softmax models competition among exclusive options. Consider separate binary outputs for labels that can coexist, then evaluate each label and calibration on representative examples.
Where to go next:
Model routing (concept): https://www.openagentschool.org/concepts/model-routing - Understand how relative routing scores differ from validated routing decisions.
Uncertainty and calibration (concept): https://www.openagentschool.org/concepts/agent-uncertainty-calibration - Check whether normalized class probabilities match outcome frequencies rather than trusting their appearance.
Multi-LLM routing (pattern): https://www.openagentschool.org/patterns/multi-llm-routing - Compare score-based selection with routing policies that also account for cost and capability.
References:
scikit-learn: Model Evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html
Dive into Deep Learning: Softmax Regression: https://d2l.ai/chapter_linear-classification/softmax-regression.html

LESSON: Artificial neuron
URL: https://www.openagentschool.org/ai-ml-foundations/neuron
Also called: unit, perceptron unit
Summary: Trace one model unit from input contributions to weighted sum, bias, and activation.
In everyday words: An artificial neuron is a small computation: multiply inputs by weights, add those contributions and a bias, then apply an activation function. Many such units are connected to make a neural network.
Why it matters: Tracing one unit gives you a manageable way to reason about an entire model. You can distinguish input evidence, learned parameters, and the rule that shapes the output.
Scenario - A synthetic escalation unit: Inputs [2, 1], weights [1, -1], and bias 0 produce a pre-activation of 1. With ReLU activation the output is also 1. This hand-set unit illustrates a computation; it is not a trained support policy.
Analogy - A mixing station with a final filter: Each input enters with a strength and sign, the station adds them, and a filter decides how the combined signal leaves.
Where the analogy stops: The biological name is an inspiration, not a claim that this equation accurately models a brain cell or human understanding.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/neuron.webp
Picture description: Valved supply vessels feed a clear mixing chamber, followed by a mesh filter and receiving vessel.
Picture caption: A neuron combines weighted inputs before an activation. The mixer is a process analogy, not a brain cell or a literal model of negative weights.
What you see: Weighted inputs. What it represents: Separately adjusted feeds stand for input contributions with different strengths.
What you see: Combined signal. What it represents: The chamber pictures the weighted sum and added bias before activation.
What you see: Final transformation. What it represents: The final filter stands for an activation applied after contributions combine.
Predict: Can an input increase while the neuron output decreases?
Try: Inspect the sign of a weight, increase its corresponding input, and follow the weighted sum through the activation.
Explain: With a negative weight, that contribution decreases as the input grows. The experiment uses sigmoid, so the output changes smoothly and is less sensitive near its limits. The separate ReLU worked example can instead clip a negative pre-activation to zero.
Optional math: a = f(sum_i(w_i * x_i) + b)
a: neuron output, also called its activation
f: chosen activation function, such as ReLU
w_i: weight for input position i
x_i: input at position i
b: bias added after the weighted sum
sum_i: add all input contributions
Keep the pre-activation and activated output separate. In the example, ReLU(2 * 1 + 1 * -1 + 0) = 1.
Common misconception: Each neuron must correspond to one human-readable idea.
Correction: A unit can respond to several patterns, and a useful representation can be spread across many units. A convenient label is not proof of what a neuron understands.
Check your understanding: Which sequence describes this artificial neuron?
Activate each weight, then ignore inputs. - Weights matter through their products with inputs; the activation follows the combined score here.
Weight inputs, add bias, then activate. - Yes. This separates contributions, the pre-activation score, and the final output.
Count positive inputs and take a majority vote. - This would discard magnitudes and weights; it is not the specified neuron computation.
Use it with purpose: A neuron output looks surprising. Which intermediate value would you inspect?
Guidance: Trace each weighted contribution, then bias, then activation. This can reveal a sign or scaling issue before you draw a conclusion about the behavior of the whole model.
Where to go next:
What is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Replace the idea of a mysterious thinking cell with a traceable computation inside a larger language model.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Batch
URL: https://www.openagentschool.org/ai-ml-foundations/batch
Also called: mini-batch, batch size
Summary: Process several examples together and distinguish an average loss from an individual error.
In everyday words: A batch is a group of examples processed together. During common mini-batch training, the model computes a loss for each example and uses their mean to form one update.
Why it matters: Batch size affects memory use, computational throughput, and how noisy the gradient estimate is. A batch average can also hide one example with a much larger error.
Scenario - Three synthetic prediction errors: For three examples, the squared errors are 1, 4, and 9. Their mean batch loss is 14/3, about 4.667. Looking only at the first example would report 1 and miss most of this batch's error.
Analogy - Checking a tray of samples: Rather than adjust a process after every sample, inspect a tray and summarize it before making one adjustment.
Where the analogy stops: A larger tray is not automatically representative. Repeated or systematically selected examples can produce a misleading average.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/batch.webp
Picture description: A tray of varied ceramic samples sits beside an inspection lens and one blank review card with an adjustment knob.
Picture caption: Inspect individual examples, then summarize a batch before one update. A larger selected tray is not automatically more representative.
What you see: One batch. What it represents: The tray groups examples processed together rather than changing the model after every item.
What you see: Per-example evidence. What it represents: Each sample contributes its own loss or gradient before aggregation.
What you see: One mean update. What it represents: The review station represents combining the examples into one averaged training signal.
Predict: In the experiment's rule y = 2x, will adding larger-input examples strengthen the mean gradient when the model weight differs from 2?
Try: Keep the weight fixed and grow the batch. Compare the individual gradients, their mean, and the batch mean squared error.
Explain: Each gradient is 2x(wx - 2x), where x is the input and w is the current weight. Larger positive inputs increase its magnitude when w differs from 2, so adding these particular rows changes the mean. At w = 2 every gradient and error is zero. This is a changing deterministic sample, not proof that larger batches always give larger gradients.
Optional math: L_batch = sum_i(loss_i) / B
L_batch: mean loss for the batch
loss_i: loss on batch example i
B: positive number of examples in the batch
sum_i: add over those B examples
The example gives (1 + 4 + 9) / 3. Some systems sum rather than average; this lesson explicitly uses the mean, so the reduction convention is not hidden.
Common misconception: A larger batch always learns a better model.
Correction: It changes resource use and the update estimate, but final quality also depends on data, optimization settings, and evaluation. Larger is not a universal improvement.
Check your understanding: What is the mean loss for example losses 1, 4, and 9?
14 - That is the sum. Divide by three examples to obtain the mean.
1 - That is only the first example's loss; a batch summary must include all three.
14/3, approximately 4.667 - Yes. Add all three losses and divide by the batch size.
Use it with purpose: Training runs out of device memory. Is reducing batch size a reasonable first experiment?
Guidance: Yes, while recording the changed setting and checking learning behavior. Compare examples processed and evaluation quality, not update counts alone, because each update may now contain fewer examples.
Where to go next:
Production ML systems (concept): https://www.openagentschool.org/concepts/production-ml-systems - Connect the meaning of a training batch to memory limits, throughput, and reproducible experiments.
AI Engineer (course): https://www.openagentschool.org/courses/ai-engineer - Apply small-scale training arithmetic when judging model-building and operational trade-offs.
References:
Dive into Deep Learning: Linear Algebra: https://d2l.ai/chapter_preliminaries/linear-algebra.html
Dive into Deep Learning: Minibatch Stochastic Gradient Descent: https://d2l.ai/chapter_optimization/minibatch-sgd.html

LESSON: Weighted connection
URL: https://www.openagentschool.org/ai-ml-foundations/connection
Also called: edge, weight
Summary: Follow the signed contribution along one link rather than treating a network line as decoration.
In everyday words: A connection carries the output of one unit into another, scaled by a weight. In a standard dense network, that signed contribution is added to other incoming contributions before the next activation.
Why it matters: Changing a single weight changes one local influence, not an entire concept. Its effect depends on the source activation and on everything the downstream computation does with the result.
Scenario - A synthetic suppressing link: A source activation of 3 travels through a weight of -2, contributing -6 to the next unit's sum. Another input or bias may offset that contribution, so -6 is not necessarily the destination output.
Analogy - An audio mixer channel: A gain control scales a channel before channels are mixed. A signed weight can also reverse the direction of its contribution.
Where the analogy stops: A weight's magnitude alone is not a reliable measure of feature importance: source scale and interacting paths also matter.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/connection.webp
Picture description: An input cable, one isolated gain-control channel, and an output cable form a clear path through an audio mixer.
Picture caption: One connection scales an upstream signal before mixing. Weight magnitude alone is not importance, and sign reversal is a mathematical property, not ordinary gain.
What you see: Source activation. What it represents: The incoming audio signal stands for the source unit's current activation.
What you see: Connection weight. What it represents: The channel control represents scaling that activation along one connection.
What you see: Local contribution. What it represents: The outgoing channel contribution will combine with other inputs downstream.
Predict: What does a connection contribute when its source is zero, even if the weight is large?
Try: Change a connection weight and compare its contribution with the source activation and destination total.
Explain: The contribution is their product. A zero source gives zero contribution for any finite weight, and a negative weight reverses the sign of a nonzero source.
Optional math: c = w * a
c: contribution along this connection
w: connection weight
a: source unit activation
For w = -2 and a = 3, c = -6. The next unit may add other contributions and a bias before applying an activation.
Common misconception: A thicker or larger-weight connection is always more important.
Correction: Contribution also depends on activation size. Even a large contribution may be cancelled or clipped downstream; importance requires more than looking at a line.
Check your understanding: A source activation is 3 and the weight is -2. What is this connection's contribution?
-6 - Yes. Multiply the source by the signed weight; do not add them.
1 - 3 + (-2) adds the values, but a connection scales by multiplication.
6 - The negative weight makes this contribution negative.
Use it with purpose: Should you remove a connection solely because its weight is small?
Guidance: Measure the effect of removal on representative examples. Source scaling and alternative paths can change how much the connection matters; use controlled comparisons rather than a visual guess.
Where to go next:
Fine-tuning (concept): https://www.openagentschool.org/concepts/fine-tuning - Understand what changes when training updates selected model weights, and why a weight is not a standalone fact.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Hidden layer
URL: https://www.openagentschool.org/ai-ml-foundations/hidden-layer
Also called: intermediate layer, hidden representation
Summary: See intermediate features that help turn raw inputs into useful outputs.
In everyday words: A hidden layer is a stage between a network's input and final output. Its units transform the previous stage into an intermediate representation that later stages can use.
Why it matters: The useful work is often in these representations rather than the final output rule. They can combine features in ways a single linear decision cannot express, especially when nonlinear activations are used.
Scenario - A synthetic difference detector: Let x1 and x2 be two sensor values. Two hidden ReLU units compute max(0, x1 - x2) and max(0, x2 - x1). Adding them gives the absolute difference. Inputs [3, 1] produce hidden values [2, 0] and output 2.
Analogy - A workshop between receiving and shipping: The middle station rearranges raw materials into parts that make the final assembly easier.
Where the analogy stops: Unlike labeled workshop parts, hidden features need not have neat human names. Their meaning can be distributed or change during training.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/hidden-layer.webp
Picture description: Raw materials, partially assembled components on a central bench, and a finished organizer show three workshop stages.
Picture caption: A hidden layer builds intermediate representations between input and output. Hidden does not mean invisible, and learned features need not have tidy names.
What you see: Raw inputs. What it represents: The receiving materials stand for the original input features.
What you see: Intermediate features. What it represents: Partially assembled pieces picture a transformed representation used by later stages.
What you see: Final output. What it represents: The finished object represents what later computations produce from intermediate features.
Predict: Can two inputs activate different hidden units yet produce a similar final result?
Try: Change an input and inspect the intermediate activations before the output. Look for units switching on or off.
Explain: A nonlinear hidden representation can change which path contributes. In the difference-detector example, swapping the two inputs swaps hidden activity while preserving the final absolute difference.
Optional math: h = f(W x + b)
h: vector of hidden activations
f: activation function applied to each component
W: weights mapping inputs to hidden units
x: input vector
b: one bias per hidden unit
The next layer receives h rather than the original x directly in this simple feed-forward example. Nonlinearity is what makes this more than another affine mix.
Common misconception: Hidden means the values cannot be inspected.
Correction: Hidden means intermediate rather than an observed input or target. Software can expose these activations, although interpreting them remains difficult.
Check your understanding: In the difference detector, what hidden values do inputs [1, 3] produce?
[2, 2] - Only one signed difference is positive; the other is clipped by ReLU.
[0, 2] - Yes. max(0, 1 - 3) is 0 and max(0, 3 - 1) is 2.
[-2, 2] - Those are pre-activation differences; the first becomes zero after ReLU.
Use it with purpose: Your model misses a feature interaction. What would justify trying a hidden layer?
Guidance: Show a concrete pattern the linear baseline cannot represent, then evaluate whether a small nonlinear layer improves held-out results. Inspect errors rather than assigning a human meaning to every unit.
Where to go next:
What is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Connect intermediate representations to how language models transform input information across stages.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Network depth
URL: https://www.openagentschool.org/ai-ml-foundations/depth
Also called: deep network, number of layers
Summary: Understand sequential transformations and why extra stages need a reason to exist.
In everyday words: Depth counts successive transformation stages in a network. Conventions differ about counting the input or output, so comparisons should state exactly which layers are counted.
Why it matters: Composing stages can build complex transformations from simpler ones. It also creates longer paths for computation and gradients, increasing training and deployment demands.
Scenario - Two synthetic affine stages: One stage computes h = 2x + 1; the next computes y = 3h - 2. Together they give y = 6x + 1. Without a nonlinearity between them, the extra stage does not create a new family of curved functions.
Analogy - An assembly line: Several stations can progressively transform a part, with each station using the previous result.
Where the analogy stops: More stations do not guarantee a better product. Redundant linear stations can collapse into one, while poor training can make a deeper network worse.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/depth.webp
Picture description: Loose wooden rails progress through a clamped assembly station to a finished frame on one continuous work line.
Picture caption: Depth adds successive transformations, not a quality guarantee. Without nonlinear operations, stacked affine stages can collapse into one.
What you see: First transformation. What it represents: The first station changes the representation received by the next stage.
What you see: Use the prior result. What it represents: The middle station illustrates sequential composition rather than parallel capacity.
What you see: Later transformation. What it represents: The final stage depends on earlier outputs but is not automatically better because the line is longer.
Predict: Should adding another transformation always improve an output?
Try: Compare shallower and deeper computations and trace the intermediate values instead of judging only the final number.
Explain: Each stage transforms the preceding representation. A changed output demonstrates composition, not improved accuracy; that requires a target, data, and evaluation. Nonlinear stages can add representational power.
Optional math: y = f_3(f_2(f_1(x)))
x: input to this illustrated three-stage composition
f_1: first transformation
f_2: second transformation
f_3: third and final transformation
y: final output after applying the functions from inside to outside
This example has three transformation stages. For the two affine stages in the scenario, substitution gives 3 * (2x + 1) - 2 = 6x + 1.
Common misconception: Stacking linear layers alone creates nonlinear behavior.
Correction: A composition of affine layers is still affine. Nonlinear activations or other nonlinear operations are needed to break that limitation.
Check your understanding: What single rule equals h = 2x + 1 followed by y = 3h - 2?
y = 6x - 1 - The second stage scales the first bias too: 3 times 1 minus 2 is +1.
y = 5x - 1 - Composed scale factors multiply rather than add.
y = 6x + 1 - Yes. Substituting the first stage into the second reveals one equivalent affine rule.
Use it with purpose: A deeper model costs more but its output looks more complex. Is that enough to adopt it?
Guidance: Compare held-out task quality, latency, memory, and training stability under clear layer-count conventions. Complexity of an internal trace is not a user benefit by itself.
Where to go next:
Agent cost optimization (concept): https://www.openagentschool.org/concepts/agent-cost-optimization - Relate sequential model work to latency and evidence-based model selection.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Network width
URL: https://www.openagentschool.org/ai-ml-foundations/width
Also called: hidden size, units per layer
Summary: Compare more parallel features with the extra parameters and memory they require.
In everyday words: Width is the number of units in a layer. A wider hidden layer can represent more features at the same stage, but the number of connections and trainable parameters usually grows too.
Why it matters: Width is a capacity and resource choice, not a quality score. Counting parameters makes the cost concrete before you spend time training a larger model.
Scenario - A small synthetic dense layer: A layer with 3 input features and 4 output units needs 12 weights and 4 biases: 16 parameters. Widening it to 8 outputs needs 24 weights and 8 biases: 32 parameters, before counting the next layer.
Analogy - Parallel workbenches: Adding benches lets a stage build more kinds of parts at once rather than adding more sequential stages.
Where the analogy stops: Benches do not automatically learn distinct useful roles. Extra units can be redundant, poorly trained, or prone to fitting noise.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/width.webp
Picture description: A shared material trolley supplies side-by-side workbenches at one stage, with a common receiving shelf.
Picture caption: Width adds parallel units within a stage. More capacity costs connections and memory, but does not guarantee useful or distinct features.
What you see: Shared inputs. What it represents: The material supply stands for input features available to multiple units.
What you see: Parallel capacity. What it represents: Side-by-side benches picture more units at the same layer, not greater depth.
What you see: More connections. What it represents: Collecting the benches' outputs suggests additional downstream connections and resource cost.
Predict: If input size stays fixed and output width doubles, what happens to this dense layer's parameter count?
Try: Compare different layer widths and count incoming weights plus one bias for each output unit. Distinguish that layer count from the experiment's whole-network total.
Explain: For one dense layer with a fixed input size, the count doubles. The experiment also counts output-layer weights and its bias, so its whole-network total need not double. Widening a hidden stage also changes the next stage's number of input connections.
Optional math: P = (d + 1) * h
P: trainable parameter count for one fully connected layer with biases
d: number of input features
h: number of output units, the layer width
Each output has d weights and one bias. With d = 3 and h = 4, P = 4 * 4 = 16. This does not count other layers.
Common misconception: A wider layer always learns more useful concepts.
Correction: Extra capacity permits more representations but does not guarantee they are useful. Data, training, and validation determine whether the capacity helps.
Check your understanding: How many parameters are in a biased dense layer with 3 inputs and 4 outputs?
16 - Yes. There are 3 times 4 weights plus 4 biases.
12 - That counts the weights but omits one bias for each output unit.
7 - Adding input and output counts does not count the connections between them.
Use it with purpose: When should a width increase earn its additional memory cost?
Guidance: When controlled evaluation shows a worthwhile improvement on unseen examples within your resource budget. Record parameter counts and test whether a smaller model already meets the requirement.
Where to go next:
Edge agents (concept): https://www.openagentschool.org/concepts/edge-agent - Model width affects whether a model fits the memory and compute limits of an edge device.
References:
Dive into Deep Learning: Multilayer Perceptrons: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html

LESSON: Data augmentation
URL: https://www.openagentschool.org/ai-ml-foundations/data-augmentation
Also called: label-preserving transformation
Summary: Create useful training variations without changing what the example is supposed to mean.
In everyday words: Data augmentation creates modified training examples using transformations that should preserve the task's label, or update the label correctly. It encourages a model to handle expected variations rather than memorize one presentation.
Why it matters: Carefully chosen variations can improve robustness when real labeled examples are scarce. An inappropriate transformation can instead teach the model that important differences do not matter.
Scenario - Synthetic parcel photos: For a toy classifier distinguishing dented from intact parcels, a small brightness change may preserve the label. Cropping out the only dent does not. A visually plausible new image can still be an invalid labeled example.
Analogy - Practicing in different rooms: You practice the same skill with different lighting and backgrounds so you do not depend on one setting.
Where the analogy stops: Changing the task itself is not useful variation. Practicing a different answer under the original label teaches the wrong relationship.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/data-augmentation.webp
Picture description: Matching practice stations in adjacent rooms keep the task materials the same while lighting and backgrounds differ.
Picture caption: Practice the same task in varied settings. Augmentation helps only when the target remains valid; changing the answer is not useful variation.
What you see: Original setting. What it represents: One room represents the original presentation of a training example.
What you see: Useful variation. What it represents: Different lighting and background represent input changes that preserve the task.
What you see: Same task evidence. What it represents: Recognizable shared materials emphasize that the information needed for the label must remain.
Predict: Do rotating and translating the experiment's square change its edge lengths, and does that make the transformations safe for every labeling task?
Try: Rotate and shift the synthetic points. Compare the original and transformed shape, then consider whether an orientation-sensitive label would still be valid.
Explain: Rotation and translation preserve pairwise distances, so this square remains a square. That geometric guarantee is not a universal label guarantee: rotating a directional arrow or a digit can change its correct label. The point experiment demonstrates geometry, while the parcel scenario asks you to judge task evidence.
Optional math: x_aug = T(x); y_aug = y
x: original training example
T: chosen label-preserving transformation
x_aug: augmented example
y: original label
y_aug: label for the augmented example
The second equality is an assumption to verify, not a universal rule. For tasks such as object detection, geometric transformations also require correctly transforming the target coordinates.
Common misconception: Any realistic-looking augmentation is safe.
Correction: Realism is insufficient. The transformation must preserve task-relevant information and label validity; flipping a directional arrow can change its correct class.
Check your understanding: Which transformation is safest for the stated parcel-damage task?
Crop away the only visible dent and keep the damaged label. - That removes the evidence needed for the label and can create misleading training pairs.
Slightly change brightness while the damage remains clearly visible. - Yes, under that explicit visibility assumption. Still inspect examples and validate on real photos.
Keep the image but randomly change its label. - That introduces label noise rather than a task-preserving input variation.
Use it with purpose: How do you check an augmentation before using it across a dataset?
Guidance: Inspect representative original-and-transformed pairs, including edge cases. Apply training augmentation after splitting the data, and evaluate on untouched real examples so near-duplicates do not leak into the test set.
Where to go next:
Synthetic training data (concept): https://www.openagentschool.org/concepts/synthetic-training-data - Use label validity and real-data evaluation to decide whether generated variations actually help.
References:
Dive into Deep Learning: Image Augmentation: https://d2l.ai/chapter_computer-vision/image-augmentation.html

LESSON: Gradient
URL: https://www.openagentschool.org/ai-ml-foundations/gradient
Also called: partial derivatives, gradient descent direction
Summary: Read local sensitivity and choose a step that reduces error without overshooting.
In everyday words: The gradient is a list of partial derivatives: how a function changes locally when each input changes while the others are held fixed. For a differentiable loss, its negative gives a locally downhill direction in ordinary Euclidean coordinates.
Why it matters: Training uses loss gradients to update weights. The gradient supplies direction and sensitivity, while the learning rate controls step size; mixing them up obscures why an update helps or fails.
Scenario - A synthetic one-weight fit: Let loss be (w - 3)^2. At w = 1, its derivative is -4. A learning rate of 0.25 gives the update 1 - 0.25 * (-4) = 2, reducing loss from 4 to 1.
Analogy - A local slope report: A slope tells you which way the nearby ground rises. Going in the opposite direction can move downhill.
Where the analogy stops: The report is local, not a map of the entire landscape. A large step can cross the valley, and a zero gradient need not identify the best solution.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/gradient.webp
Picture description: A survey stake stands beside a short straightedge resting along descending ground, with a shallow valley and rising far side beyond.
Picture caption: The supported bar follows the local incline, not the whole landscape. A gradient is local information; an overly large downhill step can overshoot and raise loss.
What you see: Current point. What it represents: The stake stands for the current parameter location where the gradient is measured.
What you see: Local slope. What it represents: The short bar represents nearby sensitivity rather than knowledge of the entire loss surface.
What you see: Step-size trade-off. What it represents: The valley and rising far side show why moving farther can overshoot a minimum.
Predict: Can a step in the downhill direction still end with a larger loss?
Try: Compare the loss before and after an update while varying the starting point or step size.
Explain: The interactive curve is L(x) = (x - 1)^2: at x = 3 its derivative is 4. A learning rate of 1.5 moves to -3 and raises loss from 4 to 16. The separate worked scenario uses the same quadratic shape with its minimum shifted to 3; do not look for that minimum on the interactive curve.
Optional math: w_new = w - eta * grad L(w)
w: current parameter or parameter vector
w_new: updated parameter
eta: positive learning rate, controlling step size
L(w): loss at the current parameter
grad: gradient: one local derivative per parameter
The minus sign moves against increasing loss. In one dimension, the gradient is the ordinary derivative; for (w - 3)^2 it is 2 * (w - 3).
Common misconception: Following the negative gradient guarantees finding the global minimum.
Correction: Step size and the shape of the loss matter. General neural-network losses are not guaranteed to be convex, and local information does not certify a global optimum.
Check your understanding: At w = 1 with gradient -4 and learning rate 0.25, what is the next w?
0 - Subtracting a negative update increases w; check the two minus signs.
5 - That uses a learning rate of 1. The stated rate scales the gradient to -1.
2 - Yes. 1 - 0.25 * (-4) = 2, which is closer to this toy loss minimum at 3.
Use it with purpose: Training loss starts jumping after updates. What should you inspect before adding model capacity?
Guidance: Check learning rate, gradient scale, data scaling, and the actual before-and-after loss. A controlled smaller-step comparison can diagnose overshooting without changing the architecture.
Where to go next:
Fine-tuning (concept): https://www.openagentschool.org/concepts/fine-tuning - Understand how a training objective turns into parameter updates and why learning-rate choice matters.
References:
Boyd and Vandenberghe: Convex Optimization: https://web.stanford.edu/~boyd/cvxbook/

LESSON: Lagrange multiplier
URL: https://www.openagentschool.org/ai-ml-foundations/lagrange-multiplier
Also called: Lagrangian, equality-constrained optimization
Summary: Optimize under an explicit constraint instead of pretending every downhill move is allowed.
In everyday words: A Lagrange multiplier connects an objective with a constraint. For a smooth equality-constrained problem, it helps identify points where no feasible local direction can improve the objective, subject to suitable regularity conditions.
Why it matters: Real systems optimize within budgets and requirements. A lower loss is not acceptable if it violates the constraint; the multiplier exposes the local trade-off rather than treating the constraint as optional.
Scenario - A synthetic resource split: Choose x and y with x + y = 4 while minimizing x^2 + y^2. The split [2, 2] has objective 8; [1, 3] has objective 10. The unconstrained minimum [0, 0] is not allowed because it fails the required total.
Analogy - The best seat on a marked row: You want the best view, but must choose a seat on a particular row. The best seat in the entire theater may not be an allowed choice.
Where the analogy stops: A multiplier describes local sensitivity under assumptions. It is not automatically a market price, and the basic equality method is not the whole theory of inequality constraints.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/lagrange-multiplier.webp
Picture description: A textured ribbon marks one allowed theater row, with a candidate seat inside it and another appealing seat outside it.
Picture caption: Optimize among allowed choices, not the whole theater. A Lagrange multiplier describes a constrained trade-off, not automatically a ticket price.
What you see: Not a feasible choice. What it represents: A seat outside the marked row may look attractive but violates the constraint.
What you see: Allowed set. What it represents: The marked row stands for choices satisfying the equality constraint.
What you see: Best within limits. What it represents: The candidate seat illustrates optimizing only among feasible alternatives.
Predict: Can the unconstrained lowest point violate the resource requirement?
Try: Change the allocation or required total and compare objective value with constraint satisfaction. Judge only feasible candidates as solutions.
Explain: At the optimum, the objective gradient can be balanced by a multiple of the constraint gradient. For this symmetric quadratic example, splitting the required total equally is best; moving off the constraint is not a valid improvement.
Optional math: L(x, y, lambda) = x^2 + y^2 - lambda * (x + y - c)
L: Lagrangian combining objective and equality constraint
x: first allocated quantity
y: second allocated quantity
lambda: multiplier used to balance the constraint gradient
c: required total allocation
Stationarity gives 2x - lambda = 0 and 2y - lambda = 0; feasibility gives x + y = c. Thus x = y = c/2 and lambda = c with this minus-sign convention. At c = 4, the minimum objective is 8. Using a plus sign for the constraint term would negate the multiplier but leave the optimal allocation unchanged.
Common misconception: Lagrange multipliers are just generic gradient descent.
Correction: They address constrained optimization. A stationary Lagrangian candidate must satisfy the constraint, and general problems require additional checks to distinguish minima from other stationary points.
Check your understanding: Which point minimizes x^2 + y^2 while satisfying x + y = 4?
[2, 2] - Yes. It is feasible and has objective 8, the minimum for this constrained quadratic.
[0, 0] - Its objective is lower, but it violates the required total of 4 and is not a feasible solution.
[1, 3] - It satisfies the constraint, but its objective 10 is worse than the equal split's 8.
Use it with purpose: An agent plan is cheaper but exceeds a hard tool-use budget. Is it the better plan?
Guidance: First establish feasibility. Use constrained optimization when requirements are explicit; for inequality budgets, methods such as KKT conditions add conditions beyond this equality example. Do not silently replace hard requirements with a preference.
Where to go next:
Agent cost optimization (concept): https://www.openagentschool.org/concepts/agent-cost-optimization - Distinguish improving an objective from satisfying the resource conditions under which improvement counts.
Budget-constrained execution (pattern): https://www.openagentschool.org/patterns/budget-constrained-execution - Apply the feasibility-first idea to agent plans governed by explicit resource budgets.
References:
Boyd and Vandenberghe: Convex Optimization: https://web.stanford.edu/~boyd/cvxbook/

LESSON: Normal distribution
URL: https://www.openagentschool.org/ai-ml-foundations/normal-distribution
Also called: Gaussian distribution, bell curve
Summary: Separate the height of a bell curve from the probability of landing in a range.
In everyday words: A normal distribution is a continuous probability model with a symmetric bell-shaped density. Its mean locates the center and its positive standard deviation controls spread. Probabilities are areas under the curve, not its height at a point.
Why it matters: Normal models appear in noise assumptions, measurement uncertainty, and statistical methods. Knowing the assumptions keeps a convenient curve from becoming an unsupported claim about real data.
Scenario - Synthetic measurement noise: Assume a sensor's measurement error is normally distributed with mean 0 and standard deviation 2 millimeters. About 68.3% of errors then fall between -2 and 2 millimeters. That percentage follows from the assumed model, not from seeing a bell-shaped drawing.
Analogy - A smooth pile of sand: The center and spread resemble where sand is concentrated and how widely it is scattered. A wider pile with the same total amount has a lower peak.
Where the analogy stops: A density curve has total area 1, and real sensor errors may be skewed or contain outliers. A literal sand pile does not establish a probability law.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/normal-distribution.webp
Picture description: A narrow sand mound and a broader lower mound sit side by side on the same matte laboratory tray.
Picture caption: A wider pile suggests greater spread and a lower peak. Sand height is not probability, and real measurements need not follow a normal distribution.
What you see: Concentrated spread. What it represents: The narrower pile suggests values concentrated closer to a center.
What you see: Wider spread. What it represents: The lower, wider pile suggests spreading the same total mass more broadly.
What you see: Area, not height. What it represents: For a probability density, probability belongs to an interval's area rather than one peak height.
Predict: If spread increases while the mean stays fixed, what happens to the peak?
Try: Change the spread and compare the curve's center, width, and height. Consider a fixed interval around the center rather than just one point.
Explain: The bell widens and its peak falls so total area stays 1. A fixed-width interval around the mean contains less probability as standard deviation increases.
Optional math: f(x) = exp(-((x - mu)^2) / (2 * sigma^2)) / (sigma * sqrt(2 * pi))
f(x): probability density at measurement value x
x: possible measurement value
mu: mean, locating the center
sigma: strictly positive standard deviation
exp: exponential function with base e
sqrt: nonnegative square root
pi: the circle constant, approximately 3.14159
Density has inverse measurement units. A density value can exceed 1 when the spread is small, while every interval probability remains between 0 and 1. A single exact point has probability zero in this continuous model.
Common misconception: A curve height of 0.2 means a 20% chance of that exact value.
Correction: Height is density. Probability is area over an interval; exact points have zero probability in a continuous normal model.
Check your understanding: Under the stated normal-noise assumption, what represents the probability of an error between -2 and 2?
The curve height at 0 - That is the density at the center, not the probability across the interval.
The area under the curve from -2 to 2 - Yes. With mean 0 and standard deviation 2, this area is about 0.683.
The interval width, 4 - Width alone is not a probability and can exceed 1; it must be combined with the density through area.
Use it with purpose: Should you assume production request latency is normal because a bell curve is convenient?
Guidance: Inspect actual latency data, especially skew and long tails. A normal model allows negative values and may misrepresent slow requests; use a distribution and summaries suited to the measured quantity.
Where to go next:
Data visualization (concept): https://www.openagentschool.org/concepts/data-visualization - Label density and probability correctly when communicating uncertainty.
Agents for Science (science): https://www.openagentschool.org/agents-for-science - Carry explicit measurement-noise assumptions into scientific workflows instead of treating a fitted curve as evidence.
References:
Dive into Deep Learning: Random Variables: https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/random-variables.html

LESSON: Z-score
URL: https://www.openagentschool.org/ai-ml-foundations/z-score
Also called: standard score, standardization
Summary: Express a value in standard-deviation units without assuming it follows a bell curve.
In everyday words: A z-score subtracts a reference mean from a value and divides by the reference standard deviation. It tells you how far above or below that mean the value lies in units of spread.
Why it matters: Standardized values let you compare differently scaled features and monitor deviations from a baseline. They are meaningful only when the baseline population and its nonzero spread are clearly defined.
Scenario - A synthetic latency baseline: Suppose training-period latency has mean 100 milliseconds and standard deviation 20 milliseconds. A new value of 140 has z-score 2. It is two baseline standard deviations above the mean, not automatically a 2% or 95% event.
Analogy - Changing the ruler: Instead of measuring distance in milliseconds or centimeters, you measure it in the typical spread of a chosen reference group.
Where the analogy stops: Changing the ruler does not change the shape of the data, prove normality, or make comparisons fair when the reference groups differ.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/z-score.webp
Picture description: Two fixed measurement pegs are compared using unnumbered rulers with different tactile tick spacing.
Picture caption: A z-score changes the ruler to reference-spread units. It does not change the data shape, prove normality, or repair an unsuitable baseline.
What you see: Reference mean. What it represents: The reference peg stands for the baseline mean subtracted from the observation.
What you see: Observed value. What it represents: The second peg stands for the value whose distance from the reference is measured.
What you see: Spread-sized units. What it represents: The ruler spacing represents dividing that distance by a positive reference standard deviation.
Predict: For a value above the mean, what happens to its z-score when reference spread increases?
Try: Hold the value and mean fixed while changing the positive reference spread; then compare a value below the mean.
Explain: A larger denominator reduces the magnitude of the z-score. Values below the mean have negative z-scores; a reference standard deviation of zero makes this formula undefined.
Optional math: z = (x - mu) / sigma
z: standardized score, with no physical unit
x: value being compared
mu: mean of the stated reference population or fitted training baseline
sigma: positive standard deviation of that same reference
For the latency example, z = (140 - 100) / 20 = 2. Normal-tail probability interpretations need an additional normal-distribution assumption.
Common misconception: Standardizing data makes its distribution normal.
Correction: Subtracting a mean and dividing by spread only changes location and scale. Skew, multiple peaks, and heavy tails do not disappear.
Check your understanding: With reference mean 100 and standard deviation 20, what is the z-score of 140?
40 - 40 is the raw difference. Divide by the reference standard deviation to express it in spread units.
7 - 140 / 20 omits subtracting the reference mean.
2 - Yes. The value is 40 above the mean, or two standard deviations.
Use it with purpose: When preparing train and test features, which data should determine the scaling mean and spread?
Guidance: Fit those values on training data only, then reuse them for validation, test, and inference. Recomputing on the test set leaks information and changes the reference used for comparison.
Where to go next:
AI-ready data (concept): https://www.openagentschool.org/concepts/ai-ready-data - Make scaling assumptions reproducible and avoid leaking evaluation data into preprocessing.
References:
Dive into Deep Learning: Random Variables: https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/random-variables.html
scikit-learn: StandardScaler: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html

LESSON: Entropy
URL: https://www.openagentschool.org/ai-ml-foundations/entropy
Also called: Shannon entropy, categorical uncertainty
Summary: Measure uncertainty in a declared probability distribution, not whether an answer is true.
In everyday words: Shannon entropy measures the average surprise of outcomes drawn from a stated probability distribution. For a finite set of categories, it is largest when the categories are equally likely and zero when one outcome is certain.
Why it matters: Entropy can summarize how concentrated a model's choices are. It cannot tell you whether the probability model is accurate or whether a low-uncertainty answer is correct.
Scenario - A synthetic two-route choice: Assume exactly two mutually exclusive, exhaustive routes with probabilities [0.5, 0.5]. Their Shannon entropy is 1 bit. If the probabilities become [1, 0], entropy is 0 bits. The declared distribution must sum to 1.
Analogy - Guessing from a bag: A bag equally split between two colors is harder to predict than a bag with only one color. Entropy describes the average uncertainty before a draw.
Where the analogy stops: The result depends on the assumed contents and sampling rule. A confident but incorrectly described bag can have low modeled entropy.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/entropy.webp
Picture description: One open draw bag contains mixed round and square tokens; another shows one token type, with an empty draw cup between them.
Picture caption: A mixed bag suggests more uncertainty before a draw. Entropy depends on declared probabilities and sampling assumptions, not just visible contents.
What you see: Less predictable. What it represents: Several plausible token types suggest greater uncertainty under a balanced-outcome assumption.
What you see: More concentrated. What it represents: One dominant token type suggests a concentrated distribution with lower modeled entropy.
What you see: Before the draw. What it represents: Entropy describes expected uncertainty before observing the sampled outcome.
Predict: For two fixed categories, does moving probability toward an even split increase or decrease entropy?
Try: Redistribute probability between the categories while keeping the total at 1. Compare a balanced choice with a concentrated one.
Explain: A balanced binary distribution has the highest entropy, 1 bit. Concentrating probability lowers entropy because one outcome becomes easier to predict under that model.
Optional math: H(p) = -sum_i(p_i * log2(p_i))
H(p): Shannon entropy of the categorical distribution p, in bits
p_i: nonnegative probability of category i; all category probabilities sum to 1
sum_i: add over every category in the declared outcome set
log2: logarithm to base 2
Use the limiting convention 0 * log2(0) = 0. For two equal probabilities, each contributes 0.5 bits, totaling 1 bit. Other logarithm bases give different units.
Common misconception: Low entropy means the model knows the right answer.
Correction: Low entropy means the model distribution is concentrated. It can be confidently wrong, so correctness and calibration require external outcomes.
Check your understanding: Which binary probability distribution has the greatest entropy?
[0.5, 0.5] - Yes. Equal probabilities maximize uncertainty over these two outcomes, giving 1 bit.
[1, 0] - This distribution has zero entropy because its outcome is certain under the stated model.
[0.8, 0.8] - These entries sum to 1.6, so they are not a valid categorical probability distribution.
Use it with purpose: Could you use high entropy to send uncertain requests for review?
Guidance: Treat entropy as one candidate signal, then evaluate errors and review costs. Also sample low-entropy predictions because confidently wrong outputs will otherwise escape that rule.
Where to go next:
Uncertainty and calibration (concept): https://www.openagentschool.org/concepts/agent-uncertainty-calibration - Distinguish concentration of a model distribution from evidence that its certainty is warranted.
References:
Dive into Deep Learning: Information Theory: https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/information-theory.html

LESSON: KL divergence
URL: https://www.openagentschool.org/ai-ml-foundations/kl-divergence
Also called: Kullback-Leibler divergence, relative entropy
Summary: Compare two probability models with an explicit direction and careful handling of impossible events.
In everyday words: KL divergence measures how a proposed distribution q differs from a reference distribution p, averaging the log probability ratio under p. The order matters: KL(p || q) is generally different from KL(q || p).
Why it matters: It appears in distribution comparison and model-training objectives. Declaring which distribution is the reference prevents an apparently small mismatch from being interpreted in the wrong direction.
Scenario - A synthetic route forecast: Suppose two route outcomes have reference probabilities p = [0.5, 0.5], while a forecast assigns q = [0.75, 0.25]. KL(p || q) is about 0.144 nats. Reversing the roles gives about 0.131 nats; the disagreement is directional.
Analogy - Packing for the wrong weather mix: You pack according to one forecast but encounter weather from another. The cost of the mismatch depends on which weather actually occurs.
Where the analogy stops: This is not literal financial regret or a symmetric distance. KL uses normalized probabilities and logarithms, not arbitrary category scores.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/kl-divergence.webp
Picture description: A lightly packed travel case leaves rain gear outside while rain is visible through the hotel window.
Picture caption: Packing for the wrong weather mix pictures directional mismatch. KL compares probability distributions, not literal travel cost or one surprising day.
What you see: Proposed model. What it represents: The packed case represents preparation based on the proposed distribution q.
What you see: Underprepared outcome. What it represents: Unused rain gear suggests an outcome given too little probability by the proposed model.
What you see: Reference conditions. What it represents: Encountered conditions stand for the reference p that weights the directional comparison.
Predict: What happens when the forecast assigns almost no probability to an outcome the reference still considers possible?
Try: Keep both distributions normalized and reduce one forecast probability where the reference has positive mass. Compare the forward and reversed divergence.
Explain: The forward divergence grows as q approaches zero on a possible p outcome. If q is exactly zero there, KL(p || q) is infinite. Matching distributions give zero.
Optional math: KL(p || q) = sum_i(p_i * ln(p_i / q_i))
KL(p || q): divergence from reference p to proposed q, in nats
p_i: reference probability of outcome i
q_i: proposed probability of the same outcome i
sum_i: sum over the shared finite outcome set
ln: natural logarithm, using base e
Both p and q must be nonnegative and each must sum to 1. Terms with p_i = 0 contribute zero by convention; p_i > 0 with q_i = 0 gives infinity. Individual terms can be negative, but the total KL is nonnegative.
Common misconception: KL is a symmetric distance, so order does not matter.
Correction: The reference distribution weights the average. Swapping p and q changes those weights and ratios, so KL is not a symmetric distance metric.
Check your understanding: What is KL(p || q) if p gives positive probability to an outcome that q assigns zero?
0 - Zero divergence requires matching distributions; declaring a possible reference event impossible is the opposite.
Infinite - Yes. The log probability ratio diverges for that positive-reference-probability outcome.
A negative value - A valid KL total is nonnegative. The issue here is a zero denominator, not a negative distance.
Use it with purpose: Before adding a KL penalty to training, what must be written down?
Guidance: Specify reference and proposed distributions, direction, outcome set, logarithm units, and any smoothing. Smoothing changes the compared distributions; do not present a clipped finite value as the exact divergence of a zero-support case.
Where to go next:
LLM post-training (concept): https://www.openagentschool.org/concepts/llm-post-training - Read distribution-divergence penalties as explicit constraints on behavior change rather than generic similarity scores.
References:
Dive into Deep Learning: Information Theory: https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/information-theory.html

LESSON: Eigenvectors
URL: https://www.openagentschool.org/ai-ml-foundations/eigenvectors
Also called: eigenvalues, eigendirections
Summary: Find directions that a square transformation scales without pushing them off their line.
In everyday words: An eigenvector is a nonzero vector that a square matrix transforms into a scalar multiple of itself. The scalar is its eigenvalue: it tells you the amount of stretching, shrinking, reversal, or collapse along that direction.
Why it matters: Eigenvectors help describe repeated transformations and variation in data. For a covariance matrix, they identify directions of variation; that special meaning does not apply to every matrix.
Scenario - A synthetic two-axis stretch: The matrix with rows [2, 0] and [0, 1] doubles horizontal components and leaves vertical ones unchanged. Vector [1, 0] is an eigenvector with eigenvalue 2. Vector [1, 1] becomes [2, 1], which is not a multiple of the original.
Analogy - Lines drawn on stretching fabric: Some lines remain on their original direction while the fabric stretches along them. Other lines tilt because their components stretch by different amounts.
Where the analogy stops: General matrices can rotate, reverse, or collapse directions; some real matrices have no real eigenvectors. The fabric picture is only a simple two-dimensional case.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/eigenvectors.webp
Picture description: A square reference swatch and a horizontally stretched rectangular swatch retain a vertical stitched line while the matching diagonal becomes shallower.
Picture caption: Unequal stretching preserves an aligned direction while tilting a diagonal. This is illustrative, not a measured matrix; general matrices can rotate, reverse, or collapse.
What you see: Before transformation. What it represents: The reference swatch establishes the original orientation of the stitched directions.
What you see: Preserved direction. What it represents: An aligned stitched line suggests an eigenvector scaled along its own line.
What you see: Direction can change. What it represents: A diagonal line can tilt when its components stretch by different amounts.
Predict: Which directions keep the experiment's input and transformed vector on the same line?
Try: Rotate the input vector and compare it with the output vector. Look for zero direction mismatch rather than assuming the coordinate axes are special.
Explain: The experiment uses matrix [2, 1; 1, 2], different from the diagonal worked example. Directions at 45 and 135 degrees are scaled by 3 and 1 respectively. Its mismatch readout is zero when input and output are parallel; away from those directions, the reported projected scale is not an eigenvalue.
Optional math: A v = lambda * v
A: square transformation matrix
v: nonzero eigenvector
lambda: eigenvalue giving the scale factor on that vector
A negative eigenvalue reverses direction along the same line. An eigenvalue of zero maps a nonzero eigenvector to zero; the zero vector itself is not an eigenvector.
Common misconception: Every vector is an eigenvector of a matrix.
Correction: Only nonzero vectors satisfying the scalar-multiple relation qualify. For a general matrix, most directions do not; a scalar multiple of the identity is a special exception.
Check your understanding: For the matrix with rows [2, 0] and [0, 1], which statement is correct?
[1, 1] has eigenvalue 2. - Its output [2, 1] is not 2 * [1, 1], so it does not satisfy the required relation.
[0, 0] is an eigenvector. - Eigenvectors are explicitly nonzero; otherwise every scalar would trivially fit.
[1, 0] has eigenvalue 2. - Yes. The matrix maps it to [2, 0], exactly twice the input vector.
Use it with purpose: A data analysis calls an eigenvector the most important direction. What should you ask?
Guidance: Ask which matrix was decomposed and what its eigenvalues measure. For covariance-based analysis, feature scaling affects the answer; high variance does not automatically mean high task relevance.
Where to go next:
Data visualization (concept): https://www.openagentschool.org/concepts/data-visualization - Interpret principal directions in a data projection without mistaking variance for usefulness.
References:
Dive into Deep Learning: Linear Algebra: https://d2l.ai/chapter_preliminaries/linear-algebra.html
NumPy: Eigenvalues and Eigenvectors: https://numpy.org/doc/stable/reference/generated/numpy.linalg.eig.html

LESSON: Singular value decomposition
URL: https://www.openagentschool.org/ai-ml-foundations/svd
Also called: SVD, low-rank approximation
Summary: Break a matrix into directions and strengths, then see what a smaller approximation discards.
In everyday words: Singular value decomposition factors a matrix into input directions, nonnegative strengths, and output directions. Unlike ordinary eigenvector decomposition, it works for rectangular matrices too.
Why it matters: Keeping only the strongest singular components produces a low-rank approximation that can reduce storage or computation. The approximation minimizes a mathematical reconstruction error, not necessarily the errors users care most about.
Scenario - A synthetic two-feature table: A matrix with rows [4, 0] and [0, 1] has singular values 4 and 1. Keeping only the strongest component reconstructs [4, 0; 0, 0]. One squared unit of entrywise reconstruction error remains because the bottom-right 1 was discarded.
Analogy - Rebuilding a chord from strong notes: A complex sound can be approximated with a few dominant components. Keeping more components restores detail.
Where the analogy stops: Small components can contain important task information. SVD components are mathematical directions, not automatically meaningful human features or literal musical notes.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/svd.webp
Picture description: Matching chime instruments contrast a complete set of available bars with selected free bars and others muted by felt dampers.
Picture caption: Keeping selected notes pictures a lower-detail reconstruction. SVD components are mathematical directions, and small discarded components may still matter.
What you see: Complete structure. What it represents: The full instrument represents information available before approximation.
What you see: Retained components. What it represents: Unmuted bars stand for the components kept in a truncated representation.
What you see: Discarded detail. What it represents: Damped bars make omitted information visible rather than pretending compression is lossless.
Predict: When you retain more singular components, can exact-arithmetic reconstruction error increase?
Try: Compare a lower-rank reconstruction with one retaining another component. Track the entries changed and the reconstruction error.
Explain: For the standard truncated SVD and the same matrix, retaining more components cannot increase squared entrywise reconstruction error. It can cost more storage, and lower reconstruction error does not by itself prove better downstream predictions.
Optional math: A = U S V^T
A: real-valued matrix to decompose
U: matrix whose columns are orthonormal output directions
S: diagonal matrix of nonnegative singular values, conventionally ordered largest first
V: matrix whose columns are orthonormal input directions
T: transpose: swap rows and columns
Orthonormal means the directions have unit length and are perpendicular. Truncated SVD keeps a chosen number of leading components; the sum of squares of discarded singular values is its squared entrywise reconstruction error.
Common misconception: Dropping small singular values is always harmless.
Correction: It loses information. Small-variance details can still be crucial for a rare class, a safety condition, or a downstream task.
Check your understanding: For singular values 4 and 1, what squared reconstruction error remains after keeping only the first?
1 - Yes. The discarded singular value is 1, whose square is 1.
0 - Exact reconstruction would require retaining the second nonzero component too.
16 - That squares the retained component. Error comes from what was discarded.
Use it with purpose: A low-rank model is smaller and reconstructs weights well. Is deployment quality established?
Guidance: No. Evaluate the compressed model on representative and rare cases, alongside memory and latency. A matrix-level error guarantee is useful evidence, not a task-level quality guarantee.
Where to go next:
Fine-tuning (concept): https://www.openagentschool.org/concepts/fine-tuning - Build intuition for low-rank structure while keeping matrix compression distinct from learning a low-rank update.
Agent cost optimization (concept): https://www.openagentschool.org/concepts/agent-cost-optimization - Test whether smaller numerical representations preserve the behaviors worth paying for.
References:
NumPy: Singular Value Decomposition: https://numpy.org/doc/stable/reference/generated/numpy.linalg.svd.html

LESSON: Ordinary least squares
URL: https://www.openagentschool.org/ai-ml-foundations/ols
Also called: OLS, least-squares regression
Summary: Fit a simple numeric baseline by minimizing the sum of squared prediction errors.
In everyday words: Ordinary least squares chooses linear-model coefficients to minimize the sum of squared differences between observed targets and predictions. In a one-feature model, it selects the slope and intercept of a fitted line.
Why it matters: A transparent baseline can expose whether a more complex model earns its cost. Squared errors emphasize large misses, so fitting well on average may still leave important patterns in the residuals.
Scenario - Synthetic preparation time: Two observed pairs are (items = 0, minutes = 1) and (items = 1, minutes = 3). The line prediction = 1 + 2 * items fits both exactly. Its prediction of 5 minutes at 2 items is extrapolation beyond the observed inputs, not a guaranteed outcome.
Analogy - Positioning a straight ruler: You place a ruler through a scatter of points so that vertical gaps are small overall, with large gaps penalized more strongly by squaring.
Where the analogy stops: The fit measures vertical prediction errors, not perpendicular distances. A straight ruler also cannot capture every curved or changing relationship.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/ols.webp
Picture description: Brass observation pegs surround a slanted unnumbered ruler, with vertical threads showing gaps between points and the candidate line.
Picture caption: OLS chooses a line using squared vertical prediction gaps. A photographic ruler placement is illustrative, not proof of an optimal or causal fit.
What you see: Observed targets. What it represents: The pegs stand for observed input-target pairs rather than predictions.
What you see: Candidate line. What it represents: The ruler stands for a linear prediction rule whose slope and intercept can be fitted.
What you see: Vertical residuals. What it represents: Vertical threads picture target-minus-prediction gaps, not perpendicular distances to the line.
Predict: If the last observed target rises while the other observations stay fixed, will the fitted slope remain unchanged?
Try: Change the last observed target and inspect the recomputed slope, intercept, fitted line, and mean squared error.
Explain: In this experiment the final input is above the input mean, so raising its target raises the fitted slope. OLS refits both coefficients to minimize squared error on the changed dataset. Other residuals may grow as the line adjusts; this is not evidence that a fixed model improved on the same evaluation data.
Optional math: SSE = sum_i((y_i - (b + w * x_i))^2)
SSE: sum of squared errors to minimize over b and w
x_i: input value for observed example i
y_i: observed numeric target for example i
b: intercept, the prediction when input is zero
w: slope, the predicted change per input unit
sum_i: add over all fitted examples
A residual is observed value minus prediction. Minimizing this sum and minimizing its mean choose the same coefficients for a fixed nonempty dataset. A unique coefficient solution requires sufficient independent feature information.
Common misconception: A perfect fitted line proves the input causes the target.
Correction: A fit describes an association in the observed data. Causal interpretation needs additional design and assumptions; perfect fit on two points says little about generalization.
Check your understanding: What does the fitted line 1 + 2 * items predict for 2 items?
4 minutes - That counts 2 * 2 but omits the intercept. Add the fixed 1 minute as well.
5 minutes, an extrapolation in this example - Yes. The arithmetic gives 5, but the observed inputs only covered 0 and 1.
Exactly 5 minutes for every future case - A model prediction is not a guarantee. This tiny dataset cannot establish future timing or causation.
Use it with purpose: Before replacing this baseline with a neural network, what evidence would justify the change?
Guidance: Inspect residual patterns and held-out error. Consider nonlinear structure, data quantity, and operational cost; keep the baseline as a comparison rather than dismissing it because it is simple.
Where to go next:
Verified analytical agents (concept): https://www.openagentschool.org/concepts/verified-analytical-agents - Require explicit numeric objectives and inspectable evidence behind an agent's regression claims.
References:
scikit-learn: Ordinary Least Squares: https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares

LESSON: Naive Bayes
URL: https://www.openagentschool.org/ai-ml-foundations/naive-bayes
Also called: conditional independence classifier
Summary: Combine a prior with feature evidence while making the independence assumption visible.
In everyday words: Naive Bayes is a family of classifiers that applies Bayes' rule while assuming features are conditionally independent given the class. It combines a class prior with the likelihood of observed features under that class.
Why it matters: It provides a fast, understandable baseline for tasks such as text classification. Its simplifying assumptions can still allow useful classification, but the resulting probabilities are not automatically reliable.
Scenario - A synthetic keyword detector: Assume spam prior 0.2 and non-spam prior 0.8. A keyword appears with probability 0.6 in spam and 0.1 in non-spam. For a message containing that keyword, unnormalized scores are 0.12 and 0.08; normalizing gives spam probability 0.6 under this toy model.
Analogy - Combining separate clues: A starting belief is adjusted by how compatible each clue is with each explanation.
Where the analogy stops: Clues can overlap. Treating two nearly identical clues as independent can count the same evidence twice and make the model too certain.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/naive-bayes.webp
Picture description: A specimen desk combines baseline materials, a leaf under a magnifier, and overlapping views of the same leaf feature.
Picture caption: Clues update a starting belief. Naive Bayes assumes independence given the class; overlapping clues can count the same evidence twice.
What you see: Starting belief. What it represents: Background specimen context stands for a class prior before the current clues.
What you see: Observed clue. What it represents: A visible feature represents evidence whose likelihood differs between candidate classes.
What you see: Overlapping evidence. What it represents: Repeated views of one feature warn against treating related clues as independent support.
Predict: With the keyword likelihoods held fixed, what happens if the spam prior increases?
Try: Compare prior-weighted class scores and then the normalized posterior. Keep the competing class prior consistent so priors sum to 1.
Explain: A higher spam prior raises the spam posterior for the same evidence. With multiple features, the naive model multiplies conditional likelihoods, making the conditional independence assumption important.
Optional math: P(c | x) = score_c / sum_k(score_k); score_c = P(c) * product_j(P(x_j | c))
P(c | x): model probability of class c after observing feature vector x
score_c: unnormalized prior-times-likelihood score for class c
P(c): class prior before observing these features
P(x_j | c): probability of observed feature value x_j within class c
product_j: multiply likelihoods over features j under the conditional independence assumption
sum_k: add scores over all candidate classes k for normalization
For the single-keyword example, 0.12 / (0.12 + 0.08) = 0.6. All class priors sum to 1. Zero likelihoods need an explicit modeling or smoothing policy, not silent division by zero.
Common misconception: Naive Bayes assumes features are independent in the whole dataset.
Correction: The assumption is conditional independence given the class. Even that assumption often fails in real text, where words and phrases can be strongly related.
Check your understanding: With class scores 0.12 for spam and 0.08 for non-spam, what is the normalized spam probability?
0.12 - That is an unnormalized score. Divide by the total score of 0.20.
0.2 - That is the prior in this scenario, before incorporating the keyword evidence.
0.6 - Yes. 0.12 / (0.12 + 0.08) = 0.6 under the stated model.
Use it with purpose: Would adding several near-duplicate keyword features necessarily improve this classifier?
Guidance: No. They may repeatedly count related evidence under an independence assumption. Compare held-out classification and calibration, and choose a feature representation and smoothing policy deliberately.
Where to go next:
Model routing (concept): https://www.openagentschool.org/concepts/model-routing - Use simple evidence-based classification as a routing baseline before adding a more expensive model.
References:
scikit-learn: Naive Bayes: https://scikit-learn.org/stable/modules/naive_bayes.html

LESSON: Support vector machine
URL: https://www.openagentschool.org/ai-ml-foundations/svm
Also called: SVM, maximum-margin classifier
Summary: Separate classes with a margin and distinguish boundary scores from probabilities.
In everyday words: A support vector machine is a model family that balances a wide separating margin with penalties for margin violations. In a linear binary classifier, the boundary is where a weighted score equals zero.
Why it matters: Margin-based learning offers a strong classical baseline for some datasets. Feature scaling and the penalty for violations shape the boundary, so a clean diagram is not enough to choose a model.
Scenario - Two synthetic sensor classes: In a one-dimensional toy dataset, value 0 is class -1 and value 2 is class +1. Score x - 1 places the boundary at 1 and gives scores -1 and +1 to the examples. With canonical margin scaling, they sit on the two margin edges.
Analogy - A road between two groups: The boundary is the centerline, and the margin is the clearance on each side. You seek separation with useful space around the boundary.
Where the analogy stops: Real classes may overlap, and a soft-margin model allows violations. The road width depends on feature units and does not establish prediction confidence.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/svm.webp
Picture description: A quiet service road separates rounded barrels from rectangular crates, with visible clearance between the road and the nearest objects.
Picture caption: A wide road pictures a separating margin. Real classes can overlap; margin scores are not probabilities, and feature units affect geometry.
What you see: One class. What it represents: One object type stands for observations assigned to one class.
What you see: Separating boundary. What it represents: The center seam represents a decision boundary between classes.
What you see: Margin clearance. What it represents: The nearest objects emphasize clearance around the boundary rather than correct-side classification alone.
Predict: Can a point be classified correctly yet still incur a margin penalty?
Try: Move the boundary or inspect a point closer to it. Compare the sign of its score with its distance from the margin target.
Explain: A positive-class point with score 0.2 is on the correct side but inside the desired margin. Hinge loss still penalizes it because signed margin 0.2 is below 1.
Optional math: hinge = max(0, 1 - y * s); s = w dot x + b
hinge: per-example hinge loss, one part of the soft-margin training objective
max: choose the larger argument
y: true class label, either -1 or +1
s: real-valued decision score
w: weight vector
x: input feature vector
dot: dot product: add matching weight-feature products
b: boundary offset
For y = +1 and s = 0.2, hinge loss is 0.8. SVM training also penalizes weight magnitude; hinge loss alone is not the complete objective. The score is not a calibrated probability.
Common misconception: Only misclassified points receive hinge loss.
Correction: Correctly classified points inside the margin also receive positive hinge loss. Classification sign and margin satisfaction are different checks.
Check your understanding: For true label +1 and score 0.2, what is hinge loss?
0.8 - Yes. max(0, 1 - 1 * 0.2) = 0.8, even though the sign is correct.
0 because the prediction has the correct sign - Correct classification is not sufficient; the signed score must reach 1 to avoid hinge loss.
0.2 - That is the decision score. The margin shortfall is 1 - 0.2.
Use it with purpose: One feature is measured in dollars and another in fractions. What should you address before fitting a linear SVM?
Guidance: Fit an appropriate scaling transformation on training data and reuse it consistently. Tune violation penalties with validation, and do not treat raw decision scores as calibrated probabilities.
Where to go next:
AI problem framing (concept): https://www.openagentschool.org/concepts/ai-problem-framing - Choose a measurable classification objective and a sensible baseline rather than assuming every task needs an LLM.
References:
scikit-learn: Support Vector Machines: https://scikit-learn.org/stable/modules/svm.html

LESSON: K-means
URL: https://www.openagentschool.org/ai-ml-foundations/k-means
Also called: centroid clustering
Summary: Group nearby examples around learned centers without pretending the groups are ground-truth labels.
In everyday words: K-means partitions points into a chosen number of groups by minimizing squared distance to each group's center. A common algorithm alternates assigning points to their nearest center and moving centers to the means of assigned points.
Why it matters: It can organize unlabeled examples for exploration or review. The result reflects feature scaling, the chosen number of groups, and initialization; it does not discover an unquestionable taxonomy.
Scenario - Four synthetic workloads: For one-dimensional values [0, 2, 8, 10] and two groups, centers 1 and 9 give assignments [0, 2] and [8, 10]. Squared distances are 1, 1, 1, and 1, totaling 4.
Analogy - Placing meeting points: Assign each person to a nearby meeting point, then move each meeting point toward the average position of its assigned people.
Where the analogy stops: Euclidean averages may be unsuitable for roads, unusual shapes, mixed data types, or very unequal group sizes. Convenience of a center does not give the group a meaningful name.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/k-means.webp
Picture description: Two loose groups of courtyard chairs surround movable meeting markers, with open space for repositioning a marker.
Picture caption: Assign nearby people to meeting points, then reconsider the centers. Euclidean convenience does not establish true categories or real travel costs.
What you see: Nearby assignments. What it represents: Chairs near one marker stand for points assigned to their nearest center.
What you see: Another center. What it represents: The second marker represents a separate group center, not a ground-truth label.
What you see: Move toward the mean. What it represents: A movable marker suggests updating a center toward the average of its assigned positions.
Predict: If a center moves toward its assigned points' mean, what happens to their total squared distance?
Try: Compare point assignments and center locations, then inspect within-group squared distance as a center or grouping changes.
Explain: For fixed nonempty assignments, the mean minimizes the sum of squared distances. Alternating assignment and center updates does not increase the objective, but different starts can settle in different local solutions.
Optional math: J = sum_i(||x_i - mu_c(i)||^2)
J: within-cluster sum of squared distances, often called inertia
x_i: feature vector for point i
c(i): cluster assigned to point i among the chosen k groups
mu_c(i): center of that assigned cluster
||...||: Euclidean length: square root of the sum of squared components
sum_i: add over all points
The square outside the length gives squared Euclidean distance. In the example, each point is one unit from its center, so J = 4. k is the requested number of clusters, not a learned class label.
Common misconception: Cluster 0 and cluster 1 are discovered true class labels.
Correction: Cluster numbers are arbitrary identifiers. You must inspect examples and task usefulness before attaching semantic labels or making decisions about people.
Check your understanding: Which center minimizes squared distance for the assigned points 0 and 2?
0 - Its squared distances sum to 0 + 4 = 4, larger than the sum at the mean.
1 - Yes. The mean is 1, giving squared distances 1 + 1 = 2.
2 - This gives 4 + 0 = 4; placing the center at the mean reduces the sum.
Use it with purpose: A cluster plot looks clean. Can you use the clusters as automatic business decisions?
Guidance: Inspect examples, scale features deliberately, test stability across starts, and assess usefulness for the specific task. Clustering alone supplies neither ground-truth labels nor a justification for consequential decisions.
Where to go next:
Data and knowledge operations (concept): https://www.openagentschool.org/concepts/data-knowledge-operations - Use clustering to organize review work while preserving provenance and human interpretation of the groups.
References:
scikit-learn: K-means: https://scikit-learn.org/stable/modules/clustering.html#k-means

LESSON: Mean squared error
URL: https://www.openagentschool.org/ai-ml-foundations/mse
Also called: MSE, squared-error loss
Summary: Square every prediction error and take the actual mean over the dataset.
In everyday words: Mean squared error averages the squared differences between numeric predictions and observed targets. Squaring removes the sign and gives larger misses disproportionately more weight.
Why it matters: MSE is a common regression objective and evaluation metric. It summarizes error magnitude, but its squared units and sensitivity to outliers affect how you should explain and use it.
Scenario - Two synthetic wait estimates: Observed waits are [2, 4] minutes and predictions are [1, 6] minutes. Errors are 1 and -2; squared errors are 1 and 4. MSE is (1 + 4) / 2 = 2.5 square minutes.
Analogy - Paying extra for large misses: Imagine a penalty that grows with the square of a miss: doubling a miss quadruples its contribution.
Where the analogy stops: This penalty is a modeling choice, not necessarily the real cost of waiting. It treats equal-sized over- and under-predictions alike.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/mse.webp
Picture description: Near and farther misses on a tabletop target demonstration sit beside dishes suggesting different penalties.
Picture caption: MSE makes large misses disproportionately costly and averages their squared errors. This chosen penalty need not match real-world costs.
What you see: Smaller miss. What it represents: A short gap represents a smaller prediction residual before squaring.
What you see: Larger miss. What it represents: A larger gap represents a residual that receives disproportionately more squared-error weight.
What you see: Average the penalties. What it represents: The dishes stand for per-example penalties that must all contribute to the dataset mean.
Predict: What happens to MSE if every prediction error doubles while the number of examples stays fixed?
Try: Change predictions and inspect each squared residual plus their mean. Compare the effect of one large miss with several small ones.
Explain: Doubling all errors multiplies every squared term by four, so the mean also quadruples. A dataset metric must include every example in the declared evaluation set.
Optional math: MSE = sum_i((y_i - yhat_i)^2) / n
MSE: mean squared error over this nonempty dataset
y_i: observed numeric target for example i
yhat_i: model prediction for the same example
n: positive number of evaluated examples
sum_i: add across all n examples
For the scenario, MSE = 2.5, not the sum 5 and not the first squared error 1. Root mean squared error takes the square root to return to the original target units.
Common misconception: Squaring one example's error gives the dataset MSE.
Correction: That is one squared error. Dataset MSE requires the actual mean of squared residuals across all evaluated examples, with any weighting explicitly stated.
Check your understanding: For observed [2, 4] and predicted [1, 6], what is MSE?
5 - 5 is the sum of squared errors. Divide by the two evaluated examples.
1 - That includes only the first squared error and ignores the second example.
2.5 - Yes. Average both squared errors: ((2 - 1)^2 + (4 - 6)^2) / 2 = 2.5.
Use it with purpose: If underestimating a wait is much worse than overestimating it, is MSE enough?
Guidance: Report it if useful, but add a metric or loss aligned with that asymmetry and inspect residuals. Compare models on the same held-out examples and units so the means are meaningful.
Where to go next:
Agent evaluation methodologies (concept): https://www.openagentschool.org/concepts/agent-evaluation-methodologies - Make metric aggregation and error costs explicit when judging numeric agent outputs.
Observability & EvalOps (skill): https://www.openagentschool.org/ai-skills#module/observability - Carry per-example errors and correctly aggregated metrics into an evaluation workflow.
References:
scikit-learn: Model Evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

LESSON: Log loss
URL: https://www.openagentschool.org/ai-ml-foundations/log-loss
Also called: cross-entropy loss, negative log likelihood
Summary: Score the probability assigned to what actually happened, not just the final yes/no choice.
In everyday words: Log loss penalizes a probability forecast according to how little probability it assigned to the observed outcome. Binary log loss evaluates two-outcome forecasts and averages the penalty over examples.
Why it matters: Two classifiers can make the same thresholded choices while expressing very different certainty. Log loss rewards probability quality and strongly penalizes assigning tiny probability to an outcome that occurs.
Scenario - A synthetic failed forecast: A ticket really needs escalation, so its label is 1. A model assigns escalation probability 0.25. Its natural-log loss is -ln(0.25), about 1.386 nats. A probability of 0.75 for that same observed outcome would give about 0.288 nats.
Analogy - A forecast scored after the event: A forecast is judged not just by its favorite outcome but by how much probability it reserved for what happened.
Where the analogy stops: A single surprising event does not prove the forecaster is poor. Probability quality is assessed across suitable examples, not by demanding certainty after every outcome.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/log-loss.webp
Picture description: A desk holds a physical weather-forecast arrangement and blank review notebook while rain is visible outside the window.
Picture caption: Log loss scores the probability reserved for the observed event. One surprise does not prove a forecast is poor; evaluate many appropriate examples.
What you see: Earlier forecast. What it represents: The prepared arrangement stands for probabilities assigned before the outcome was known.
What you see: Observed event. What it represents: Rain identifies which outcome probability should be scored after the event.
What you see: Score the forecast. What it represents: The review notebook represents penalizing low probability for the outcome that occurred.
Predict: For a positive label, what happens as the predicted positive probability approaches zero?
Try: Keep the observed label fixed and change the predicted probability. Compare a modest error with a confident wrong forecast.
Explain: The loss grows without bound as probability of the observed event approaches zero. For a negative label, the complementary probability 1 - p is the one that matters.
Optional math: L = -sum_i(y_i * ln(p_i) + (1 - y_i) * ln(1 - p_i)) / n
L: mean binary log loss, in nats
y_i: observed label 0 or 1 for example i
p_i: predicted probability of label 1 for that example
ln: natural logarithm, base e
n: positive number of examples
sum_i: sum over all n examples
Use 0 < p_i < 1 for finite direct calculation. At endpoints use limiting conventions: probability zero for the observed event means infinite loss. Software clipping changes those endpoints and should be disclosed. Binary probabilities p_i and 1 - p_i sum to 1.
Common misconception: A correct thresholded label always has zero log loss.
Correction: A probability of 0.6 for the observed positive class gives a correct decision at threshold 0.5 but still positive loss. Zero is reached only when the observed event receives probability 1 under the limiting definition.
Check your understanding: For an observed positive outcome, which forecast has smaller log loss?
p = 0.75 - Yes. It assigns more probability to the event that occurred, giving loss about 0.288 rather than 1.386.
p = 0.25 - This gives less probability to the observed event and therefore a larger negative-log penalty.
They are equal because both are valid probabilities. - Validity only checks range and normalization. Log loss distinguishes how much probability was assigned to the actual outcome.
Use it with purpose: A model improves accuracy but worsens log loss. What should you investigate?
Guidance: Inspect confident mistakes and probability calibration on the same examples. Choose metrics that reflect whether you need hard decisions, useful probability estimates, or both; do not hide one change behind another.
Where to go next:
Uncertainty and calibration (concept): https://www.openagentschool.org/concepts/agent-uncertainty-calibration - Evaluate probability forecasts instead of equating a normalized output with reliable confidence.
Agent Evals & Benchmarking Lab (course): https://www.openagentschool.org/courses/agent-evals-benchmarking-lab - Practice comparing metrics that reward different aspects of a model's behavior.
References:
scikit-learn: Model Evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

LESSON: R-squared
URL: https://www.openagentschool.org/ai-ml-foundations/r-squared
Also called: R2, coefficient of determination
Summary: Compare squared prediction error with a mean-target baseline, including scores below zero.
In everyday words: R-squared compares a model's sum of squared prediction errors with the squared error of predicting the evaluation targets' mean. A value of 1 is perfect, 0 matches that baseline, and a negative value is worse than the baseline.
Why it matters: A model can produce plausible numbers yet lose to a simple constant prediction. R-squared makes that comparison explicit, but it does not measure causality or guarantee useful absolute error.
Scenario - A synthetic worse-than-baseline model: Observed targets are [1, 3], with mean 2. Constant predictions [4, 4] have squared error sum 9 + 1 = 10. The mean baseline [2, 2] has squared error sum 1 + 1 = 2. R-squared is 1 - 10/2 = -4.
Analogy - Competing against a simple benchmark: A sophisticated forecast must beat a contestant who always says the average. A negative score means it lost by this squared-error comparison.
Where the analogy stops: The benchmark uses the evaluation targets' mean as a definition of the metric, not a deployed forecasting strategy. Other operational baselines may be harder or more relevant.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/r-squared.webp
Picture description: A physical comparison board places varied observed pegs beside a constant-height benchmark and a separate candidate prediction row.
Picture caption: A model must beat the mean-target squared-error benchmark. R-squared can be negative; the evaluation-set mean is a metric reference, not a deployable forecast.
What you see: Observed outcomes. What it represents: The varied pegs represent the targets against which both forecasts are judged.
What you see: Simple mean baseline. What it represents: The constant row pictures always predicting the evaluation targets' mean.
What you see: Candidate forecast. What it represents: A more complex forecast can incur greater squared error than the constant baseline.
Predict: Can predictions far from all targets drive R-squared below zero?
Try: Change predictions while holding observed targets fixed. Compare their total squared error with the mean-target baseline.
Explain: Once model error exceeds baseline error, the ratio is above 1 and R-squared becomes negative. There is no finite lower bound. If all targets are constant, the denominator is zero and the ordinary formula is undefined.
Optional math: R2 = 1 - SSE / SST
R2: R-squared score
SSE: sum of squared differences between observed targets and model predictions
SST: sum of squared differences between observed targets and their evaluation-set mean
This definition requires nonconstant targets, so SST > 0. In the example, 1 - 10/2 = -4. Some libraries substitute finite scores for constant-target cases; that convention is not the raw mathematical formula.
Common misconception: R-squared always lies between 0 and 1.
Correction: It can be negative when predictions are worse than the mean-target baseline. The nonnegative guarantee sometimes associated with in-sample OLS requires particular fitting conditions and does not apply to arbitrary predictions.
Check your understanding: If model squared error is 10 and mean-baseline squared error is 2, what is R-squared?
0.8 - That reverses the ratio. The model is worse than baseline, so its score should not be positive.
-4 - Yes. 1 - 10/2 = -4; a negative value is valid for this nonconstant target set.
0 because negative scores are impossible - There is no zero floor in the standard formula.
Use it with purpose: A dashboard suppresses negative R-squared values. What should change?
Guidance: Show the negative score and the baseline comparison rather than clipping away the failure. Report absolute error too, and explicitly handle constant-target evaluation sets as undefined or a documented library convention.
Where to go next:
Agent evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Include simple baselines and preserve unfavorable scores instead of presenting success-shaped summaries.
References:
scikit-learn: R-squared Score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html

LESSON: F1 score
URL: https://www.openagentschool.org/ai-ml-foundations/f1-score
Also called: F-measure, harmonic mean of precision and recall
Summary: Balance precision and recall for a named positive class without hiding threshold choices.
In everyday words: F1 is the harmonic mean of precision and recall. Precision asks how many predicted positives were right; recall asks how many actual positives were found. F1 is high only when both are high.
Why it matters: It is useful when finding positive cases matters more than counting the many easy negatives. However, F1 ignores true negatives and does not encode every cost of false alarms or missed cases.
Scenario - A separate seven-ticket review: This worked example uses seven synthetic tickets, not the interactive lab's six: a classifier correctly flags 3 urgent tickets, falsely flags 1 routine ticket, misses 2 urgent tickets, and correctly leaves 1 routine ticket unflagged. Precision is 3/4, recall is 3/5, and F1 is 6/(6 + 1 + 2) = 2/3, about 0.667.
Analogy - Gathering the right items: Precision is how much of your basket belongs there; recall is how much of the desired collection you managed to gather. F1 rewards doing both reasonably well.
Where the analogy stops: The analogy does not assign a price to a wrong item or a missed item. Equal balancing is a choice and may not match a high-stakes task.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/f1-score.webp
Picture description: A basket contains desired round apples and a differently shaped unwanted item, while more desired apples remain outside.
Picture caption: Precision asks what belongs in the basket; recall asks what desired items were missed. F1 balances both without pricing every mistake.
What you see: Correct selections. What it represents: Desired apples inside the basket represent true positives.
What you see: Wrong selections. What it represents: An unwanted item inside the basket represents a false positive that lowers precision.
What you see: Missed desired items. What it represents: Desired apples left outside represent false negatives that lower recall.
Predict: If you lower the positive-decision threshold, must F1 improve?
Try: Compare true positives, false positives, and false negatives at different thresholds before comparing the resulting F1.
Explain: Lowering the threshold commonly retrieves more positives but can add false positives. For fixed scores, recall cannot decrease as the threshold lowers, but precision and F1 need not improve.
Optional math: F1 = 2 * TP / (2 * TP + FP + FN)
F1: F1 score for the named positive class at a specified threshold
TP: true positives: correctly predicted positive examples
FP: false positives: negative examples incorrectly predicted positive
FN: false negatives: positive examples incorrectly predicted negative
The formula avoids dividing precision by recall separately. If TP, FP, and FN are all zero, its denominator is zero; report an explicit undefined-value convention. Multiclass averaging must also be specified.
Common misconception: A high F1 proves a classifier is good for every use case.
Correction: F1 ignores true negatives, depends on class and threshold, and balances two error types in a particular way. Real decision costs and affected groups still need evaluation.
Check your understanding: In the separate seven-ticket example, with TP = 3, FP = 1, and FN = 2, what is F1?
0.75 - That is precision, 3/4. F1 also incorporates the two missed positives.
0.6 - That is recall, 3/5. F1 combines precision and recall rather than using only one.
2/3, approximately 0.667 - Yes. The harmonic mean is 2 * 3 / (2 * 3 + 1 + 2) = 6/9, not precision or recall alone.
Use it with purpose: Should a safety-critical alert threshold maximize F1 by default?
Guidance: First define the cost of missed hazards and unnecessary alerts. Select the threshold on validation data, report the confusion counts and relevant subgroup performance, then assess it on held-out data.
Where to go next:
Agent testing and benchmarks (concept): https://www.openagentschool.org/concepts/agent-testing-benchmarks - Make positive-class, threshold, and averaging conventions part of a reproducible benchmark.
Observability & EvalOps (skill): https://www.openagentschool.org/ai-skills#module/observability - Trace a headline metric back to the false positives and false negatives that operators experience.
References:
scikit-learn: Model Evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html
Google ML Crash Course: Accuracy, Precision, and Recall: https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall

LESSON: Correlation
URL: https://www.openagentschool.org/ai-ml-foundations/correlation
Also called: Pearson correlation, Pearson r
Summary: Measure linear co-movement while keeping causation, outliers, and nonlinear patterns separate.
In everyday words: Pearson correlation measures the direction and strength of a linear association between paired numeric variables. With nonconstant variables, it ranges from -1 to 1. It does not establish that one variable causes the other.
Why it matters: Correlation helps explore data and detect related signals, but it is easy to overinterpret. A strong association may come from a shared cause, selection effects, or chance; a nonlinear relationship may have zero Pearson correlation.
Scenario - A synthetic service-load pattern: Three observation pairs record request load [1, 2, 3] and tool calls [2, 4, 6]. Their Pearson correlation is +1 because they fall on a rising straight line. These three pairs alone do not show that changing load will cause tool calls to double.
Analogy - Two gauges moving together: When two gauges rise and fall together, they show co-movement. Their readings can be associated even if neither controls the other.
Where the analogy stops: Both gauges might respond to a third process. Also, Pearson correlation only measures linear co-movement, not every possible relationship.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/correlation.webp
Picture description: Two analog gauges connect by separate hoses to one shared tabletop reservoir rather than to each other.
Picture caption: Two gauges can move together because of a shared source. Correlation alone does not show one causes the other, and a still image cannot measure it.
What you see: First measurement. What it represents: One gauge represents one variable observed across paired measurements.
What you see: Second measurement. What it represents: The other gauge represents a variable that may move with the first without controlling it.
What you see: Possible shared cause. What it represents: The common reservoir makes a third-factor explanation tangible rather than drawing a causal arrow between gauges.
Predict: For a fixed nonzero trend in this experiment, will stronger departures from the line move the correlation magnitude toward zero?
Try: Hold the trend fixed and increase the residual strength, then compare positive and negative trend directions. Inspect the paired scatter alongside the coefficient.
Explain: The fixed residual pattern adds departures that have zero linear correlation with the input. Strengthening it reduces the magnitude of correlation without changing the sign set by the nonzero trend. This controlled numerical relationship still provides no evidence of causation.
Optional math: r = sum_i(dx_i * dy_i) / sqrt(sum_i(dx_i^2) * sum_i(dy_i^2))
r: Pearson correlation coefficient for paired observations
dx_i: x value of pair i minus the mean of all x values
dy_i: y value of pair i minus the mean of all y values
sum_i: add over the same paired observations i
sqrt: nonnegative square root
Both variables must have nonzero spread; otherwise the denominator is zero and correlation is undefined. For x = [-1, 0, 1] and y = [1, 0, 1], r = 0 despite the exact nonlinear relationship y = x squared.
Common misconception: Correlation proves causation, and zero correlation proves independence.
Correction: Neither follows from Pearson r. A third factor can create association, and a nonlinear dependence can have zero linear correlation.
Check your understanding: Request load and tool calls have Pearson r = 1 on three observations. What can you conclude?
They have a perfect positive linear association in those observations. - Yes. That describes the observed pairs without claiming a causal mechanism or future guarantee.
Increasing request load must cause tool calls to rise. - Correlation alone cannot establish intervention effects; other factors and study design matter.
Future observations must follow the same line. - A small historical pattern does not guarantee behavior under new conditions.
Use it with purpose: An agent recommends changing a policy based only on a correlation. What evidence is missing?
Guidance: Inspect the paired data and plausible shared causes. Seek an appropriate experiment or defensible causal design before claiming the policy will produce the desired effect.
Where to go next:
Verified analytical agents (concept): https://www.openagentschool.org/concepts/verified-analytical-agents - Require agents to separate descriptive statistics from causal claims.
Agents for Science (science): https://www.openagentschool.org/agents-for-science - Use observed associations to generate hypotheses, then design evidence that can test them.
References:
SciPy: Pearson Correlation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html

LESSON: Cosine similarity
URL: https://www.openagentschool.org/ai-ml-foundations/cosine-similarity
Also called: normalized dot product, angular similarity
Summary: Compare vector direction independently of positive rescaling, and handle zero vectors explicitly.
In everyday words: Cosine similarity divides the dot product of two nonzero vectors by the product of their lengths. It measures directional alignment, ranging from -1 for opposite directions to 1 for the same direction.
Why it matters: Embedding retrieval often uses directional similarity to rank candidate evidence. A high score means closeness in the chosen representation, not that a document is correct, relevant in context, or safe to quote.
Scenario - A synthetic document comparison: A query vector [1, 1] and document vector [2, 0] have dot product 2 and lengths sqrt(2) and 2. Their cosine similarity is 1/sqrt(2), about 0.707. Replacing the document vector with [20, 0] changes its size but not this cosine score.
Analogy - Two arrows pointing on a map: Cosine asks whether arrows point in similar directions, not whether one arrow is longer.
Where the analogy stops: Embedding directions are learned representations, not geographic meanings. Directional agreement is not factual agreement, and an arrow with zero length has no defined direction.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/cosine-similarity.webp
Picture description: Two wooden arrows of different lengths share a pivot on a blank street map and point in nearby directions.
Picture caption: Cosine compares arrow directions rather than positive length scaling. Embedding alignment is not truth, and a zero vector has no defined direction.
What you see: Shared origin. What it represents: A common origin makes the angle between the nonzero vectors visible.
What you see: Direction matters. What it represents: The shorter arrow's heading contributes to angular similarity independently of its length.
What you see: Length is normalized. What it represents: The longer shaft does not automatically increase cosine similarity because vector lengths are divided out.
Predict: Will positively scaling one whole vector change cosine similarity?
Try: Compare an original vector with a positively scaled copy while the other vector stays fixed. Contrast cosine with the unnormalized dot product.
Explain: The scaling factor appears in both numerator and length denominator, so cosine is unchanged. The unnormalized dot product changes; a negative scaling instead reverses direction and flips the cosine sign.
Optional math: cosine(a, b) = (a dot b) / (||a|| * ||b||)
a: first nonzero vector
b: second nonzero vector of the same dimension
dot: sum of matching component products
||a||: Euclidean length of a: square root of the sum of its squared components
||b||: Euclidean length of b
cosine(a, b): directional similarity score, not a probability
The mathematical formula is undefined for zero vectors. Some libraries return a conventional zero similarity for them; disclose that handling rather than calling a directionless vector orthogonal.
Common misconception: Cosine similarity is the same as any dot product and gives a truth probability.
Correction: It is a normalized dot product, equal to the ordinary dot product only when both vectors have unit length. The resulting score is not a truth probability.
Check your understanding: What happens to cosine similarity if [2, 0] is replaced with [20, 0] against a fixed nonzero query?
It becomes ten times larger. - That describes the unnormalized dot product. Cosine divides out the positive length scaling.
It stays the same. - Yes. Positive scaling preserves direction, and the common scale factor cancels.
It becomes a probability of 1. - Vector magnitude does not turn a similarity into a calibrated probability or guarantee perfect alignment.
Use it with purpose: The highest-cosine document contradicts a trusted source. Should the agent use it uncritically?
Guidance: No. Similarity finds candidates; grounding requires checking relevance, authority, recency, and conflicting evidence. Evaluate retrieval quality with actual questions and document judgments, not cosine values alone.
Where to go next:
Retrieval engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Choose and test a similarity metric appropriate to the embedding model.
Hallucination and grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Separate finding nearby text from verifying evidence for an answer.
RAG Systems (skill): https://www.openagentschool.org/ai-skills#module/rag-systems - Apply directional retrieval scores within a workflow that still checks retrieved evidence.
References:
scikit-learn: Cosine Similarity: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html

LESSON: Datasets & Labels
URL: https://www.openagentschool.org/ai-ml-foundations/datasets-and-labels
Also called: features, targets, annotation, label quality
Summary: Decide what an example means before asking a model to learn from it.
In everyday words: A dataset is a collection of examples gathered for a purpose. In supervised learning, each example pairs input features with a target label: the answer the system is meant to predict. A label is a recorded judgment or measurement, not automatically the truth. Its meaning depends on the policy, measurement process, and time when it was collected.
Why it matters: A learner can become excellent at reproducing a bad labeling rule. Before increasing model size, ask whether examples cover the intended users and whether two reviewers would assign the same answer. Missing rare cases, inconsistent definitions, and labels recorded after a decision can defeat an otherwise sound project.
Scenario - The fictional library help desk: A library wants to route synthetic messages to either account help or book search. One reviewer labels "I cannot renew this book" as account help; another labels it book search. You have 100 invented messages, including only five account-lockout cases. Write a one-sentence routing rule and decide how to handle ambiguous examples before training.
Analogy - Practice cards with an answer key: Features are the front of each practice card; labels are the answer on the back. A consistent answer key makes practice useful. Repeating a mistaken card teaches the mistake more firmly.
Where the analogy stops: Unlike a classroom answer key, real labels may be uncertain, disputed, or change when policy changes. Some learning tasks also have no human-provided labels.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/datasets-and-labels.webp
Picture description: Blank practice cards with a wooden leaf, a flipped card with a removable sticker, and a review tray with two different stickers.
Picture caption: An example and its recorded answer belong together; the answer still needs a trustworthy rule.
What you see: Example. What it represents: The object on the card stands for the input features presented to a learner.
What you see: Recorded label. What it represents: The removable sticker stands for an assigned target, not unquestionable truth.
What you see: Review the rule. What it represents: The differing stickers stand for label disagreements that need a shared definition.
Predict: If the recorded labels become less reliable while the examples stay the same, should a high score against those labels become more trustworthy?
Try: Compare the fixed feature rows, rule labels, and recorded labels. Change the labeling threshold or corrupt the first few labels; notice that neither action resamples the rows. Separately, relabel the renewal message and identify one rare case you would collect next.
Explain: Counting examples helps reveal coverage; it does not settle what the labels should mean. A score measures agreement with recorded targets. Review disagreements, record the labeling rule and source, and keep genuinely uncertain cases visible instead of forcing a convenient answer.
Optional math: class share = n_c / N
class share: Fraction of all examples assigned to the selected class.
n_c: Number of examples with that class label.
N: Total number of examples; must be greater than zero.
Five lockout examples among 100 messages give a share of 0.05, or 5%. This describes representation, not label correctness or the class frequency your future users will produce.
Common misconception: More labeled data always fixes a weak model.
Correction: More inconsistent or unrepresentative labels can reinforce the wrong behavior. Audit definitions, coverage, consent, and provenance before expanding the collection.
Check your understanding: Two reviewers repeatedly disagree about renewal messages. What is the best first move?
Duplicate those messages to give the model more practice. - Duplication amplifies the disagreement without resolving it. First define what destination should own a renewal request.
Clarify the routing rule and review a sample together. - Yes. An explicit rule and reviewed disagreements improve the target itself. Preserve unresolved cases and document the decision.
Delete every difficult message from the dataset. - That makes the dataset easier but hides real user needs. Keep a documented ambiguity path rather than silently removing the hard cases.
Use it with purpose: A teammate offers 10,000 cheap labels with no description of how they were produced. Accept them?
Guidance: Request source, permitted use, labeling instructions, class counts, and a reviewed sample first. Accept only if the labels match the decision you intend to support; quantity alone is not evidence.
Where to go next:
AI-Ready Data (concept): https://www.openagentschool.org/concepts/ai-ready-data - Turn a labeling rule into a repeatable data-quality and provenance process.
Synthetic Training Data & Distillation (concept): https://www.openagentschool.org/concepts/synthetic-training-data - Apply the same label definitions and review standards when examples come from a generator.
Model Improvement Engineering (course): https://www.openagentschool.org/courses/model-improvement-engineering - Practice reviewing policy-grounded labels before improving a refund agent.
References:
Gebru et al. - Datasheets for Datasets: https://arxiv.org/abs/1803.09010

LESSON: Training & Inference
URL: https://www.openagentschool.org/ai-ml-foundations/training-and-inference
Also called: model fitting, prediction, weights, parameters
Summary: Separate changing a model from using the model you already have.
In everyday words: Training adjusts a model's parameters using examples and an objective. Inference uses a chosen set of parameters to produce an output for an input. Ordinary inference does not update those parameters, even when the application keeps a conversation history or retrieves new documents.
Why it matters: The distinction tells you where a change belongs. Supplying today's policy in context can change an answer without retraining. Improving a persistent behavior may call for training, but it also creates a new model version that needs evaluation, deployment, and rollback.
Scenario - A fictional parcel-time estimator: A courier estimates travel minutes from distance using invented trips. During training it adjusts a slope and an offset to reduce errors. At dispatch time it receives a new distance and returns minutes using the frozen values. Adding a new trip to a spreadsheet does not change the deployed estimator until a training and release process uses it.
Analogy - Adjusting a recipe versus cooking dinner: Training resembles testing recipes and changing ingredient amounts. Inference resembles making a meal with the selected recipe. A different order changes the meal, not the recipe instructions.
Where the analogy stops: Models do not literally understand recipes. Some systems intentionally run online learning, but that is an explicit update mechanism, not an automatic property of every prediction.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/training-and-inference.webp
Picture description: Kitchen trial bowls sit left of a closed recipe binder; a finished soup prepared for service sits on the right.
Picture caption: Training changes the recipe; inference uses the selected recipe to serve a new request.
What you see: Adjust the recipe. What it represents: The trial bowls represent training that changes model parameters.
What you see: Chosen parameters. What it represents: The closed recipe binder represents a fixed model version selected for use.
What you see: Use the recipe. What it represents: The finished soup represents inference, which uses rather than rewrites the model.
Predict: If you change only the new parcel distance, which should change: the prediction, the learned parameters, or both?
Try: Inspect the small numerical training/inference example. Compare changing an input with changing the model parameters. Describe which change would require issuing a new model version in the courier scenario.
Explain: Input changes can alter outputs while the model stays fixed. A parameter change alters the function used for future inputs. This browser exercise demonstrates a tiny numerical model; no real LLM training occurs here, and no provider key is needed.
Optional math: prediction = w * x + b
prediction: Estimated travel time in minutes.
w: Learned slope, measured in minutes per unit of distance.
x: Input distance for the current trip.
b: Learned offset in minutes, such as a fixed handling delay.
*: Multiplication.
Training can change w and b; inference substitutes a new x into the chosen equation. The demo fixes b at zero and updates only w. A language model has a much larger function, but this separation still helps you reason about its lifecycle.
Common misconception: Every chat message teaches the model permanently.
Correction: A message may affect the current context or an application memory store. That is different from updating model weights. Later provider training depends on the product, settings, and data policy.
Check your understanding: An assistant reads a newly retrieved policy and changes its answer. What does that establish?
Its input context changed; its weights need not have changed. - Correct. Retrieval supplies information during inference. A changed answer alone is not evidence of training.
It completed a fine-tuning run. - Fine-tuning requires a parameter-update process. Reading a retrieved document does not establish that any update occurred.
It will remember that policy in every future session. - Context may disappear after the request or session. Persistent recall requires deliberate storage or a separate training process.
Use it with purpose: Prices change every morning. Should you retrain nightly just to expose the latest prices?
Guidance: Start with an authoritative lookup or retrieval at inference time. Consider training for a measured behavior gap, not as the default database-update mechanism.
Where to go next:
Atomic LLM Training (concept): https://www.openagentschool.org/concepts/atomic-llm-training - Follow the parameter-update loop after distinguishing it from a prediction.
What Is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Connect ordinary token prediction to inference with a fixed set of learned parameters.
Fine-Tuning (concept): https://www.openagentschool.org/concepts/fine-tuning - Recognize when a behavior change requires a deliberate parameter update rather than new context.
Fine-Tuning vs. RAG vs. Prompting (course): https://www.openagentschool.org/courses/fine-tuning-rag-prompting - Choose whether missing knowledge belongs in retrieval or a measured behavior gap merits training.
References:
scikit-learn - Getting Started: fitting and predicting: https://scikit-learn.org/stable/getting_started.html

LESSON: Train, Validation & Test
URL: https://www.openagentschool.org/ai-ml-foundations/train-validation-test
Also called: holdout, data split, cross-validation, held-out sets
Summary: Give practice, model selection, and final evaluation different jobs.
In everyday words: The training set supplies examples for parameter fitting. The validation set helps choose model settings and development decisions without fitting the base model on those examples. The test set is reserved for a final evaluation of the chosen procedure. These are roles, not three magic percentages.
Why it matters: If every evaluation result influences the next revision, you gradually adapt to the evaluation examples. A final held-out test protects a more honest estimate. Splitting must also respect the deployment question: new people, future time periods, or new documents may require different boundaries.
Scenario - Predicting next month's repair requests: A fictional repair shop has 1,000 synthetic visits, with repeat visits by the same customer. You want performance on new customers next month. Randomly splitting rows can place almost identical visits on both sides. Plan customer-separated development data and a later final evaluation that matches the time boundary.
Analogy - Practice, rehearsal, and an unseen exam: Training is practice with answers. Validation is a rehearsal used to choose your approach. The test is the unseen exam after that approach is fixed.
Where the analogy stops: A rehearsal can also be overused. Real datasets contain families of related examples, so shuffling individual cards does not always create a genuinely unseen exam.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/train-validation-test.webp
Picture description: An accessible practice-card stack, a separate rehearsal folder, and a sealed opaque exam envelope occupy three distinct desk stations.
Picture caption: Practice fits the model, rehearsal guides choices, and the sealed exam protects final evidence.
What you see: Train. What it represents: The accessible practice cards represent examples used to fit parameters.
What you see: Validate. What it represents: The rehearsal folder represents development evidence used to choose settings.
What you see: Test stays unseen. What it represents: The sealed envelope represents final evaluation kept outside fitting and selection.
Predict: If you give nearly every example to training, what happens to your ability to judge the model reliably?
Try: Explore the split allocation and inspect how many examples each role receives. Then sketch a split by customer and time for the repair shop; explain why changing percentages alone cannot prevent related visits crossing the boundary.
Explain: More training examples can help fitting, but tiny evaluation sets make estimates unstable. Keep the final test untouched by tuning. Validation guides selection; if you repeatedly optimize against it, obtain fresh evidence rather than relabeling it a final test.
Optional math: N = n_train + n_validation + n_test
N: Total number of examples assigned to exactly one split.
n_train: Number assigned to parameter fitting.
n_validation: Number assigned to development and selection.
n_test: Number reserved for final evaluation.
The counts add up when the partitions do not overlap. Non-overlapping row IDs are necessary but insufficient: duplicate content, customers, or future information can still connect the splits.
Common misconception: A held-out test is still independent after I use it to pick the best model.
Correction: Once its results influence selection, it has served a development role. Do not tune on the final test and then report that same score as untouched evidence.
Check your understanding: You try 30 settings and pick the one with the best test score. What should you do next?
Report the best score as an independent final estimate. - The test influenced selection, so that estimate is optimistic. Calling the split "test" does not restore independence.
Fit all 30 models directly on the test labels. - That removes the remaining evaluation boundary. Model fitting belongs in training, with development choices separated from final evidence.
Treat that set as development data and obtain a fresh final evaluation. - Yes. Be transparent about the selection process and freeze the procedure before using new independent evidence.
Use it with purpose: Your model will predict future weekly demand. Is a random row split enough?
Guidance: Use a time-aware evaluation that withholds later periods and prevents future-derived features. Choose the split to imitate the future decision, not merely a familiar ratio.
Where to go next:
Production ML Systems (concept): https://www.openagentschool.org/concepts/production-ml-systems - Carry split boundaries and dataset versions into release evidence.
Agent Testing & Benchmarks (concept): https://www.openagentschool.org/concepts/agent-testing-benchmarks - Reserve benchmark cases for evaluation instead of silently using them to select every revision.
Agent Evals & Benchmarking Lab (course): https://www.openagentschool.org/courses/agent-evals-benchmarking-lab - Build evaluation sets that are not silently reused as practice material.
References:
scikit-learn - Cross-validation: evaluating estimator performance: https://scikit-learn.org/stable/modules/cross_validation.html

LESSON: Overfitting
URL: https://www.openagentschool.org/ai-ml-foundations/overfitting
Also called: generalization gap, memorization, underfitting, regularization
Summary: Notice when learning the practice examples stops helping on new ones.
In everyday words: Overfitting happens when a model adapts too closely to details of the training data that do not transfer to new examples. It may learn useful structure and accidental noise together. The warning is not simply a complicated model; it is poor generalization relative to its training performance.
Why it matters: A beautifully low training error can conceal a brittle service. Comparing training and validation behavior helps you decide whether more fitting is useful. Simpler models, regularization, early stopping, or better data can help, but none replaces a deployment-matched evaluation.
Scenario - The fictional cafe demand curve: You have twelve invented days of cafe sales. One rainy day included a large private event. A flexible curve can pass through every point, including that unusual spike. Your job is to predict ordinary future days, not draw the prettiest line through last week.
Analogy - Memorizing a practice quiz: Someone can memorize the exact order of answers and look perfect on the practice sheet while struggling when the numbers change. Learning the method transfers; memorizing accidents does not.
Where the analogy stops: Memorization and generalization are not absolute opposites. Large models can memorize some examples and still generalize elsewhere, so inspect task-specific evidence.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/overfitting.webp
Picture description: A worn practice sheet with wooden shapes, a matching peg strip, and a fresh sheet with rearranged shapes illustrate memorized order versus new cases.
Picture caption: Remembering one answer order is not the same as learning a method that transfers.
What you see: Familiar practice. What it represents: The worn arrangement represents training examples seen repeatedly.
What you see: Memorized order. What it represents: The matching peg strip represents a shortcut tied to accidental details of the practice set.
What you see: Changed cases. What it represents: The rearranged sheet represents new examples where the shortcut may no longer help.
Predict: As the example model becomes more flexible, must performance on unseen days improve whenever training error falls?
Try: Compare training error with held-out clean error as the toy blends toward interpolation of noisy labels. Explain why the lowest training error alone would not guide your choice, then describe the separate validation and final-test procedure a real project needs.
Explain: The display checks a deliberately chosen model against four clean held-out points; it does not run an independent validation selection loop. A growing error gap can suggest fitting noise. In real work, high errors on both splits may instead suggest underfitting or missing information; inspect examples before prescribing regularization.
Optional math: gap = validation error - training error
gap: Difference between two errors measured using the same metric.
validation error: Average error on examples used for development evaluation.
training error: Average error on examples used for fitting.
This is a development diagnostic, not the demo's literal output label: the display reports training MSE and held-out clean MSE. In a real validation procedure, a positive gap means validation error is larger. Sample size, noise, and split differences matter; there is no universal overfitting threshold.
Common misconception: Zero training error proves the model is ready.
Correction: It proves agreement with the training examples under that metric. Readiness requires independent examples, important failure cases, and acceptable consequences when predictions are wrong.
Check your understanding: Training error keeps falling while validation error rises. Which response is most justified?
Keep training until both must reach zero. - Further fitting can reinforce noise; there is no guarantee validation error will follow training error down.
Use validation to compare an earlier or more constrained model. - Correct. This targets the observed generalization problem while leaving the final test out of selection.
Move the hardest validation cases into training and report the remainder. - That changes the exam to make the result look better. Preserve the evaluation purpose rather than removing inconvenient evidence.
Use it with purpose: A larger model wins on training data but loses on validation. Which would you ship?
Guidance: Prefer the candidate with stronger relevant validation evidence, then check it once against the reserved final evaluation. Investigate the gap and report uncertainty instead of choosing by size or training score.
Where to go next:
Fine-Tuning (concept): https://www.openagentschool.org/concepts/fine-tuning - Use generalization evidence to decide whether an additional training intervention actually helps.
Atomic LLM Training (concept): https://www.openagentschool.org/concepts/atomic-llm-training - Relate repeated parameter updates and falling training loss to a separate generalization check.
Fine-Tuning vs. RAG vs. Prompting (course): https://www.openagentschool.org/courses/fine-tuning-rag-prompting - Compare simpler interventions before assuming a larger fitted model is better.
References:
scikit-learn - Cross-validation and overfitting: https://scikit-learn.org/stable/modules/cross_validation.html

LESSON: Data Leakage
URL: https://www.openagentschool.org/ai-ml-foundations/data-leakage
Also called: target leakage, train-test contamination, future leakage
Summary: Catch information that sneaks across the boundary of an honest prediction.
In everyday words: Data leakage occurs when model development uses information that would not be available for the intended prediction. It includes future outcomes inside features, related or duplicate examples across splits, and preprocessing fitted using evaluation data. Leakage can make an ordinary model look astonishingly capable.
Why it matters: You cannot fix a dishonest evaluation by adding more decimal places. A release based on leaked information can fail immediately when the shortcut disappears. Trace when every feature becomes available and which examples informed every fitted transformation.
Scenario - The mysteriously accurate refund predictor: A synthetic support table predicts whether a refund will be approved when a request arrives. It includes a "refund completed" field written three days later. The model looks nearly perfect. A second leak appears when repeated copies of the same ticket land in both training and test.
Analogy - An answer visible through the exam paper: The model appears to solve the question, but a clue from the answer sheet is already visible. Remove the clue and the apparent skill vanishes.
Where the analogy stops: Leakage is often accidental and subtle, not deliberate cheating. A preprocessing average or duplicated document can cross the boundary without exposing an obvious answer column.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/data-leakage.webp
Picture description: Colored answer stickers show through a thin exam sheet on the left; an opaque sheet on a separate clipboard has no answer backing.
Picture caption: A leaked clue can look like skill until the clue is no longer available.
What you see: Shortcut shows through. What it represents: The visible sticker pattern represents information unavailable at the intended prediction time.
What you see: Answer source. What it represents: The exposed backing card represents the outcome information that leaked into development.
What you see: Clue unavailable. What it represents: The separate opaque sheet represents prediction without access to the forbidden shortcut.
Predict: Would a better score still be useful if the model obtained it from information created after the decision?
Try: Contrast the leaky predictions with predictions where the unavailable answer-derived feature is replaced by the stated value 0.5. For the refund table, mark each field as available at arrival or only later, and group duplicate tickets before splitting.
Explain: Both toy comparisons use the same four labels; the second is not an independently collected test set, and the replacement rule is not a production recommendation. Ask, "Could this information exist at prediction time?" Split first, fit preprocessing only on the training portion, and fit transforms again inside each cross-validation training fold.
Optional math: apparent error reduction = clean error - leaked error
apparent error reduction: How much lower error appears when the evaluation contains a forbidden shortcut.
clean error: Error using the same metric without access to the leaked answer feature.
leaked error: Error from an evaluation contaminated by unavailable information.
The demo uses Brier error, the average squared difference between predicted probabilities and binary labels; lower is better. A positive apparent reduction is not a real improvement if it came from leakage. The size depends on the shortcut and the stated missing-feature rule, not a universal law.
Common misconception: Deleting the target column guarantees no leakage.
Correction: Target proxies, future timestamps, duplicate entities, and preprocessing fitted on the full dataset can still expose evaluation information. Audit the whole pipeline, not just column names.
Check your understanding: You standardize all examples, then split into training and test. What should change?
Split first and learn the standardization statistics only from training. - Yes. Apply those training-derived statistics to validation and test so evaluation examples do not influence the fitted transform.
Standardize the test set using its own mean for every deployment. - That defines a different, batch-dependent procedure and may not match deployment. Use the transformation fitted by the training pipeline.
Keep the pipeline because preprocessing cannot leak labels. - Even without labels, evaluation inputs can influence a fitted transformation. Feature selection using labels is another, even clearer leakage risk.
Use it with purpose: A new feature suddenly doubles your score. Celebrate or investigate?
Guidance: Investigate its creation time, lineage, and relationship to the target. Re-run a clean split and deployment-realistic pipeline before counting the gain as useful.
Where to go next:
Production ML Systems (concept): https://www.openagentschool.org/concepts/production-ml-systems - Carry point-in-time feature checks and clean evaluation boundaries into production.
Synthetic Evaluation Data (concept): https://www.openagentschool.org/concepts/synthetic-evaluation-data - Prevent generated test cases and their paraphrases from leaking into training or development prompts.
Agent Evals & Benchmarking Lab (course): https://www.openagentschool.org/courses/agent-evals-benchmarking-lab - Keep benchmark evidence separate from development examples and tuning decisions.
References:
scikit-learn - Common pitfalls: data leakage: https://scikit-learn.org/stable/common_pitfalls.html#data-leakage

LESSON: Embeddings
URL: https://www.openagentschool.org/ai-ml-foundations/embeddings
Also called: vectors, semantic representations, vector search
Summary: Represent items as numbers, then ask what their geometry can and cannot tell you.
In everyday words: An embedding represents an item as a vector: an ordered list of numbers. Learned embeddings place useful relationships into that numerical space according to a training objective. Items might be words, passages, images, or products; the meaning of "near" depends on the model, data, and comparison method.
Why it matters: Embeddings help retrieve related material even when wording differs. But nearby vectors are candidates, not verified answers. The choice of model, chunking, distance measure, and evaluation queries determines whether retrieval serves the actual task.
Scenario - Finding a fictional museum policy: A visitor asks, "Can I bring my toddler's pushchair?" The handbook says "strollers are permitted." A useful representation may connect these expressions even without the same words. A nearby page about wheelchair storage is related, but it may not answer the stroller question.
Analogy - A map organized by useful resemblance: Imagine a shop map where products used together sit nearby, even if their names begin with different letters. Coordinates let you compare locations efficiently.
Where the analogy stops: Real embedding dimensions usually are not neat human labels like "kindness" or "truth." A map can encode bias, miss negation, and place contradictory claims near each other.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/embeddings.webp
Picture description: A watering can sits near seedlings and gardening gloves, while plain books and a reading lamp occupy a separate shop display across an aisle.
Picture caption: Nearby representations can suggest related uses, not guaranteed agreement or truth.
What you see: An item. What it represents: The watering can represents one item placed in a representation space.
What you see: Useful neighbors. What it represents: Nearby gardening supplies represent items sharing useful relationships under a chosen representation.
What you see: Different use. What it represents: The separate reading display represents distance under that chosen notion of resemblance, not a truth ranking.
Predict: If two vectors point in a similar direction, does that prove their associated statements agree?
Try: Move through the toy vector example and compare geometric similarity. Then write two museum statements that share a topic but disagree, such as allowing versus prohibiting strollers.
Explain: The toy embeddings are hand-designed numbers; they do not learn language. The display uses Euclidean distance: smaller means nearer. Geometry shows how retrieval works, not semantic competence. Production retrieval needs a trained representation and checks that retrieved passages actually support the answer.
Optional math: cosine similarity = (u dot v) / (length(u) * length(v))
cosine similarity: Directional similarity between two nonzero vectors, ranging from -1 to 1.
u, v: Two vectors with the same number of coordinates.
dot: Multiply matching coordinates and add the products.
length: Euclidean length: square each coordinate, add, then take the square root.
*: Multiplication.
Cosine is an alternative to the demo's Euclidean distance, not its displayed calculation. Dividing by both lengths compares direction rather than absolute location. A zero vector has no direction, so cosine is undefined for it even though Euclidean distance still works. Neither metric establishes factual support.
Common misconception: A high similarity score means the retrieved answer is true.
Correction: It indicates closeness under a representation and metric. Check source authority, relevance, date, and whether the passage entails the claim before answering.
Check your understanding: Your top result says strollers are prohibited, while the query asks whether they are allowed. Why can similarity still be high?
Embeddings guarantee agreement, so the source must be wrong. - Similarity is not agreement. The source may be the decisive evidence precisely because it discusses the same policy and gives a negative answer.
The two texts discuss the same topic despite different conclusions. - Correct. Retrieve candidates by resemblance, then read for the actual rule, including negation and exceptions.
Vector length directly counts true statements. - Embedding length has no general truth-count interpretation. Its behavior depends on the model and training objective.
Use it with purpose: Keyword search misses paraphrases. What is a responsible next experiment?
Guidance: Compare embedding or hybrid retrieval on representative questions, including negations and rare terms. Measure evidence recall and answer support rather than trusting a pretty cluster.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-embeddings-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-embeddings-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-embeddings-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-embeddings-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Retrieval Engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Turn vector resemblance into evaluated retrieval with relevant, supporting evidence.
RAG Basics (concept): https://www.openagentschool.org/concepts/rag-basics - Place vector retrieval inside the retrieve-then-generate workflow, with source support checked afterward.
Agentic RAG (pattern): https://www.openagentschool.org/patterns/agentic-rag - See how an agent can refine retrieval and verify a candidate passage rather than accepting the nearest vector.
References:
Mikolov et al. - Efficient Estimation of Word Representations in Vector Space: https://arxiv.org/abs/1301.3781

LESSON: Tokens
URL: https://www.openagentschool.org/ai-ml-foundations/tokens
Also called: tokenization, subwords, context window, BPE
Summary: See why model input units are not the same thing as words or characters.
In everyday words: A tokenizer converts input into units called tokens and maps them to IDs a model can use. For text, tokens may represent whole words, word pieces, punctuation, or bytes. The exact split depends on the tokenizer and its vocabulary; there is no universal conversion from words to tokens.
Why it matters: Token counts affect context limits, processing work, and many API prices. Code, uncommon names, and different languages can have very different token counts for similar-looking text. Truncating by characters alone can discard the wrong information or misestimate a request budget.
Scenario - A fictional multilingual booking note: A booking assistant receives an English sentence, a customer name with accents, and a long reference code. The team assumes every word is one token and reserves no space for the answer. You need a budget that includes system instructions, retrieved context, the user message, and the expected output.
Analogy - Packing text with reusable tiles: Common pieces can use one large tile; unfamiliar pieces may need several smaller tiles. The tile set determines how many pieces are required.
Where the analogy stops: Tokens are numerical vocabulary entries, not physical tiles, and the model may use special control tokens. Some tokenizers operate on bytes, so visible characters do not always align with boundaries.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/tokens.webp
Picture description: Two equal-size wooden trays are filled with large ceramic tiles on the left and smaller reusable tiles on the right.
Picture caption: The available pieces change how input is divided; tokens are not always whole words.
What you see: Larger pieces. What it represents: The large tiles represent familiar pieces that a tokenizer may encode as single units.
What you see: Smaller pieces. What it represents: The small tiles represent segmentation into more units when the vocabulary requires it.
Predict: Will replacing a familiar word with a long product code necessarily preserve the token count?
Try: Inspect the demonstration's text units and compare its counting behavior. List which parts of the booking request consume input space, then reserve room for an answer before adding another document.
Explain: This is a toy tokenizer, explicitly not a provider tokenizer. Its count illustrates segmentation only; do not use it for billing or context-limit guarantees. Measure production requests with the exact model tokenizer and account for message formatting and special tokens.
Optional math: available input = context limit - reserved output
available input: Input-token budget under a simplified shared input/output context limit.
context limit: Maximum combined token capacity assumed for this example.
reserved output: Tokens set aside for the generated response.
This budgeting model requires input plus reserved output to fit the assumed limit. Real providers can impose separate limits and additional accounting rules; use their exact specification rather than this toy count.
Common misconception: One token is always one word.
Correction: A word can span multiple tokens, and a token can include whitespace or punctuation. Token-to-word ratios vary across languages, domains, and tokenizer versions.
Check your understanding: A toy demo counts 80 tokens. What can you safely conclude about a provider request?
Every provider will bill exactly 80 input tokens. - The provider may use a different vocabulary, formatting, and special tokens. Toy counts cannot establish billing.
The request must contain exactly 80 words. - Tokens and words are different units. Even this demo may split or combine text differently from word boundaries.
The demo illustrates segmentation; the actual model needs its own count. - Yes. Use the actual tokenizer and request format before enforcing budgets or reporting cost.
Use it with purpose: A long retrieved document pushes a request over its limit. What should you cut?
Guidance: Keep the instruction, question, and most relevant evidence; remove redundant context deliberately and reserve answer space. Do not blindly truncate the end where an exception or citation may live.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-tokens-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-tokens-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-tokens-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-tokens-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
What Is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Connect token IDs to the next-token prediction process.
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Choose which instructions and evidence deserve limited input space.
References:
Hugging Face Transformers - Tokenization algorithms: https://huggingface.co/docs/transformers/tokenizer_summary

LESSON: Attention
URL: https://www.openagentschool.org/ai-ml-foundations/attention
Also called: attention mechanism, self-attention, query key value, attention weights
Summary: Follow a weighted mixture of information without mistaking it for a reason.
In everyday words: Attention computes how much information to combine from different positions. In a common form, a query is compared with keys to produce scores; normalized scores become weights used to mix value vectors. A transformer learns the projections that create queries, keys, and values.
Why it matters: This mechanism lets a representation depend on relevant surrounding information rather than treating each token in isolation. It also explains why a context window is not a simple bag of words. But an attention display shows an internal calculation, not a complete account of why an answer is correct.
Scenario - The fictional dispatch note: A note says, "The blue crate is fragile; move it gently." A toy display gives different weights to information positions while forming a new value. You can inspect how emphasizing one value changes the mixture, but cannot infer from the toy that a model has understood what "it" refers to.
Analogy - Mixing audio channels: A sound desk gives each channel a volume and combines the results. Attention weights play a similar role when combining numerical value vectors.
Where the analogy stops: A loud channel is not necessarily the cause of the listener's interpretation. Real transformers contain multiple heads, layers, residual paths, and learned transformations absent from a simple mixer.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/attention.webp
Picture description: A microphone, an unlabeled analog mixer with differently positioned faders, and a connected speaker illustrate adjustable contributions to one mix.
Picture caption: Attention mixes contributions; a large mixing weight is not a complete explanation.
What you see: Information source. What it represents: The microphone represents incoming information available to a mixing operation.
What you see: Relative weights. What it represents: The mixer faders represent how strongly individual values contribute to the mixture.
What you see: Combined output. What it represents: The speaker represents the resulting weighted mixture, not proof of causal reasoning.
Predict: If a position receives more weight, what happens to the mixture when its value is unlike the other values?
Try: Compare the displayed scores, normalized weights, and combined value. Change the relative emphasis and trace the arithmetic rather than assigning a psychological explanation to a highlighted word.
Explain: Attention is a weighted sum of values. Scores are normalized together, so increasing one relative score redistributes weight. The toy does not establish language understanding, causal reasoning, or a faithful explanation of any real model answer.
Optional math: output = sum_i(a_i * v_i), with sum_i(a_i) = 1
output: Combined scalar or vector produced by this attention mixture.
i: Index of a value position included in the mixture.
a_i: Nonnegative normalized attention weight for position i.
v_i: Value at position i; vectors must have matching dimensions.
sum_i: Add the terms across all included positions.
*: Multiply a value by its scalar weight.
With values 2 and 10 and weights 0.75 and 0.25, the mixture is 4. This worked example illustrates value mixing, not how a trained model learned its queries or keys.
Common misconception: The largest attention weight tells me the true reason for the answer.
Correction: A weight records one internal mixing operation. Other heads, layers, and pathways contribute; a causal explanation requires additional evidence and interventions.
Check your understanding: One word has the largest weight in one attention head. What is the defensible statement?
That word caused the final answer and proves the reasoning. - One head's weight cannot establish the full causal story or correctness. Other computations may change or bypass that contribution.
Its value receives the largest mixing weight in that operation. - Exactly. This describes the observed computation without turning a local weight into an unsupported explanation.
The model has assigned that word the highest probability of being true. - Attention weights are not truth probabilities. They weight values for a representation update.
Use it with purpose: A stakeholder wants an attention heatmap as proof that an answer used the right evidence. Enough?
Guidance: Use it as an exploratory visualization, not proof. Check answer support in the source and test behavior when relevant evidence is removed or changed.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-attention-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-attention-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-attention-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-attention-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
What Is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Place weighted value mixing inside the larger transformer prediction process.
Hallucination & Grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Distinguish an internal visualization from evidence that supports a generated claim.
References:
Vaswani et al. - Attention Is All You Need: https://arxiv.org/abs/1706.03762
Jain and Wallace - Attention is not Explanation: https://aclanthology.org/N19-1357/

LESSON: Temperature
URL: https://www.openagentschool.org/ai-ml-foundations/temperature
Also called: sampling, logits, generation randomness
Summary: Change sampling variety without pretending to change what is true.
In everyday words: During token sampling, temperature rescales model scores before they become a probability distribution. Lower positive temperature concentrates probability on higher-scoring choices; higher temperature spreads it more evenly. It changes selection behavior, not the model's stored knowledge or access to evidence.
Why it matters: You may want variety when drafting names and consistency when formatting records. Neither goal is the same as truth. Temperature interacts with decoding rules such as top-k or top-p filtering, and greedy decoding may ignore it entirely.
Scenario - A fictional exhibition title assistant: A museum wants several titles for an exhibit, then a factual label listing the opening date. More varied title drafts can be useful. A varied date is not. The team must decide separately how to sample creative wording and how to retrieve and verify the date.
Analogy - A weighted raffle: Imagine candidate tokens receiving raffle tickets. Lower temperature makes the favored candidate dominate the ticket pile; higher temperature gives alternatives more relative chance.
Where the analogy stops: The raffle contains whatever candidates the model scores, including wrong ones. Changing ticket allocation cannot check the museum calendar or make an unsupported fact reliable.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/temperature.webp
Picture description: Two raffle bowls contain the same ticket types; one is dominated by a single type and the other has a more varied mixture.
Picture caption: Temperature changes the mix of possible choices, not which choices are true.
What you see: Concentrated choice. What it represents: The dominant ticket type represents a lower-temperature distribution favoring high-scoring candidates.
What you see: More varied choice. What it represents: The more balanced mix represents a higher-temperature distribution, without a truth guarantee.
Predict: If the highest-scoring candidate is false, will lowering temperature fix the factual problem?
Try: Compare how the fixed candidate scores become different probability distributions as temperature changes. Notice whether the most likely candidate changes, and separate that observation from any claim about truth.
Explain: For fixed scores and positive temperature, scaling changes relative probability but preserves their order. A lower temperature can make a wrong favorite more consistent. This deterministic display shows probabilities; it is not a live LLM or evidence of generated-answer quality.
Optional math: p_i = exp(z_i / T) / sum_j(exp(z_j / T))
p_i: Sampling probability assigned to candidate i.
i, j: Candidate indices; j runs over every candidate in the denominator.
z_i, z_j: Unnormalized model scores, also called logits.
T: Temperature, strictly greater than zero in this formula.
exp: Exponential function, raising the constant e to the given power.
sum_j: Add across all candidates so the probabilities total one.
T = 1 leaves ordinary softmax scaling unchanged. A system's temperature-zero option commonly selects greedily; zero is not substituted into this division. Provider behavior and other decoding settings still matter.
Common misconception: Temperature is a truth dial.
Correction: It is a sampling control. Grounding, tool results, task design, and evaluation address factual reliability; a repeatable answer can still be wrong.
Check your understanding: A model repeatedly gives the wrong opening date at low temperature. What is the best fix?
Provide an authoritative date source and verify the answer. - Correct. You are fixing the evidence problem rather than hoping a different sampling distribution discovers the calendar.
Lower temperature further because consistency guarantees truth. - Consistency can lock in the same wrong favorite. Obtain the missing evidence and evaluate factual support.
Raise temperature until one date sounds plausible. - More variety can produce additional wrong dates. Plausibility is not verification.
Use it with purpose: You need diverse slogan drafts and exact inventory facts in one workflow. Use one setting for both?
Guidance: Separate the tasks. Allow evaluated variety for drafts; use authoritative inventory lookup and output checks for facts. Measure each task instead of treating temperature as a universal quality knob.
Where to go next:
What Is an LLM? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Connect score distributions to next-token selection rather than truth estimation.
Hallucination & Grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Address unsupported facts through evidence rather than assuming a sharper sampling distribution is truthful.
Agentic RAG (pattern): https://www.openagentschool.org/patterns/agentic-rag - Use retrieved evidence to address missing facts instead of changing sampling variety.
References:
Hugging Face Transformers - Generation configuration: https://huggingface.co/docs/transformers/main_classes/text_generation

LESSON: Calibration
URL: https://www.openagentschool.org/ai-ml-foundations/calibration
Also called: reliability diagram, confidence, probability calibration, overconfidence
Summary: Check whether confidence matches how often predictions are actually right.
In everyday words: Calibration concerns whether stated probabilities match observed frequencies across comparable predictions. If a system assigns 80% confidence to many selected answers, roughly 80% of those answers should be correct for that group to be well calibrated. Accuracy instead asks how often the selected answer is right overall.
Why it matters: A workflow that escalates uncertain cases needs probabilities it can rely on, not just confident wording. Calibration can differ by class, subgroup, and deployment period. A well-calibrated system can still be too inaccurate to use, while an accurate one can be dangerously overconfident on its remaining errors.
Scenario - The fictional ticket-routing confidence review: A help desk examines 100 synthetic predictions all labeled 90% confident. Only 70 are correct. Another system gives the same predicted labels but reports 70% confidence. Its accuracy has not changed, yet its confidence is a better description of this group's outcomes.
Analogy - A weather forecast you can keep score on: Across many days forecast as 70% likely to rain, rain should occur about seven times in ten. One dry day does not disprove the forecast.
Where the analogy stops: A single case cannot establish calibration, and grouping choices can hide poor behavior for a particular audience. A language model saying "70% sure" is not automatically a measured probability.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/calibration.webp
Picture description: Blank forecast cards, an unmarked rain collector, and a tray of observation cards with removable raindrop tokens sit on a weather workbench.
Picture caption: A forecast earns trust through repeated outcomes, not one confident statement or one rainy day.
What you see: Forecast. What it represents: The forecast cards represent a stated probability before an event is observed.
What you see: Observed event. What it represents: The rain collector represents what actually happened, separate from the prediction.
What you see: Compare many cases. What it represents: The observation tray represents checking forecast reliability across comparable occasions.
Predict: Can a probability estimate become more honest without changing the observed outcomes?
Try: Compare the forecast probability with the observed positive-event frequency in the display. Decide whether the event is overpredicted or underpredicted, then contrast that forecast with the scenario's confidence in a selected answer.
Explain: The toy forecasts a positive class, not the correctness of whichever class is selected; these are different calibration targets. Reliability is assessed over collections, with enough examples to account for uncertainty. Fit recalibration on separate development data, never the final test. Recheck on new data; calibration is not sampling variety.
Optional math: bin gap = abs(mean forecast - observed frequency)
bin gap: Absolute calibration difference within one group of similar event forecasts.
abs: Absolute value, keeping the magnitude and discarding the sign.
mean forecast: Average predicted probability of the positive event in the group.
observed frequency: Fraction of examples where that positive event actually occurred.
A forecast of 0.9 versus an observed frequency of 0.7 gives a gap of 0.2, or 20 percentage points. If the event being forecast is "my selected answer is correct," its observed frequency is accuracy instead. A single bin is not a complete calibration measure and depends on the sample and grouping.
Common misconception: A calibrated model is necessarily accurate.
Correction: A weak system can honestly report low confidence. Calibration measures probability reliability; accuracy and the costs of mistakes remain separate requirements.
Check your understanding: Two systems make identical predictions, but one reduces confidence from 90% to 70% on a group with 70% correctness. What changed?
Accuracy increased by 20 percentage points. - The selected labels did not change, so neither did correctness. Confidence adjustment and accuracy improvement are different.
Calibration improved for that group, while accuracy stayed the same. - Correct. The reported probability now matches that group's observed rate more closely; verify this on independent data and important subgroups.
Every individual prediction is now guaranteed correct. - A 70% group still contains errors. Calibration describes frequencies, not guarantees for individual cases.
Use it with purpose: Can you auto-approve every case where an assistant writes "95% confident"?
Guidance: Not from wording alone. Validate the confidence signal against labeled outcomes, examine subgroup reliability and error costs, and choose an escalation threshold with independent evidence.
Where to go next:
Agent Uncertainty & Calibration (concept): https://www.openagentschool.org/concepts/agent-uncertainty-calibration - Design abstention and human escalation using validated uncertainty signals.
Human-Agent Delegation (concept): https://www.openagentschool.org/concepts/human-agent-delegation - Use measured uncertainty and error consequences to decide when a person should take over.
Climate Science Sample (science): https://www.openagentschool.org/agents-for-science/sample/climate-science - Practice separating an intervention ranking from uncertainty in the assumptions behind it; this extension requires sign-in.
References:
Guo et al. - On Calibration of Modern Neural Networks: https://arxiv.org/abs/1706.04599

LESSON: Reinforcement Learning
URL: https://www.openagentschool.org/ai-ml-foundations/reinforcement-learning
Also called: RL, policy, reward, return, exploration
Summary: Learn how actions, rewards, and delayed consequences shape a policy.
In everyday words: Reinforcement learning improves a policy, a rule for choosing actions, using rewards from interaction with an environment. Unlike a table of correct answers for every situation, feedback may arrive after several actions. The objective usually concerns cumulative reward, so a tempting immediate reward can conflict with a better long-term outcome.
Why it matters: RL can suit sequential decisions when feedback and a safe learning environment exist. But a reward is a designed proxy for what you want. An agent can improve that proxy while doing the wrong job, especially if evaluation and safety constraints are poorly specified.
Scenario - A fictional stockroom simulator: A simulated cart chooses between a short aisle with a small immediate reward and a longer route that completes a delivery later. The real goal is correct deliveries without collisions. Rewarding distance traveled instead might teach the cart to circle forever. All actions here concern invented states, not a physical robot.
Analogy - Learning a board game from its score: A player tries moves, sees consequences, and gradually favors strategies that lead to better outcomes. A move can look unhelpful now but enable a win later.
Where the analogy stops: A real reward function may omit important harms, and exploration outside a simulator can be costly. RL is not permission to let an untested agent experiment on users.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/reinforcement-learning.webp
Picture description: A wooden game piece faces a branching route, a reachable next board area, and a separate cup of unmarked reward counters.
Picture caption: Actions lead to consequences and feedback; a high game score can still reward the wrong behavior.
What you see: Choose an action. What it represents: The piece at the branch represents a policy choosing among available actions.
What you see: What comes next. What it represents: The next board area represents consequences whose future value must be estimated.
What you see: Reward signal. What it represents: The counters represent designed feedback, not a guarantee that the intended task was achieved.
Predict: If future reward matters less, how should that change an update target built from an immediate reward and a positive next-state value?
Try: Trace the current action-value estimate, immediate reward, discounted next-state estimate, and updated value. Compare the role of the discount with the update rate. Then write one stockroom reward loophole and an independent outcome check.
Explain: An action value estimates cumulative reward after choosing an action. The toy moves one estimate toward a reward plus a discounted estimate of what comes next; it does not roll out or train a policy. Discounting weights future benefit, while the update rate controls how far this estimate moves. Neither validates the reward itself.
Optional math: Q_new = Q_old + alpha * (r + gamma * V_next - Q_old)
Q_new: Updated estimate of cumulative reward for the current action.
Q_old: The action-value estimate before this update.
alpha: Update rate between zero and one: how far to move toward the target.
r: Immediate reward observed after taking the action.
gamma: Discount factor between zero and one weighting the next-state estimate.
V_next: Estimated best action value at the next state; fixed in this nonterminal toy transition.
*: Multiplication.
The target is the immediate reward plus discounted next-state value. Subtracting the old estimate gives the update error; alpha controls how much of it to apply. This is one Q-learning-style update, not a measured full return or a guarantee of learning. A terminal transition needs no next-state contribution.
Common misconception: If reward goes up, the agent must be doing the intended task better.
Correction: A policy may exploit a loophole in the reward. Track the real outcome, constraints, and held-out scenarios separately from the optimization score.
Check your understanding: A cart earns points for movement and learns to circle instead of delivering. What failed?
The policy did not optimize its assigned reward. - It may have optimized that reward very effectively. The issue is that movement was a poor stand-in for successful delivery.
Delayed rewards cannot be used in RL. - RL can handle delayed reward. This failure concerns the meaning of the reward, not a ban on delayed consequences.
The reward proxy did not capture the intended outcome. - Yes. Redesign and validate the reward, add constraints, and independently measure completed deliveries and safety.
Use it with purpose: Your workflow has no reliable outcome signal and costly exploratory actions. Start with RL?
Guidance: First build measurable outcomes and a bounded simulator or offline evaluation. A supervised baseline or explicit workflow may be more justified until safe feedback exists.
Where to go next:
Agent Learning (concept): https://www.openagentschool.org/concepts/agent-learning - Compare reward-driven policy improvement with other ways an agent can adapt.
Grader & Reward Engineering (concept): https://www.openagentschool.org/concepts/grader-reward-engineering - Test whether the score rewards the task you actually want rather than a convenient loophole.
References:
Gymnasium - Basic usage and the reinforcement learning interaction loop: https://gymnasium.farama.org/introduction/basic_usage/

LESSON: RLHF & RLAIF
URL: https://www.openagentschool.org/ai-ml-foundations/rlhf-rlaif
Also called: human feedback, AI feedback, preference learning, reward model
Summary: Trace who supplies feedback, what is preferred, and how that signal can mislead.
In everyday words: RLHF means reinforcement learning from human feedback; RLAIF uses AI-generated feedback. In common language-model pipelines, comparisons between candidate responses help build a preference or reward signal that guides policy improvement. The key distinction is the feedback source, not a guarantee that either source is correct.
Why it matters: Human reviewers can disagree, and AI judges can reproduce biases or favor polished wording over correctness. You need a rubric, provenance, disagreement review, and independent outcome checks. Feedback source and optimization method are separate choices: not every preference-training method uses an online RL loop.
Scenario - The fictional refund-answer review: Two answers respond to a synthetic return request. Answer A is friendly but promises an ineligible refund; answer B politely explains the actual policy. A human reviewer follows a policy rubric. An AI judge prefers A because it sounds helpful. Your task is to resolve the disagreement using evidence, not majority confidence.
Analogy - A judging panel and its scorecard: The panel supplies preferences, and a learner practices to score better under them. Changing the panel from people to software changes the source of judgments.
Where the analogy stops: A panel's approval is not the same as objective truth. If its scorecard rewards style over policy compliance, a better score can mean a worse assistant.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/rlhf-rlaif.webp
Picture description: Two candidate vessels face a blank review clipboard and two judging stations, indicated by a pencil and a closed laptop.
Picture caption: Changing the judge changes the feedback source, not the need for a sound rubric.
What you see: Candidate responses. What it represents: The alternative vessels represent responses compared under the same task.
What you see: Shared rubric. What it represents: The blank scorecard represents the criteria used to judge the alternatives.
What you see: Feedback sources. What it represents: The pencil and laptop cues distinguish human and AI feedback without making either objectively correct.
Predict: If human and AI preferences disagree, does increasing the amount of AI feedback make the labels objectively correct?
Try: Inspect how the toy feedback example combines or compares preference signals. Independently score the two refund answers for policy correctness and helpfulness, keeping those dimensions separate.
Explain: Feedback volume cannot repair a faulty rubric. Audit AI judgments against reviewed examples, retain the source and rubric version, and examine systematic disagreements. The browser compares synthetic signals; it does not train an LLM or demonstrate actual RLHF performance.
Optional math: agreement = matching judgments / reviewed pairs
agreement: Fraction of reviewed response pairs receiving the same preference from two judging sources.
matching judgments: Number of pairs on which the sources agree under the same tie-handling rule.
reviewed pairs: Total number of compared pairs; must be positive.
If sources agree on 16 of 20 pairs, agreement is 80%. This is not 80% truth: both sources might share a bias. Examine why they agree and disagree against task evidence.
Common misconception: AI feedback removes human bias and makes judging objective.
Correction: AI judges inherit training choices, prompt sensitivity, and systematic preferences. Humans still choose the principles, examples, and acceptance criteria; audit both sources.
Check your understanding: An AI judge repeatedly favors charming but policy-breaking answers. What should you do?
Revise and validate the rubric against independently reviewed policy cases. - Correct. Diagnose the grading failure before optimizing against it, and keep a separate evaluation of actual policy outcomes.
Generate more judgments with the unchanged rubric. - More feedback can scale the same systematic mistake. Validate what the judge rewards before expanding volume.
Assume the AI judge is more objective than reviewers by definition. - The source does not guarantee objectivity. Compare both sources against explicit policy evidence and document disagreement.
Use it with purpose: You can afford only a small human review budget. Replace it entirely with an AI judge?
Guidance: Use reviewed examples to validate and audit AI feedback, prioritize disagreements and high-impact cases, and preserve an independent final test. Savings are useful only if the feedback remains fit for purpose.
Where to go next:
Learning from Human & AI Feedback (concept): https://www.openagentschool.org/concepts/human-ai-feedback - Design source-aware feedback with rubrics, provenance, and disagreement review.
Grader & Reward Engineering (concept): https://www.openagentschool.org/concepts/grader-reward-engineering - Validate what human and AI judges reward before their preferences steer a policy update.
LLM Post-Training (concept): https://www.openagentschool.org/concepts/llm-post-training - Separate the source of feedback from the optimization procedure that uses it to change model behavior.
Self-Reflection (pattern): https://www.openagentschool.org/patterns/self-reflection - Contrast inference-time critique and revision with parameter-changing preference training.
References:
Ouyang et al. - Training language models to follow instructions with human feedback: https://arxiv.org/abs/2203.02155
Bai et al. - Constitutional AI: Harmlessness from AI Feedback: https://arxiv.org/abs/2212.08073

LESSON: Post-Training
URL: https://www.openagentschool.org/ai-ml-foundations/post-training
Also called: SFT, DPO, preference optimization, supervised fine-tuning
Summary: Choose a justified behavior update after a model has learned broad patterns.
In everyday words: Post-training adapts a pretrained model toward desired behavior. Supervised fine-tuning learns from demonstrations; preference optimization learns from comparisons; reinforcement learning can optimize a reward signal. These methods have different data and feedback needs, and there is no universal requirement to use every method in sequence.
Why it matters: A polished training pipeline can optimize the wrong problem. Diagnose whether the failure comes from missing information, unclear instructions, faulty tools, or a persistent model behavior. Start with the least costly intervention that can close the measured gap, then compare against a frozen baseline.
Scenario - The fictional policy assistant release review: An assistant answers in the wrong format and also quotes an outdated refund window. Demonstrations might help the recurring format behavior, but retrieving the current policy addresses the stale fact more directly. The team must not count an improved style score as proof that refund decisions became correct.
Analogy - Job-specific practice after general education: A broadly educated recruit practices the organization's tasks, observes good examples, and receives feedback. The practice focuses behavior rather than starting all learning from scratch.
Where the analogy stops: A model is not a human recruit. Training can introduce regressions or overfit examples, and updating weights is not a reliable substitute for a current source of facts.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/post-training.webp
Picture description: Plain reference books, a worked packing example, and a separate unfinished parcel with a blank inspection card sit at a training bench.
Picture caption: Focused practice builds on prior learning; the new behavior still needs an independent check.
What you see: Broad prior learning. What it represents: The reference books represent capabilities developed before task-specific adaptation.
What you see: Focused practice. What it represents: The worked parcel represents demonstrations or feedback directed at a particular behavior.
What you see: Check the new task. What it represents: The unfinished parcel and inspection card represent evaluation, not an assumed training gain.
Predict: Could a candidate improve on the training objective while becoming worse on an important behavior you did not reward?
Try: Inspect the post-training comparison and identify which outputs are modeled assumptions. Write a baseline, one proposed intervention, and two independent acceptance checks for the policy assistant.
Explain: This browser comparison is a simplified illustration, not measured post-training lift; no real LLM training occurs. In a real experiment, freeze datasets and the baseline, validate the grader, record the method, and evaluate regressions and costs before promotion.
Optional math: measured lift = candidate score - baseline score
measured lift: Observed difference on the same independent evaluation and metric.
candidate score: Score of the proposed updated system.
baseline score: Score of the frozen comparison system.
For a higher-is-better metric, positive lift favors the candidate on that metric only. Report sample size and uncertainty, and check whether other metrics or failure costs worsened. A toy score is not empirical lift.
Common misconception: DPO, RLHF, and supervised fine-tuning are interchangeable names for the same recipe.
Correction: They differ in objectives, data, and optimization. DPO directly uses preference pairs without the classic separate reward-model plus online RL procedure; supervised fine-tuning imitates demonstrations.
Check your understanding: The assistant uses last year's price but otherwise follows instructions. What is the best first intervention?
Run the most expensive reward optimization immediately. - The observed gap is missing current information. First test a reliable lookup and confirm whether a persistent behavior problem remains.
Train on the final test cases until the score improves. - That contaminates the evaluation. Development examples and final release evidence must remain separate.
Supply current prices through an authoritative lookup and evaluate again. - Yes. Fix the diagnosed information gap with the smallest justified intervention before considering parameter updates.
Use it with purpose: A candidate wins on average but makes more policy-breaking refunds. Promote it?
Guidance: Reject or constrain it until the safety-critical regression is addressed. Promotion requires predefined outcome checks, independent evidence, acceptable costs, and a rollback plan, not only a higher aggregate score.
Where to go next:
LLM Post-Training (concept): https://www.openagentschool.org/concepts/llm-post-training - Compare demonstration, preference, and reward-based methods against a diagnosed behavior gap.
Fine-Tuning (concept): https://www.openagentschool.org/concepts/fine-tuning - Choose a parameter-update method only after simpler interventions fail to close a measured gap.
Atomic LLM Training (concept): https://www.openagentschool.org/concepts/atomic-llm-training - Bridge the toy gradient calculation to the data, objective, update, and evaluation steps of actual training.
Model Improvement Engineering (course): https://www.openagentschool.org/courses/model-improvement-engineering - Build a baseline-to-candidate evidence packet instead of equating a toy curve with real training success.
References:
Rafailov et al. - Direct Preference Optimization: Your Language Model is Secretly a Reward Model: https://arxiv.org/abs/2305.18290
Ouyang et al. - Training language models to follow instructions with human feedback: https://arxiv.org/abs/2203.02155

LESSON: Synthetic Data
URL: https://www.openagentschool.org/ai-ml-foundations/synthetic-data
Also called: generated training data, data provenance, model collapse, distillation
Summary: Generate useful practice while preserving provenance and independent evidence.
In everyday words: Synthetic data is produced by a generator, simulator, or explicit rules rather than collected directly as the target real-world observations. It can supply controlled variations and rare practice cases. Its usefulness depends on source validity, coverage, label quality, and how well it represents the deployment task.
Why it matters: Generated examples can cheaply repeat the same mistake or omit the same uncommon cases. Fluent text is not a validated label. Record where examples came from, which rules or model produced them, what review occurred, and whether any source overlaps the final evaluation.
Scenario - A fictional lost-property practice set: A transit team creates invented lost-item messages to train a routing assistant. The generator produces hundreds of suitcase requests but few mobility-aid cases, and sometimes invents a pickup policy. You need broader practice without importing made-up policy labels or copying the evaluation questions into training.
Analogy - A flight simulator with a documented world: A simulator can repeat rare conditions safely and vary one factor at a time. Its value comes from controlled practice tied to meaningful conditions.
Where the analogy stops: Success inside a simulator is not proof of success outside it. If weather, instruments, or objectives are unrealistic, practice can teach habits that fail in the real world.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/synthetic-data.webp
Picture description: A stationary simulator cockpit, a separate scenario binder with weather props, and a distinct runway model illustrate designed practice and outside conditions.
Picture caption: Generated practice is useful only when its assumptions and independent checks are trustworthy.
What you see: Generated practice. What it represents: The simulator represents examples produced under controlled rules rather than observed deployment events.
What you see: Document assumptions. What it represents: The scenario binder represents provenance, generation rules, and known coverage limits.
What you see: Independent check. What it represents: The separate runway model represents validation against outside conditions, not proof from the same simulator.
Predict: If generated volume doubles but all new examples repeat the same template, has useful coverage doubled?
Try: Compare original rows, generated rows, and predictions on clean held-out examples. Explore how biased duplicates move the fitted offset. Then design a lost-property case that fills a genuine gap and record the source rule needed to verify its label.
Explain: The toy fits one offset, showing how repeated biased labels affect independent error; it does not run a review filter. In a real pipeline, separate count, acceptance, diversity, and correctness. Deduplicate, review labels, and keep an independent test that was not used to prompt or tune the generator.
Optional math: acceptance rate = accepted examples / generated examples
acceptance rate: Fraction of generated examples that pass the stated review process.
accepted examples: Count retained after the documented checks.
generated examples: Total produced before filtering; must be greater than zero.
This is a review-planning metric, not an output of the offset-fitting demo. Keeping 60 of 100 examples gives 60% acceptance. It measures the filter outcome, not truth or diversity; a weak filter can accept nearly everything and still produce poor training material.
Common misconception: Synthetic data is automatically private, correct, and unlimited fresh knowledge.
Correction: A generator can reproduce sensitive or copyrighted source material, invent labels, or repeat narrow patterns. Review permitted use, provenance, duplicates, and independent performance; generated does not mean risk-free.
Check your understanding: A generator rewrites the final test questions into training examples. Are the splits still independent?
Yes, because the words are different. - Paraphrases can preserve the same answer and case structure. Content lineage matters, not just exact string matching.
No; keep evaluation cases out of the generation and tuning process. - Correct. Document source families, prevent cross-split derivatives, and evaluate on independent cases not used to steer generation.
Yes, if an AI judge approves the paraphrases. - A quality judgment does not remove the relationship to test material. Approval and split independence are separate checks.
Use it with purpose: Synthetic-only scores look excellent. Is the dataset ready for a production claim?
Guidance: Validate on independently collected or independently authored representative cases with reviewed labels. Compare real failure modes and subgroups, and describe remaining gaps rather than calling generator agreement real-world success.
Where to go next:
Synthetic Training Data & Distillation (concept): https://www.openagentschool.org/concepts/synthetic-training-data - Build source-aware generation, filtering, and training splits with independent evaluation.
Synthetic Evaluation Data (concept): https://www.openagentschool.org/concepts/synthetic-evaluation-data - Design evaluation cases independently of the generator and training examples being assessed.
Materials Discovery Sample (science): https://www.openagentschool.org/agents-for-science/sample/materials-discovery - Contrast simulated candidate scores with evidence and follow-up validation; this extension requires sign-in.
References:
Gebru et al. - Datasheets for Datasets: https://arxiv.org/abs/1803.09010
Shumailov et al. - The Curse of Recursion: Training on Generated Data Makes Models Forget: https://arxiv.org/abs/2305.17493

LESSON: Distribution Shift
URL: https://www.openagentschool.org/ai-ml-foundations/distribution-shift
Also called: data drift, covariate shift, label shift, prior shift, concept drift
Summary: Distinguish changed inputs, changed class frequencies, and changed rules.
In everyday words: Distribution shift means the data-generating distribution differs between development and use. Covariate shift changes input frequencies while assuming the outcome rule given inputs stays fixed. Label or prior shift changes class frequencies while assuming inputs within each class stay distributed the same. Concept drift changes the relationship between inputs and outcomes.
Why it matters: Different changes call for different responses. More mobile traffic is not the same problem as a new refund policy. An input alert can reveal change without proving quality fell; outcome labels and slice-level evaluation help determine whether predictions, confidence, or decisions need adjustment.
Scenario - Three changes at a fictional event desk: First, more users submit short mobile messages. Second, cancellations become a larger share of requests after a storm. Third, the organizer changes which tickets are refundable. These suggest input shift, prior shift, and concept drift respectively, but their formal assumptions must be checked; real incidents can combine all three.
Analogy - A shop facing a new season: The customers arriving may change, the mix of products they want may change, or the store's return rules may change. A clerk needs to identify which change occurred before adapting.
Where the analogy stops: Real shifts overlap and labels may arrive late. A histogram alone cannot tell whether the correct response to a given input changed.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/distribution-shift.webp
Picture description: Wet-weather arrival props, a changed seasonal product display, and a separate return-policy binder show three different changes in one shop.
Picture caption: Who arrives, what is requested, and which rule applies can change separately.
What you see: Changed inputs. What it represents: The entrance props represent changed incoming conditions, which need not imply worse predictions.
What you see: Changed demand mix. What it represents: The seasonal display represents changed class frequencies; formal prior-shift assumptions still need checking.
What you see: Changed rule. What it represents: The replacement policy insert represents a changed correct outcome for an otherwise similar request.
Predict: If average message length changes but labeled performance stays stable, is automatic retraining necessarily justified?
Try: Compare the reference and shifted distributions in the toy display. Classify each event-desk change, then name one input measure and one outcome measure you would monitor separately.
Explain: A visible shift is a prompt to investigate, not proof of failure. Examine affected slices, delayed outcomes, and calibration. The demonstration changes a simplified distribution; it cannot diagnose every type of drift or establish production quality.
Optional math: P_new(X, Y) != P_reference(X, Y)
P_new: Probability distribution in the new operating period or population.
P_reference: Probability distribution in the comparison data.
X: Input features, such as message content and channel.
Y: Target outcome or class, such as the correct request route.
!=: Is not equal to.
The joint input-outcome distribution changes. Covariate, prior, and concept changes describe different parts or assumptions of that change; class imbalance at one point in time is not itself evidence of drift.
Common misconception: Any data-drift alert proves the model is wrong and must be retrained.
Correction: Inputs can change without harming the task, and outcomes can change without an obvious input alert. Investigate labeled quality, slice coverage, policy changes, and operational impact first.
Check your understanding: The same refund request now has a different correct answer because the policy changed. What is the clearest diagnosis?
Only class imbalance, with no change in the input-outcome relationship. - The correct answer for the same request changed. That is more than a different proportion of classes.
Only a tokenizer change. - The scenario describes a policy change, not text segmentation. Trace the decision rule and its effective date.
Concept drift: the correct outcome given the request changed. - Correct. Update the authoritative policy and evaluate affected cases; a shift in class frequencies may also occur, but it is not the whole diagnosis.
Use it with purpose: Traffic shifts to a new language and an input monitor alarms. What is your next step?
Guidance: Sample and review outcomes for that language, check retrieval coverage and calibration, and use a safe fallback if evidence is insufficient. Choose data, model, or workflow changes based on the diagnosed gap.
Where to go next:
Production ML Systems (concept): https://www.openagentschool.org/concepts/production-ml-systems - Link data changes to delayed labels, release decisions, canaries, and rollback.
Agent Observability (concept): https://www.openagentschool.org/concepts/agent-observability - Combine input shifts, traces, and outcome measures to investigate which part of a deployed workflow changed.
Observability & EvalOps (skill): https://www.openagentschool.org/ai-skills#module/observability - Monitor inputs and task outcomes separately so an alert leads to a diagnosis, not automatic retraining.
References:
Lipton, Wang, and Smola - Detecting and Correcting for Label Shift with Black Box Predictors: https://arxiv.org/abs/1802.03916

LESSON: Quantization
URL: https://www.openagentschool.org/ai-ml-foundations/quantization
Also called: low precision, integer inference, bit width, rounding error
Summary: Trade numerical precision for storage or execution benefits, then measure the real task.
In everyday words: Quantization represents numerical values using a smaller set of levels, often reducing the bits used for model weights or activations. Values are scaled, rounded, and sometimes clipped to fit a representable range. A reconstructed value is an approximation, not generally the original number.
Why it matters: Lower precision can reduce memory and sometimes improve speed or energy use on supported hardware. But actual gains depend on kernels, metadata, memory traffic, and workload. Small weight errors can have uneven effects on outputs, so numerical error is not the same measure as end-to-end quality loss.
Scenario - A fictional offline inventory helper: A warehouse wants a local model to fit on a constrained device. You compare two precision settings on invented weights, then plan real measurements for item extraction, latency, and peak memory. Rare item codes matter more than a small average numerical error suggests.
Analogy - A ruler with fewer marks: Measuring with a coarse ruler rounds lengths to nearby marks. Fewer marks simplify representation but lose detail, especially when small differences matter.
Where the analogy stops: A neural network combines many rounded values through nonlinear operations. The ruler's local rounding error does not directly predict an assistant's factual accuracy or hardware speed.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/quantization.webp
Picture description: Fine and coarse unnumbered rulers sit beside identical wooden samples, with a gauge block illustrating a length between coarse marks.
Picture caption: Fewer marks lose numerical detail; the effect on the real task must still be measured.
What you see: Finer levels. What it represents: The closely spaced ruler grooves represent more available numerical values.
What you see: Coarser levels. What it represents: The widely spaced grooves represent fewer levels and larger rounding steps.
What you see: Detail between marks. What it represents: The gauge block represents a value that cannot be preserved exactly on the coarse grid.
Predict: Will reducing the number of representable levels always preserve the original values? Does a small average error prove task quality is unchanged?
Try: Compare original and reconstructed values in the quantization example. Observe rounding and any range limits, then choose one rare inventory case you would include in an independent quality evaluation.
Explain: The toy exposes numerical approximation, not a compressed live LLM. Storage estimates may omit scales and other runtime overhead. Measure the deployed configuration on target hardware and task data; do not convert a weight-error percentage directly into an accuracy claim.
Optional math: reconstructed = lower + s * round((w - lower) / s)
reconstructed: Approximate value after rounding to a uniform grid.
lower: Value at the first grid level; the toy grid starts at -1.
s: Positive spacing between adjacent representable levels.
w: Original scalar value, assumed inside the usable range for this simplified formula.
round: Choose a nearest integer using the implementation's tie rule.
*: Multiplication.
With a grid starting at -1 and spacing 0.1, a value of 0.26 reconstructs as 0.3. This formula assumes the value is within range. Real quantizers also define clipping, scales per tensor or group, and sometimes zero points. More bits permit more levels but do not guarantee a particular task-quality improvement.
Common misconception: A 1% weight reconstruction error means exactly 1% less task accuracy.
Correction: Numerical error and task quality measure different things. Sensitivity varies by layer, input, and task; evaluate the actual outputs, especially important rare cases.
Check your understanding: A quantized candidate has tiny average weight error. What else is needed before deployment?
Nothing; low weight error guarantees speed and answer quality. - Neither follows automatically. Hardware may lack efficient support, and a small numerical change can affect sensitive cases.
Task-quality, latency, and memory measurements on the target configuration. - Correct. Judge the operational tradeoff with representative and high-impact cases, not just a local rounding metric.
Multiply the numerical error by the model size to obtain accuracy. - There is no general conversion from parameter error to task accuracy. Run an independent task evaluation instead.
Use it with purpose: A smaller model fits the device but misreads rare inventory codes more often. Keep it?
Guidance: Compare the failures with operational costs and acceptance criteria. Try a different precision configuration or fallback, and retain the candidate only if measured quality and resource use meet the requirements.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-quantization-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-quantization-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-quantization-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-quantization-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Production ML Systems (concept): https://www.openagentschool.org/concepts/production-ml-systems - Evaluate low-precision candidates as complete serving configurations with regression gates.
Agent Cost Optimization (concept): https://www.openagentschool.org/concepts/agent-cost-optimization - Compare measured resource savings with quality costs rather than treating bit width as the whole budget.
Edge Agents (concept): https://www.openagentschool.org/concepts/edge-agent - Connect reduced precision to local-device memory constraints and evaluate behavior on the actual hardware.
Cost & Performance (skill): https://www.openagentschool.org/ai-skills#module/cost-perf - Measure quality, memory, latency, and cost together instead of assuming fewer bits always mean faster inference.
Model Routing & Cost Control (course): https://www.openagentschool.org/courses/model-routing-cost-control - Place compression beside routing and fallback options when choosing an affordable serving strategy.
References:
Jacob et al. - Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference: https://arxiv.org/abs/1712.05877

LESSON: KV Cache
URL: https://www.openagentschool.org/ai-ml-foundations/kv-cache
Also called: key-value cache, attention state, inference cache
Summary: Reuse attention work while keeping an honest budget for the memory it occupies.
In everyday words: A key-value (KV) cache holds intermediate attention tensors for tokens a model has already processed. Keys help a new query locate earlier information; values supply the information to mix. During ordinary causal generation, earlier keys and values can be reused instead of recomputed at every step. They are numerical state, not a dictionary of remembered answers.
Why it matters: Long conversations and many simultaneous requests can consume substantial memory even when model weights stay unchanged. A faster computation strategy can therefore reduce the number of requests that fit on a device. Count live token state separately from weights before promising concurrency.
Scenario - A fictional sewing-room workbench: An invented tailoring assistant serves two conversations, each with 1,024 cached tokens. Its fictional architecture has 32 layers and eight key/value heads of width 128, stored in two-byte elements. You must decide whether doubling conversation length is a harmless change or doubles this part of the memory bill.
Analogy - Keep the cut pieces, not a finished outfit: A tailor keeps already-cut cloth pieces in trays so the next seam does not require tracing and cutting everything again. Extra garments in progress need extra trays. The pieces are reusable intermediate work, not finished outfits waiting to be handed to anyone.
Where the analogy stops: Cloth does not compute attention. Cached keys and values are tied to model state, token positions and preceding context; they cannot be reused just because two conversations sound similar.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/kv-cache.webp
Picture description: Garment templates and uncut linen sit beside two separate trays of already-cut cloth panels on a tailoring bench.
Picture caption: Keep intermediate work for the next step; more work in progress occupies more space.
What you see: Work before reuse. What it represents: Templates and uncut fabric stand for computations that produce attention state.
What you see: Saved attention state. What it represents: Cut panels represent reusable keys and values, not a completed answer.
What you see: Another live sequence. What it represents: The additional tray represents separate working memory for another active request.
Predict: The initial two conversations use 256 MiB of KV state. If cached tokens per sequence doubles from 1,024 to 2,048, does state memory double even though the model weights do not change?
Try: Read the initial 128 KiB per token and 256 MiB total. Set Cached tokens per sequence to 2048, then restore 1024 and change Active sequences from 2 to 4. Finally change Key/value heads per layer from 8 to 4.
Explain: Either doubling produces 512 MiB; halving the head count with four sequences returns to 256 MiB. These are dense tensor-size calculations, not measured speedups. The model still computes each new query and its attention over available state.
Worked engineering example: Budget the fictional tailoring deployment
1. Compute 2 * 32 * 8 * 128 * 2 = 131,072 bytes per token. Why: Every layer stores both key and value elements; key/value heads determine this size.
2. Multiply by 1,024 tokens and two active sequences. Why: These conversations have separate dense state, giving 268,435,456 bytes.
3. Convert to 256 MiB and budget weights and buffers separately. Why: A correct tensor subtotal is not the total device-memory requirement.
4. Test the intended length and concurrency together before release. Why: Admission limits should match the busiest supported case, not an idle single chat.
Outcome: The initial cache subtotal is 256 MiB; either doubled length or doubled concurrency makes it 512 MiB, before other allocations.
Tradeoff - Retain longer histories: Gain: Earlier context remains available for attention. Cost or risk: More state occupies memory and attending to long histories still costs work.
Tradeoff - Use a supported lower-precision cache: Gain: Fewer bytes can reduce memory pressure. Cost or risk: Conversion cost and answer-quality changes require implementation-specific evaluation.
Tradeoff - Share compatible prefixes: Gain: Identical prefixes may avoid duplicate state. Cost or risk: Identity, isolation, eviction and sharing behavior become part of correctness.
Before real use: Record cache dtype, layer/head layout and supported window behavior for the deployed version.
Before real use: Load-test maximum active lengths and concurrency; include allocator and workspace headroom.
Before real use: Check model, token positions, template and tenant isolation before cross-request reuse.
Before real use: Observe eviction, preemption and out-of-memory errors without logging private tensor contents.
Your turn: Keep the initial architecture, but use four sequences of 2,048 tokens. The per-token state is already 128 KiB. Finish the total.
Hint: Multiply the initial 256 MiB by both independent doubling factors.
Compare your reasoning: 1,024 MiB of dense KV state, excluding weights and runtime overhead. This is four times the initial subtotal, not a measured device requirement.
Optional math: M = 2 * L * H * D * T * B * S
M: Dense key/value cache size in bytes.
2: Two stored tensors: keys and values.
L: Layer count, fixed at 32 in this fictional architecture.
H: Key/value heads per layer, not necessarily query heads.
D: Elements per head, fixed at 128 here.
T: Cached tokens in each sequence.
B: Bytes per stored tensor element.
S: Number of equally long active sequences; * means multiplication.
Divide bytes by 1,048,576 for mebibytes (MiB). This deliberately excludes model weights, allocator padding and temporary buffers. Sharing prefixes, sharding, compressed caches and sliding-window attention need different accounting.
Common misconception: A KV cache remembers previous answers and can serve them to similar questions.
Correction: KV state supports attention computation. Exact-prefix prompt caching may reuse compatible state across requests; semantic caching instead reuses a stored answer. These have different correctness and permission boundaries.
Check your understanding: Four conversations double in length with no prefix sharing. What happens to their dense KV state?
It doubles even though the weights are unchanged. - Yes. Token count multiplies the stored state for every active sequence. Weights and attention state are different memory budgets.
It stays fixed because inference does not train. - Frozen weights do not imply fixed working memory. Each added cached token contributes keys and values in the modeled layers.
It becomes a searchable collection of complete answers. - The tensors support new attention calculations; they do not become a semantic answer store as the conversation grows.
Use it with purpose: A deployment fits its weights but runs out of memory during long concurrent chats. What do you inspect?
Guidance: Measure active sequence lengths, cache allocation, head layout and other runtime buffers. Consider admission limits, safe context reduction or supported cache strategies; do not blame weight loading alone or silently discard required context.
Where to go next:
AI Infrastructure (concept): https://www.openagentschool.org/concepts/ai-infrastructure - Separate model-weight capacity from the working memory required to serve live requests.
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Choose useful context while respecting the memory cost of retaining it.
References:
Hugging Face Transformers - How caching works: https://huggingface.co/docs/transformers/main/en/cache_explanation

LESSON: Prompt Caching
URL: https://www.openagentschool.org/ai-ml-foundations/prompt-caching
Also called: prefix caching, exact-prefix reuse, cached input
Summary: Reuse an identical prompt prefix without confusing a cache hit with a reused answer.
In everyday words: Prompt caching reuses computation for an unchanged beginning of a rendered prompt. It often preserves compatible key-value attention state from earlier requests. The matching prefix must be exact under the serving system, including relevant instructions, tools and settings. The model still processes new input and generates an output.
Why it matters: Stable instructions placed before changing user content can save input work. A timestamp or changed tool definition near the beginning may destroy reuse after that point. Reuse changes neither the total context requirement nor the need to authorize a request.
Scenario - A fictional community print shop: A synthetic print-shop assistant sends 1,000 stable instruction tokens and 200 new order tokens. Only the first 800 stable tokens match a retained compatible entry. The operator wants to call this an 80% whole-request hit; you need to distinguish prefix coverage from total input coverage.
Analogy - Reuse the unchanged printing setup: A print shop keeps a physical layout jig for the unchanged top of an invitation, then positions fresh pieces for a new event. Keeping the jig saves setup, but the shop must still produce the new invitation.
Where the analogy stops: A physical jig can sometimes be patched in the middle. Causal attention state depends on earlier tokens, so matching a later paragraph does not restore a prefix broken earlier.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/prompt-caching.webp
Picture description: A print-shop layout jig holds a fixed floral border, with loose decorative pieces and a fresh blank invitation sheet nearby.
Picture caption: An unchanged beginning can reuse setup; a new response still has to be made.
What you see: Exact stable prefix. What it represents: The fixed jig represents compatible work for the unchanged beginning of a prompt.
What you see: New suffix. What it represents: Loose pieces represent new input that has not been processed in the cached prefix.
What you see: Generate anew. What it represents: The fresh sheet represents an output still to be generated, not an answer-cache hit.
Predict: With 1,000 stable tokens, a 0.8 exact-prefix fraction and 200 new tokens, will 80% of all input be reused?
Try: Inspect Reused input tokens=800, Fresh input tokens=400 and Reused input share about 66.67%. Change Exact-prefix fraction to 1, then turn Matching state still retained to 0. Restore it and turn Model, scope and settings match to 0.
Explain: A full stable-prefix match reuses 1,000 of 1,200 tokens, about 83.33%, not the new suffix. Either retention or compatibility failure makes all 1,200 tokens fresh. The fraction describes exact prefix length, never semantic resemblance.
Worked engineering example: Account for one repeated print-shop request
1. Identify the longest exact matching prefix: 800 tokens. Why: Only the beginning before the first change can reuse causal prefix state.
2. Check model/template identity, permission scope and retained state. Why: An identical string alone does not prove the entry is eligible or resident.
3. Subtract 800 from 1,200 total input tokens. Why: The remaining 400 includes 200 unmatched stable tokens and the 200-token suffix.
4. Count output generation separately and inspect usage metrics. Why: A prefix hit does not cache the answer or guarantee a particular bill.
Outcome: 800 reused and 400 fresh input tokens; 66.67% input reuse in the toy, with no claimed provider price or latency.
Tradeoff - Keep stable content early: Gain: Longer unchanged prefixes may be reusable. Cost or risk: Careless reordering can alter instruction priority or task behavior.
Tradeoff - Retain cache entries longer: Gain: Later repeated requests may hit. Cost or risk: Retention costs and data-governance requirements vary by provider and version.
Before real use: Version rendered templates, tools, model identifiers and multimodal inputs in cache identity.
Before real use: Verify tenant isolation and allowed retention rather than treating a cache key as authorization.
Before real use: Observe cache-read tokens and cold/warm latency separately.
Before real use: Check current provider minimum-prefix, retention and pricing rules; do not copy toy counts into configuration.
Your turn: The stable prefix is 1,000 tokens and fully matches; the suffix grows to 1,000 tokens. Fill in reused, fresh and reuse share.
Hint: The suffix is new even though the prefix matches perfectly.
Compare your reasoning: 1,000 reused, 1,000 fresh, 50% input reuse. If compatibility fails, reused becomes zero and all 2,000 are fresh.
Optional math: R = floor(P * f) * E; F = P + U - R
R: Reused input tokens in this accounting exercise.
P: Stable-prefix token count.
f: Fraction of that prefix known to match exactly.
E: Eligibility: one when compatible and retained, zero otherwise.
F: Fresh input tokens that still need processing.
U: New suffix tokens; floor rounds down to a whole token and * multiplies.
Reused share divides R by the entire input P + U, not just the stable part. This model omits provider minimum lengths, cache blocks and pricing. With no input, the displayed share is defined as zero rather than dividing by zero.
Common misconception: A prompt-cache hit returns the same answer and makes cached tokens disappear from context.
Correction: A hit reuses compatible input computation, not the final response. All input still occupies context. Sampling, changed suffixes and current instructions can produce different outputs.
Check your understanding: A tool definition changes near the start of an otherwise repeated prompt. What is safe to assume?
The unchanged ending guarantees the previous full cache hit. - Matching suffix text does not repair earlier prefix changes. The cached state includes the effect of the preceding rendered context.
Reuse after the changed prefix must be reconsidered. - Correct. Inspect the rendered prefix and relevant settings, then measure actual cache-read usage rather than assuming a hit.
Switch to semantic matching to share its attention tensors. - Similar wording does not make attention state compatible. Semantic answer caching is a different mechanism with different checks.
Use it with purpose: Your cache-hit rate fell after adding request timestamps. What do you change first?
Guidance: Inspect where timestamps enter the rendered prompt. Keep genuinely stable instructions and tool definitions before variable content when semantically appropriate, but never reorder away necessary policy. Compare provider cache-read metrics before claiming savings.
Where to go next:
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Arrange stable and variable context while preserving the meaning and authority of instructions.
Agent Cost Optimization (concept): https://www.openagentschool.org/concepts/agent-cost-optimization - Measure cache reads and actual billed usage instead of assuming every repeated request is cheaper.
References:
OpenAI - Prompt caching and rendered-prefix identity: https://developers.openai.com/api/docs/guides/prompt-caching
vLLM - Automatic Prefix Caching design and isolation: https://docs.vllm.ai/en/latest/design/prefix_caching/

LESSON: Semantic Caching
URL: https://www.openagentschool.org/ai-ml-foundations/semantic-caching
Also called: answer cache, similarity cache, response reuse
Summary: Reuse a similar question’s answer only when meaning, permission and freshness still line up.
In everyday words: Semantic caching searches stored question-and-answer entries using a numerical representation of meaning, called an embedding. It can return a previous answer for a sufficiently similar request instead of generating a new one. Similarity is only a candidate-selection signal: access rights, policy versions, factual freshness and task equivalence still matter.
Why it matters: A near match can be dangerously different: "Can I cancel?" and "Can I cancel after delivery?" may require opposite answers. An impressive hit rate can hide wrong or cross-user answers. Evaluate false hits alongside saved work.
Scenario - A fictional pottery studio help shelf: A synthetic studio assistant stores a public answer about firing-class cancellations. The new query has distance 0.2 from that entry, below a 0.3 limit. The entry is ten minutes old with a thirty-minute lifetime and matching permission/version scope. A private booking question must not borrow another member’s answer just because its wording resembles it.
Analogy - Borrow a matching instruction card from the right shelf: A pottery studio keeps reusable instruction cards near sample clay pieces. A similar-looking piece suggests which card to inspect, but a sealed members-only drawer and an expiry marker still determine whether the card may be used.
Where the analogy stops: Physical resemblance is not a semantic embedding, and an unexpired card can still be incorrect. Neither a similarity threshold nor a time limit proves answer equivalence.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/semantic-caching.webp
Picture description: Similar clay bowls and blank instruction cards sit near a locked drawer and a sand timer on a pottery studio shelf.
Picture caption: Resemblance finds a candidate; permission and freshness still decide whether it may be reused.
What you see: Candidate resemblance. What it represents: Similar bowls represent related questions, not guaranteed answer equivalence.
What you see: Permission boundary. What it represents: The closed drawer represents explicit access scope that similarity cannot override.
What you see: Freshness window. What it represents: The timer represents an expiry rule; an unexpired answer can still be incorrect.
Predict: The initial entry passes all gates and costs 20 ms to look up. Will making it older than its lifetime preserve that hit because the wording remains close?
Try: Confirm Eligible cache hit=1 and Modeled response time=20 ms. Set Entry age to 30, equal to Time to live. Restore age 10, then turn Permissions and version match to 0. Finally restore scope and lower Maximum allowed distance to 0.1.
Explain: Each change independently causes a miss: expired age, denied scope, then insufficient similarity. A miss costs 620 ms, the 20 ms lookup plus 600 ms fresh generation. These are supplied illustrative timings, not measured cache quality or provider performance.
Worked engineering example: Approve a reusable studio answer
1. Compare distance 0.2 with the evaluated threshold 0.3. Why: This finds a candidate, not proof of equivalent questions.
2. Require the same allowed audience and current policy version. Why: A public class rule and a private booking answer have different access scopes.
3. Check age 10 is strictly below lifetime 30. Why: At age 30 the entry is expired under this policy.
4. Return the eligible answer and audit a sample of near hits. Why: All gates can pass while a semantic false match still slips through.
Outcome: Initial lookup returns the eligible answer in the toy’s 20 ms; expiry or a scope failure instead takes the 620 ms fresh path.
Tradeoff - Loosen similarity matching: Gain: More paraphrases may reuse answers. Cost or risk: Opposite intent, exceptions or changed entities can create false hits.
Tradeoff - Shorten time to live: Gain: Reduces stale-answer exposure. Cost or risk: More misses pay both lookup and generation work.
Tradeoff - Partition by user or policy: Gain: Preserves access and version boundaries. Cost or risk: Smaller sharing pools reduce hit rates and complicate invalidation.
Before real use: Filter by authenticated scope and current policy/data version before returning any candidate.
Before real use: Test near-duplicate negatives, dates, negation and personalized questions, not only paraphrase positives.
Before real use: Document expiry and whether access refreshes lifetime; verify the installed library behavior.
Before real use: Track false hits, stale hits and sensitive-content exclusions alongside latency and hit rate.
Your turn: Keep distance 0.2, threshold 0.3 and matching scope, but set lifetime to zero. Fill in hit eligibility and time.
Hint: Freshness requires age strictly below lifetime; an age of zero is not below zero.
Compare your reasoning: No hit at any permitted age. With the initial 600 ms generation cost, the fresh path costs 620 ms including lookup.
Optional math: hit = (d <= t) AND (a < L) AND s; time = 20 + (1 - hit) * G
d: Supplied query-to-entry distance; lower is closer here.
t: Maximum acceptable distance for this chosen metric.
a: Entry age in minutes.
L: Time to live in minutes; equality means expired.
s: One when permission and version checks pass, zero otherwise.
hit: One only if all three conditions pass; AND means all must hold.
G: Fresh generation time in milliseconds; time includes a fixed 20 ms lookup.
The exercise deliberately limits distances to 0..1; real embedding metrics may have other ranges, such as cosine distance up to two. A threshold is evaluated for the exact embedding model and use case, not transferred as a universal confidence score.
Common misconception: A sufficiently close embedding guarantees the stored answer is safe and correct.
Correction: Negation, dates, identity and policy exceptions may preserve similarity while changing the answer. A safe cache needs explicit scope filters, invalidation and evaluated false-hit behavior; some tasks should bypass it.
Check your understanding: A very close cached answer belongs to another customer. What should the cache do?
Return it because high similarity overrules access filters. - Similarity is not authorization. This would turn a performance optimization into a cross-customer data disclosure.
Raise the distance threshold until every user gets a hit. - A looser threshold admits more candidates but cannot grant permission. It can also increase wrong-answer reuse.
Reject that entry and follow the authorized fresh-answer path. - Yes. Scope is a hard eligibility condition. Only search or generate using information the current customer is permitted to access.
Use it with purpose: Should today’s account balance use the same semantic cache as evergreen opening hours?
Guidance: Usually bypass shared answer reuse for personalized or rapidly changing facts. If caching is justified, bind identity, authoritative data version and a reviewed freshness policy, and evaluate near-miss queries before enabling it.
Where to go next:
Retrieval Engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Calibrate similarity and metadata filtering with realistic near-match counterexamples.
Agent Security (concept): https://www.openagentschool.org/concepts/agent-security - Treat cache lookup as a data-access operation with explicit permission boundaries.
References:
Redis Vector Library - Cache LLM Responses, TTL and access filters: https://docs.redisvl.com/en/stable/user_guide/03_llmcache.html

LESSON: Effective Context
URL: https://www.openagentschool.org/ai-ml-foundations/effective-context
Also called: usable context, context capacity, long-context recall
Summary: Distinguish information that fits from information a model can reliably use.
In everyday words: Context capacity is the amount of tokenized material a model can accept for a request under its limits. Effective context is the information it can actually use reliably for the task. Capacity is a limit; useful recall is empirical behavior that depends on the model, information placement, distractions and the question.
Why it matters: A larger window can fit irrelevant material while burying the decisive evidence. Even when every needed fact is present, a model may miss or misapply it. Evaluate task success and evidence use rather than treating a token-limit advertisement as a recall guarantee.
Scenario - A fictional museum volunteer desk: A synthetic museum guide has an 8,000-token window, with 2,000 reserved for instructions, tools and output. Six useful exhibit cards and four irrelevant cards each cost 500 tokens. In the deliberately poor packing order, irrelevant cards go first. You must keep essential evidence available without pretending this demonstrates model understanding.
Analogy - A desk can hold papers without making them useful: An archive desk has limited clear space. Put unrelated folders down first and fewer relevant exhibit cards fit. Even a card fully visible on the desk can be overlooked by someone answering a question.
Where the analogy stops: Desk area models only packing. Human attention is not a transformer, and this exercise cannot measure language-model recall, positional bias or reasoning.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/effective-context.webp
Picture description: Bulky folders fill part of a desk organizer, leaving room for a few artifact cards while other matching cards sit outside it.
Picture caption: A limited workspace can exclude useful evidence; fitting it still does not guarantee it will be used.
What you see: Distractors first. What it represents: Bulky folders represent context consuming capacity before useful evidence is selected.
What you see: Evidence available. What it represents: Cards inside the organizer represent retained evidence, not demonstrated model recall.
What you see: Evidence left out. What it represents: Cards outside the organizer represent useful material lost under the toy packing rule.
Predict: Initially twelve whole-card slots fit and all six useful cards remain. If you add six distractors while keeping the window unchanged, how many useful cards survive this ordering?
Try: Confirm Whole-card slots=12 and Relevant cards retained=6. Change Distractor cards placed first from 4 to 10. Then raise Context capacity from 8000 to 10000 while keeping the other controls unchanged.
Explain: Ten distractors leave room for only two useful cards: 33.33% of required evidence. A 10,000-token capacity provides sixteen slots and restores all six. The displayed percentage is retained evidence, not an invented model recall score.
Worked engineering example: Find the museum’s missing evidence
1. Reserve 2,000 of 8,000 tokens, leaving 6,000. Why: Instructions, tools and output must fit alongside exhibit material.
2. Divide by 500 to obtain twelve whole-card slots. Why: Whole-card packing prevents a half-visible card from counting as complete evidence.
3. Admit ten distractors, then only two of six relevant cards. Why: The demonstrated ordering is intentionally poor and makes lost evidence visible.
4. Compare better selection with a larger capacity in a real evaluation. Why: Both can restore evidence availability, but neither proves the model will use it correctly.
Outcome: The poor ordering retains two useful cards at 8,000 tokens. Either removing four distractors or adding 2,000 tokens of capacity can retain all six.
Tradeoff - Send more source material: Gain: Potentially useful details remain available. Cost or risk: More tokens cost memory and processing and may add distractions.
Tradeoff - Retrieve or summarize selectively: Gain: Concentrates relevant evidence within a smaller budget. Cost or risk: Selection or compression can omit the exact exception the learner needs.
Before real use: Use the deployed model/tokenizer limits and account for tool and output reservations.
Before real use: Test facts at beginning, middle and end with varied distractors and multi-step questions.
Before real use: Measure source-supported answer quality separately from retained-token percentages.
Before real use: Surface omitted required evidence and retain provenance when summarizing or compacting context.
Your turn: At the initial 8,000 capacity and 2,000 reserve, use ten distractors. You know there are twelve slots. How many distractors must you remove to fit all six relevant cards?
Hint: Six of twelve slots must be available for useful evidence.
Compare your reasoning: Remove four distractors, leaving six distractors plus six relevant cards. All evidence fits; reliable recall still needs a separate model evaluation.
Optional math: slots = floor(max(0, C - R) / k); kept = min(E, max(0, slots - D))
C: Total context capacity in tokens.
R: Combined instruction, tool and output reserve.
k: Positive number of tokens per whole card.
E: Number of relevant evidence cards.
D: Distractor cards admitted before relevant ones.
slots: Number of whole cards that fit after reserving space.
kept: Relevant cards retained; floor rounds down, min/max choose smaller/larger values.
The packing rule never splits a card and reports reserve overflow separately. Real systems should surface a budget failure or deliberately select material rather than silently truncating important evidence. Retention is only a prerequisite for using evidence.
Common misconception: If all documents fit inside the window, the model will recall every relevant fact.
Correction: Fitting is not reliable use. Long-context research demonstrates position-sensitive failures for evaluated models and tasks, but does not establish one universal failure curve for all current models. Test your own questions and positions.
Check your understanding: All six evidence cards are retained. Which conclusion does the experiment support?
The evidence is available, but answer quality is still unmeasured. - Correct. This exercise counts cards, not model reasoning. Test evidence attribution and task answers before claiming effective recall.
The model has perfect recall on this question. - No model ran. Even a real model receiving every card might overlook a fact, use the wrong version or draw an unsupported conclusion.
Larger capacity always improves the model’s accuracy. - Capacity admits more tokens; it does not distinguish helpful context from distractions. Quality must be measured separately.
Use it with purpose: A support agent accepts the full manual but misses a key exception. Should you only buy a bigger window?
Guidance: First inspect evidence selection, ordering, explicit questions and source versions. Compare full-context and targeted retrieval on held-out cases, including exceptions in different positions. Increase capacity only if it solves a measured availability bottleneck.
Where to go next:
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Select useful evidence and reserve space without mistaking window size for understanding.
Agent Testing & Benchmarks (concept): https://www.openagentschool.org/concepts/agent-testing-benchmarks - Measure evidence use across positions and distractions on deployment-matched tasks.
References:
Liu et al. - Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172
Anthropic - Context windows and input/output accounting: https://platform.claude.com/docs/en/build-with-claude/context-windows

LESSON: Speculative Decoding
URL: https://www.openagentschool.org/ai-ml-foundations/speculative-decoding
Also called: assisted generation, draft-and-verify decoding, speculative inference
Summary: Let a fast drafter propose tokens while a target model remains responsible for accepting them.
In everyday words: Speculative decoding asks a cheaper proposal mechanism to draft several next tokens, then has the target model verify them together. A usable prefix can advance generation by multiple tokens per target pass. Draft agreement affects efficiency; it does not replace target verification or certify factual truth.
Why it matters: Drafting takes time and memory too. When few proposed tokens survive, or the server is already busy, the extra work can outweigh saved target passes. Measure target-equivalent behavior and end-to-end latency on the intended workload, not just draft accuracy.
Scenario - A fictional bakery order checker: A synthetic order assistant proposes four tokens. The greedy target agrees through token three and disagrees at position four. Drafting costs two milliseconds per token, and a verification pass costs twenty milliseconds. You compare the verified result with twenty milliseconds per ordinary target step, not with an unverified draft.
Analogy - An apprentice lines up items for a checker: At a bakery counter, an apprentice sets out a proposed row of pastries. A checker approves from the beginning until the first wrong item, replaces it, and sends only that approved prefix onward. Items beyond the mismatch wait for a new check.
Where the analogy stops: Pastries do not have conditional token probabilities. This analogy and experiment model greedy prefix agreement; sampled decoding requires a distribution-preserving acceptance and correction procedure.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/speculative-decoding.webp
Picture description: A proposed pastry row includes a different pastry; a reference bun and tongs separate it from a clean tray of matching buns.
Picture caption: A draft proposes a sequence; the target checks the prefix before anything is accepted.
What you see: Draft proposal. What it represents: The proposed row represents tokens that cost work even if later rejected.
What you see: Target verification. What it represents: The reference station represents checking agreement with the target, not checking factual truth.
What you see: Verified prefix. What it represents: The separate tray represents only the accepted beginning and target correction or bonus.
Predict: Four drafted tokens with a mismatch at position four yield three accepted draft tokens plus one target correction. Does paying 28 ms beat four ordinary 20 ms target steps?
Try: Inspect Accepted draft prefix=3, Emitted tokens=4, Draft plus verification=28 ms and Ordinary target time=80 ms. Set First target mismatch position to 1. Then set it to 9, meaning no mismatch within the allowed draft.
Explain: The initial modeled speed ratio is 80/28, about 2.86. A first-position mismatch emits only the correction: 20/28, about 0.71, a slowdown. With no mismatch, four accepted tokens plus one target bonus give five emitted tokens and 100/28, about 3.57. None is a real benchmark.
Worked engineering example: Compare equal amounts of verified output
1. Accept only the three tokens before mismatch position four. Why: Autoregressive proposals after a rejected token do not retain a valid prefix.
2. Add the target correction to obtain four emitted tokens. Why: The target, not the apprentice draft, determines the corrected next step.
3. Charge all four draft tokens: 4 * 2 + 20 = 28 ms. Why: Rejected work still consumed resources.
4. Compare with 4 * 20 = 80 ms ordinary target work. Why: The baseline must produce the same amount of verified output.
Outcome: The initial accounting ratio is about 2.86; moving the mismatch to the first position changes it to a slowdown of about 0.71.
Tradeoff - Propose a longer draft: Gain: More accepted tokens can amortize a target pass. Cost or risk: An early mismatch wastes more proposal work and state.
Tradeoff - Use a more capable drafter: Gain: It may match the target more often. Cost or risk: Extra compute and memory may erase the acceptance benefit.
Tradeoff - Enable at high concurrency: Gain: Some implementations can still benefit. Cost or risk: Contention and batch scheduling may favor ordinary decoding instead.
Before real use: Verify method support for the deployed model, tokenizer, sampling settings and engine version.
Before real use: Test target output/distribution guarantees separately from factual task quality.
Before real use: Measure accepted prefix lengths, draft time, verification time and memory across traffic classes.
Before real use: Include high-load and early-rejection counterexamples; retain a rollback to ordinary decoding.
Your turn: Keep four proposed tokens and two milliseconds per draft token, but use a 100 ms verification pass with no mismatch. Is five-token emission faster than the 20 ms ordinary target?
Hint: Full agreement emits five tokens; compare 100 ms ordinary work with drafting plus verification.
Compare your reasoning: No. The speculative cost is 108 ms versus 100 ms ordinary work: ratio about 0.93. Perfect draft agreement does not guarantee a speedup.
Optional math: A = min(K, m - 1); E = A + 1; speed ratio = E * t / (K * d + v)
A: Number of consecutive accepted draft tokens.
K: Number of proposed draft tokens.
m: First mismatch position; nine means none for drafts of at most eight.
E: Emitted tokens, including a target correction or bonus.
t: Ordinary target-step time in milliseconds.
d: Draft time per proposed token in milliseconds.
v: Target verification-pass time in milliseconds; min chooses the smaller value.
The numerator and denominator produce the same number of verified tokens under the toy’s greedy convention. A ratio below one means added cost outweighs saved steps. Stop tokens, context limits, shared-device contention and variable pass durations are omitted.
Common misconception: A more accurate draft always speeds up any serving workload.
Correction: A slow or memory-hungry draft can erase its benefit, and high-load scheduling can change the tradeoff. Target agreement is not factual accuracy. Supported methods and output-equivalence guarantees depend on implementation, sampling and version.
Check your understanding: The target rejects the first proposed token. What is valid in the demonstrated greedy scheme?
Keep later draft tokens that look plausible in isolation. - Later tokens were proposed under the rejected prefix. Skipping the mismatch would no longer follow the target’s verified sequence.
Emit the target correction and discard this draft suffix. - Yes. No draft prefix survived. The target still advances one token, but the full drafting cost was paid and may cause a slowdown.
Accept the draft to preserve the advertised speed ratio. - That abandons target verification. A speed claim does not justify changing the decoding contract without explicitly choosing another method.
Use it with purpose: A draft model improves acceptance but consumes half the serving device. Enable it by default?
Guidance: Compare verified output behavior, tail latency, throughput and memory under realistic concurrent traffic. Include draft and verification costs. Keep a measured fallback or disable speculation where it loses; do not assume a universal speedup.
Where to go next:
AI Infrastructure (concept): https://www.openagentschool.org/concepts/ai-infrastructure - Account for draft memory, verification passes and scheduling contention together.
Agent Testing & Benchmarks (concept): https://www.openagentschool.org/concepts/agent-testing-benchmarks - Compare target-equivalent behavior and latency using controlled workload-specific measurements.
References:
Hugging Face - Assisted Generation and greedy verification: https://huggingface.co/blog/assisted-generation
vLLM - Speculative Decoding methods and workload caveats: https://docs.vllm.ai/en/latest/features/speculative_decoding/

LESSON: Token Budgets
URL: https://www.openagentschool.org/ai-ml-foundations/token-budgets
Also called: context budget, output reservation, token accounting
Summary: Reserve space for instructions, tools and output before deciding how much input to send.
In everyday words: A token budget allocates a model request’s limited capacity among content, message templates, tool definitions or results, and generated output. Tokens are tokenizer-specific text or modality units, not simply words. Context capacity, maximum output, billed usage and rate-limit accounting are related but different constraints.
Why it matters: A request that appears to fit on visible user text can fail once a tool schema or output reservation is included. Cutting the output cap indiscriminately can also produce incomplete answers. Make the whole request visible before deciding what to shorten.
Scenario - A fictional picnic packing list: A synthetic picnic-planning assistant has an 8,192-token capacity. Its user and retrieved content total 5,000 tokens, templates and tools add 800, output reserves 2,000, and an application safety margin is 256. You must decide whether another 500-token recipe fits without squeezing out the promised response.
Analogy - Pack the basket, including its containers: A picnic basket must hold food, bulky containers and room for items collected later. Counting only the food makes a full basket look deceptively empty. A deliberate spare corner is different from the space a real container already occupies.
Where the analogy stops: Tokens are not physical volume and the model may not use its full output allowance. Providers can impose separate output or multimodal limits even when this simple capacity sum fits.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/token-budgets.webp
Picture description: An open picnic basket contains food parcels, a bulky reusable container and an intentionally empty compartment.
Picture caption: Content, its packaging and reserved answer space must fit in the same budget.
What you see: Content tokens. What it represents: Food parcels represent user input and retrieved material.
What you see: Template and tools. What it represents: The bulky container represents overhead that consumes capacity even when users do not see it.
What you see: Output reservation. What it represents: The empty compartment represents room reserved for generation, not tokens already produced.
Predict: The initial budget needs 8,056 tokens and has 136 left. Will adding 500 content tokens still fit?
Try: Confirm Required tokens=8056, Signed headroom=136 and Room for output=2136. Increase User and retrieved content from 5000 to 5500. Then reduce Requested output reserve from 2000 to 1600.
Explain: The extra recipe makes required tokens 8,556 and headroom -364, so the budget fails. Reducing output reserve to 1,600 produces 8,156 required and 36 headroom. This repairs arithmetic only; you still must check that a shorter answer can satisfy the task.
Worked engineering example: Keep the picnic answer inside its budget
1. Sum 5,000 content and 800 known overhead tokens. Why: The visible request alone omits 800 real input tokens.
2. Add 2,000 output tokens and 256 safety tokens. Why: The model needs room to answer, and the application has chosen a separate uncertainty margin.
3. Compare 8,056 with 8,192: only 136 remain. Why: A small positive margin cannot absorb a new 500-token recipe.
4. Review the requested answer before reducing its reservation. Why: Arithmetic fit is not a guarantee of a complete response.
Outcome: The original request fits; adding 500 tokens does not. Reserving 1,600 output tokens restores 36 headroom only if that answer budget is sufficient.
Tradeoff - Reserve more output: Gain: Allows longer or more detailed responses. Cost or risk: Leaves less capacity for input; separate output limits may still bind.
Tradeoff - Compress context: Gain: Makes room for selected information. Cost or risk: Summarization can lose exact wording, provenance or exceptions.
Before real use: Count fully rendered requests using the deployed model’s tokenizer or official counting endpoint.
Before real use: Include tools, history, modality and reasoning rules for the specific model/version.
Before real use: Detect budget overflow and output-length termination distinctly from ordinary completion.
Before real use: Compare predicted counts with reported usage; keep capacity, spend and rate-limit budgets separate.
Your turn: Keep the initial content, overhead and safety margin. Room for output is 2,136 tokens. Can you reserve 2,200?
Hint: Subtract requested output from available output room; do not subtract overhead twice.
Compare your reasoning: No: headroom is -64 tokens. Reserve at most 2,136 in the arithmetic model, and still respect the provider’s separate output limit.
Optional math: H = C - (I + O + G + S)
H: Signed headroom in tokens; negative means the request budget exceeds capacity.
C: Configured context capacity for the model.
I: User and retrieved content tokens.
O: Rendered template and tool overhead tokens.
G: Requested generated-output reservation.
S: Additional application safety margin, separate from known overhead.
A nonnegative H passes this one accounting gate. Count with the deployed tokenizer and rendered format; include reasoning tokens according to that model’s rules. Cached input still counts toward context, and reserving G does not mean G tokens will actually be generated or billed.
Common misconception: Only the visible user message counts, and cached instructions are free space.
Correction: The rendered request can include system instructions, history, tools and multimodal material. Cached input still occupies context even when its price or processing cost differs. Inspect usage and model-specific limits.
Check your understanding: The sum exceeds capacity by 364 tokens. Which response preserves a clear engineering contract?
Drop the last evidence silently and report a successful budget. - Silent truncation can remove the very exception the answer needs. Report the budget issue and deliberately choose what can be removed.
Ignore tool definitions because users cannot see them. - Visibility is not token accounting. Tool schemas and template overhead still consume request capacity under the serving format.
Revise input or output allocation explicitly, then recheck task needs. - Correct. Restore a valid budget without pretending the shortened request or response is automatically equivalent to the original task.
Use it with purpose: A new tool schema pushes a working assistant over its context limit. What do you do?
Guidance: Count the rendered change, remove unnecessary schema or irrelevant history where safe, and reserve enough output for the task. Surface failures instead of silently clipping instructions. Recheck separate model output and request-size limits.
Where to go next:
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Allocate context by usefulness while preserving necessary instructions and evidence.
Agent Cost Optimization (concept): https://www.openagentschool.org/concepts/agent-cost-optimization - Separate reserved capacity from actual token usage and provider-specific billing.
References:
Anthropic - Context windows, tools, thinking and output limits: https://platform.claude.com/docs/en/build-with-claude/context-windows

LESSON: Prefill & Decode
URL: https://www.openagentschool.org/ai-ml-foundations/prefill-and-decode
Also called: prompt processing, autoregressive decode, time to first token
Summary: Separate reading the prompt from generating the answer, and separate both from waiting in line.
In everyday words: Prefill processes the supplied context and builds attention state. Decode then advances the output token by token using that state. Queueing happens before service begins and is a separate source of delay. Time to first token (TTFT) measures when the response starts; completion latency measures when it finishes.
Why it matters: A long pause before any text may come from queueing, input work or buffering, while a slow trickle afterward may come from decoding. The appropriate fix depends on the phase. Streaming exposes early progress; it does not automatically shorten the total computation.
Scenario - A fictional neighborhood radio announcement: A synthetic announcement assistant waits two seconds, processes 1,000 input tokens at 1,000 tokens per second, then produces 101 output tokens. In this simplified timing convention, its first token arrives at the end of prefill; the remaining 100 arrive at fifty tokens per second.
Analogy - Read the script before speaking it: An empty radio booth has a waiting tray outside, an open script on the desk, and a microphone ready for the announcement. Waiting for the booth, reading the material and speaking are separate stretches on the listener’s clock.
Where the analogy stops: A language model is not reading a human script aloud. Actual timing includes hardware scheduling and network buffers, and telemetry systems may assign first-token computation to different phases.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/prefill-and-decode.webp
Picture description: Blank script folders wait outside an open radio booth; an open binder and microphone occupy separate places inside.
Picture caption: Waiting, processing the prompt and generating output are different parts of the clock.
What you see: Queue wait. What it represents: The outside tray represents delay before model service begins.
What you see: Prefill. What it represents: The open binder represents processing the supplied context before output begins.
What you see: Decode. What it represents: The microphone represents progressive output, not a guarantee of constant token speed.
Predict: Initially the first token arrives at three seconds and completion at five. If you shorten output from 101 tokens to one, does the initial three-second wait disappear?
Try: Read Prefill seconds=1, Time to first token=3, Remaining decode seconds=2 and Completion seconds=5. Set Output tokens including the first to 1. Restore 101, then set Queue wait to 0.
Explain: One output token leaves TTFT at three seconds but completion also becomes three. Removing the queue instead gives TTFT one second and completion three. The experiment distinguishes the two improvements rather than attributing both to a faster model.
Worked engineering example: Explain the radio assistant’s five-second response
1. Record the two-second queue separately. Why: No prefill work is assumed to occur during this wait.
2. Compute 1,000 / 1,000 = one second of prefill. Why: Under this convention the first token is ready when prefill ends.
3. Compute (101 - 1) / 50 = two seconds of remaining decode. Why: Counting all 101 tokens again would double-count first-token work.
4. Report TTFT three seconds and total five seconds. Why: The listener’s initial wait and time to the full answer answer different questions.
Outcome: Removing the queue reduces both timings by two seconds; shortening output to one token changes only completion time in the toy.
Tradeoff - Shorten input: Gain: May reduce prefill work. Cost or risk: Can remove necessary evidence or instructions.
Tradeoff - Stream output: Gain: Makes progress visible before completion. Cost or risk: Partial answers, buffering and cancellation need explicit handling.
Before real use: Define timestamp boundaries for arrival, service start, first token and completion.
Before real use: Measure queue, prefill, inter-token gaps and network buffering across input/output lengths.
Before real use: Check phase interference under concurrency instead of assuming constant rates.
Before real use: Track cancellations and truncated streams separately from successfully completed answers.
Your turn: Keep 101 output tokens at fifty tokens per second and the two-second queue, but double input to 2,000 tokens. Complete TTFT and total.
Hint: Prefill is now two seconds; remaining decode is still two.
Compare your reasoning: TTFT is four seconds and completion is six. No output-token or queue change was needed to worsen both timings.
Optional math: TTFT = Q + I / P; T = TTFT + (N - 1) / D
TTFT: Time to first token in seconds under the stated convention.
Q: Queue wait before prefill begins, in seconds.
I: Input token count.
P: Assumed prefill rate in input tokens per second.
N: Output token count, including the first token.
D: Assumed later-token decode rate in tokens per second.
T: Time from request arrival to complete output, in seconds.
Subtracting one avoids counting the first token twice: this toy includes it at the end of prefill. Rates are supplied constant assumptions, not measured hardware performance. Real prefill, decoding and queueing can interfere under load.
Common misconception: Streaming makes generation finish sooner, so it fixes all latency problems.
Correction: Streaming can reveal output earlier and allow cancellation, but the same number of tokens may take the same total computation. Diagnose queue, prefill, decode and transport separately before choosing a remedy.
Check your understanding: In the initial experiment, only output length falls from 101 to one token. Which timing changes?
Completion falls from five to three seconds; TTFT stays three. - Yes. No later tokens remain to decode, but queueing and prefill still occur before the first token appears.
TTFT falls to zero because only one token is requested. - The service still waits two seconds and processes the input for one second. Output length does not remove those phases in this model.
The prefill rate doubles automatically. - Prefill rate is a separate supplied control. The experiment does not invent a hardware-speed change when output length changes.
Use it with purpose: Users wait before the first character, but the rest streams quickly. Should you shorten every answer?
Guidance: Measure queue wait, prefill, first-token arrival and buffering first. Shorter answers may reduce total time but not this initial bottleneck. Consider appropriate capacity, prefix reuse or smaller relevant input based on the measured phase.
Where to go next:
AI Infrastructure (concept): https://www.openagentschool.org/concepts/ai-infrastructure - Connect serving phases to compute, memory and scheduling bottlenecks.
Agent Observability (concept): https://www.openagentschool.org/concepts/agent-observability - Instrument first-token and completion timing without collapsing queueing into model execution.
References:
Hugging Face - Streaming and perceived versus completion latency: https://huggingface.co/docs/text-generation-inference/en/conceptual/streaming
Hugging Face Transformers - How caching works during generation: https://huggingface.co/docs/transformers/main/en/cache_explanation

LESSON: Continuous Batching
URL: https://www.openagentschool.org/ai-ml-foundations/continuous-batching
Also called: iteration-level batching, inflight batching, dynamic request admission
Summary: Refill finished inference slots instead of waiting for every request in a group to finish.
In everyday words: Continuous batching lets a serving engine change the active request set between generation steps. Finished requests leave and waiting requests can enter without waiting for a whole fixed group. It improves resource use for unequal output lengths, subject to memory, prefill and fairness constraints. This is inference scheduling, not a training batch that updates weights.
Why it matters: A short answer should not leave its processing slot idle while an unrelated long answer finishes. But filling every slot is not the only objective: new long prompts can delay existing streams, and memory pressure can cause preemption. Measure user latency as well as aggregate work.
Scenario - A fictional bicycle service stand: Four synthetic jobs A, B, C and D arrive together. A and C need two output ticks each; B and D need six. Two slots are available, each producing one token per shared tick of ten milliseconds. Compare fixed groups A/B then C/D with immediately refilling a free slot.
Analogy - Use the free repair stand: Two bicycle stands hold jobs of different lengths. A waiting bicycle can move into the first stand that opens rather than waiting for both mechanics to finish their original pair. The stand is useful only while a job occupies it.
Where the analogy stops: GPU work is not two independent mechanics. This fixed-tick simulation excludes prefill, memory contention and the way real batch size can change per-step duration.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/continuous-batching.webp
Picture description: One bicycle occupies a repair stand, a second stand is empty, and a waiting bicycle rests in a nearby floor rack.
Picture caption: A waiting request can enter a freed slot while another request is still running.
What you see: Still active. What it represents: The occupied stand represents a longer inference request still generating.
What you see: Slot freed. What it represents: The empty stand represents capacity released by a completed request.
What you see: Admit waiting work. What it represents: The waiting bicycle represents a request that need not wait for the original group to finish.
Predict: Does C have to wait until B finishes at tick six, or can it start when A finishes at tick two?
Try: Inspect initial static completion 120 ms and continuous completion 100 ms. Read continuous finishes A=20, B=60, C=40 and D=100 ms. Change Simultaneous decoding slots to 1, then to 4.
Explain: With two slots, C runs at ticks two through four and D at four through ten while B finishes at six. One slot makes both methods take 160 ms; four slots make both take 60 ms. The benefit comes from refilling otherwise idle slots, not a universal acceleration factor.
Worked engineering example: Schedule the four bicycle jobs
1. Start A and B at tick zero. Why: Both slots are free and FIFO means first in, first out.
2. Start C at tick two when A finishes. Why: Continuous admission does not wait for B’s remaining four ticks.
3. Start D at tick four when C finishes. Why: The same slot is reusable again while B continues on the other slot.
4. Compare last completion at tick ten with static tick twelve. Why: The shared tick is ten milliseconds in both schedules, so completion is 100 versus 120 ms.
Outcome: Continuous admission saves two ticks and four idle slot-ticks in this exact four-job simulator; one or four slots remove that advantage.
Tradeoff - Admit as soon as a slot opens: Gain: Can reduce idle capacity and waiting. Cost or risk: New prefill work can interfere with active decoding in real engines.
Tradeoff - Limit active sequences: Gain: Protects cache memory and per-request responsiveness. Cost or risk: May leave resources underused or increase queue wait.
Tradeoff - Favor short requests: Gain: May improve typical completion time. Cost or risk: Long requests can starve without an explicit fairness policy.
Before real use: Benchmark the actual prompt/output length mix and arrival pattern, not only equal-length jobs.
Before real use: Measure queue wait, inter-token gaps, completion tails and throughput together.
Before real use: Observe cache occupancy, preemption and memory failures at configured admission limits.
Before real use: Verify deployed engine/version scheduling support; TGI documentation now marks that project as maintenance-mode.
Your turn: Set both A/C and B/D lengths to six tokens with two slots. Predict static and continuous completion before moving the controls.
Hint: The first pair frees both slots at the same tick.
Compare your reasoning: Both take twelve ticks, or 120 ms at the initial tick duration. Equal-size pairs leave no refill opportunity between their completion times.
Optional math: idle slot-ticks = S * T - W
S: Number of simultaneous decoding slots.
T: Ticks until the final request completes under the chosen schedule.
W: Sum of required output tokens across the four jobs; one token uses one slot-tick.
idle slot-ticks: Unused capacity across that schedule; * means multiplication.
Initially W is sixteen. Static groups use twelve ticks, so idle capacity is 2 * 12 - 16 = eight slot-ticks. Continuous admission needs ten ticks, giving four idle slot-ticks. This is an exact calculation for the declared simulator, not a real-engine utilization metric.
Common misconception: Continuous batching is a training optimization that changes weights whenever a request finishes.
Correction: These requests use fixed model weights. The scheduler changes which inference sequences share an iteration. Training batches have a different purpose: combining examples for a parameter-update procedure.
Check your understanding: Why do the two scheduling methods tie when this experiment uses four slots?
Every output length becomes identical. - A and C still need two tokens while B and D need six. Changing slot count does not change the declared job lengths.
All four jobs can start immediately, so no waiting job can refill a slot. - Correct. Both schedules finish when the longest initial job finishes. Continuous admission helps only when there is useful waiting work to admit.
The model learns to generate all answers in one tick. - No training occurs and one token per active job per tick remains the simulator rule. The tie is a scheduling fact.
Use it with purpose: Throughput rises after enabling aggressive admission, but active streams develop long pauses. Keep it?
Guidance: Compare inter-token and tail latency under the intended mix of prompt and output lengths. Tune admission, prefill scheduling and memory limits to the service objective instead of treating full occupancy as the sole success criterion.
Where to go next:
AI Infrastructure (concept): https://www.openagentschool.org/concepts/ai-infrastructure - Connect inference scheduling to shared compute and cache-memory constraints.
Agent Deployment (concept): https://www.openagentschool.org/concepts/agent-deployment - Validate admission behavior and latency objectives before enabling a scheduling change.
References:
Hugging Face Text Generation Inference - Continuous batching features: https://huggingface.co/docs/text-generation-inference/en/index

LESSON: Latency & Throughput
URL: https://www.openagentschool.org/ai-ml-foundations/latency-and-throughput
Also called: tail latency, request throughput, latency percentiles
Summary: Measure how long each user waits as well as how much work the service completes.
In everyday words: Latency is elapsed time for a request or phase. Throughput is completed work per unit of time. Time to first token (TTFT) describes when output starts, while completion latency describes when it ends. Tail percentiles reveal slower experiences that an average may hide.
Why it matters: A busy system can complete many requests while some users wait far too long. A high tokens-per-second number can also reflect long answers rather than more useful completed tasks. Declare the unit, observation window, workload and treatment of failures before comparing services.
Scenario - A fictional ferry ticket counter: A synthetic service completes twenty ticket-help requests within a ten-second observation window. Nineteen take two seconds each; one takes twelve seconds and began before the window. Its first-token times are set to one fifth of completion time for this exercise. No real traffic was measured.
Analogy - Many tickets served, one traveler still waiting: At a ferry counter, a tray of completed ticket envelopes represents the work finished during a period. A separate waiting basket represents individual experience. A large completed tray cannot tell you how long the slowest traveler waited.
Where the analogy stops: The photograph is not a queue measurement, and the synthetic timings do not derive from arrival or service rates. This lesson computes descriptive statistics, not a capacity forecast.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/latency-and-throughput.webp
Picture description: A tray of completed ticket envelopes, a single envelope beside a bell, and an unfinished envelope with a sand timer occupy a ferry counter.
Picture caption: Count completed work and inspect individual waits; neither tells the whole story alone.
What you see: Throughput. What it represents: The finished tray represents work completed within a declared observation period.
What you see: First response. What it represents: The bell and single envelope represent when a particular request first produces output.
What you see: Slow completion. What it represents: The unfinished envelope represents an individual wait that an average can hide.
Predict: With nineteen two-second requests and one twelve-second request, can a 2.5-second mean conceal a twelve-second tail?
Try: Read throughput 2 requests/second, mean 2500 ms, p95 2000 ms, p99 12000 ms and p99 first token 2400 ms. Raise only Duration of the twentieth request to 20000. Then change the observation window from 10 to 20 seconds.
Explain: The longer request makes the mean 2900 ms and p99 20000 ms, while p95 stays 2000 ms. Doubling the window halves throughput to one request/second without changing any per-request durations. The controls describe a cohort; they do not simulate a causal load change.
Worked engineering example: Read the ferry counter’s timing report honestly
1. Count twenty completions over ten seconds: two per second. Why: Throughput counts completed requests, not average latency or inferred concurrency.
2. Average nineteen times 2,000 plus one times 12,000: 2,500 ms. Why: The mean combines everyone but dilutes the lone long wait.
3. Read sorted ranks nineteen and twenty: 2,000 and 12,000 ms. Why: Nearest-rank p95 and p99 expose different tail positions in this small sample.
4. Report sample size and first-token timing separately. Why: Twenty synthetic cases and an assumed fraction cannot establish real-world tail reliability.
Outcome: Two completions per second coexist with a twelve-second completion tail; a single average would conceal that experience.
Tradeoff - Increase batching or concurrency: Gain: Can increase aggregate completed work. Cost or risk: Queueing, first-token delay and tail latency may worsen.
Tradeoff - Report finer tail percentiles: Gain: Makes rare slow experiences more visible. Cost or risk: Reliable estimates require many observations and careful histogram resolution.
Before real use: State whether throughput counts requests, input tokens, output tokens or successful tasks.
Before real use: Collect first-token and completion distributions with sample counts and error/cancellation rates.
Before real use: Avoid survivor bias: inspect unfinished requests and warm-up/observation boundaries.
Before real use: Compare the same quality target, prompt/output distribution, hardware and concurrency.
Your turn: Keep nineteen 2,000 ms requests, but set the twentieth to 100 ms. Which value becomes p99?
Hint: Sort first; the twentieth request is no longer the slowest request.
Compare your reasoning: p99 is 2,000 ms, p95 is also 2,000 ms, and the mean is 1,905 ms. A label like "twentieth request" does not define its latency rank.
Optional math: throughput = N / W; mean = sum(L_i) / N; p_q = sorted(L)[ceil(q * N)]
N: Number of completed requests, fixed at twenty.
W: Observation-window duration in seconds.
L_i: Completion latency of request i in milliseconds.
sum: Addition of all twenty completion latencies.
q: Percentile fraction, such as 0.95 or 0.99.
p_q: Nearest-rank percentile: sort ascending and take the one-based position ceil(q * N).
ceil: Round upward to a whole rank; * means multiplication.
With twenty requests, p95 is the nineteenth sorted value and p99 is the twentieth. This tiny sample has coarse tails and cannot establish a service guarantee. A longer observation window can contain completions from requests started earlier.
Common misconception: The service with highest throughput necessarily gives each user the fastest response.
Correction: Batching and concurrency can raise aggregate completions while worsening first-token delay or tail completion time. Compare distributions at the same workload and quality, including failures and requests that never finished.
Check your understanding: One request gets much slower but nineteen stay unchanged. Why might p95 remain unchanged here?
Percentiles ignore every slow request by definition. - They do not. p99 selects the slowest value in this twenty-case sample. Different ranks expose different parts of the distribution.
The service secretly completed more requests per second. - The completion count and observation window did not change. A slower duration cannot by itself imply higher throughput.
Nearest-rank p95 selects the nineteenth value, below the lone outlier. - Yes. p95 alone misses this one-in-twenty extreme. Report appropriate tails and sample counts rather than hiding it behind a single statistic.
Use it with purpose: A benchmark reports only aggregate output tokens per second. Is it enough to choose an interactive assistant?
Guidance: Request first-token and completion distributions, request success and quality, input/output lengths, concurrency, arrival rate and test duration. Compare equal tasks and include unfinished work; a large aggregate token number is not a user-experience verdict.
Where to go next:
Agent Observability (concept): https://www.openagentschool.org/concepts/agent-observability - Track first-token, completion and failure signals rather than relying on one average.
Agent Testing & Benchmarks (concept): https://www.openagentschool.org/concepts/agent-testing-benchmarks - Compare performance under declared workload, measurement-window and quality conditions.
References:
Hugging Face - Streaming and end-user latency: https://huggingface.co/docs/text-generation-inference/en/conceptual/streaming
Hugging Face TGI v3 - Benchmark methodology and boundary effects: https://huggingface.co/docs/text-generation-inference/en/conceptual/chunking

LESSON: Model Routing
URL: https://www.openagentschool.org/ai-ml-foundations/model-routing-basics
Also called: LLM routing, model selection policy, cost-quality routing
Summary: Choose among measured, permitted and available models without treating cheapness as quality.
In everyday words: Model routing selects a candidate model for a request using task requirements and operational constraints. A large language model (LLM) router might use rules or learned predictions, but a responsible decision also checks permission, availability and evaluated task quality. Cost optimization happens within those boundaries, not instead of them.
Why it matters: The cheapest model may fail a crucial task class, while the most expensive model may be unavailable or forbidden for the data. Routing can save resources, but misclassification, stale evaluation results and silent fallback can sacrifice quality. An explicit no-route outcome is sometimes the correct result.
Scenario - A fictional neighborhood repair dispatcher: A synthetic dispatcher compares a small assistant with an 80% held-out pass rate and a larger one with 95% on the same task class. Policy requires 90%. Both are permitted and available, and invented costs are one and five units. These numbers illustrate a choice; they are not benchmarks for named models.
Analogy - Choose an eligible repair bench: A dispatch desk has a simple repair kit, a specialist bench and a waiting tray. The dispatcher first checks which bench is qualified and open for this job, then chooses the less costly eligible option. If neither qualifies, the job waits rather than being sent blindly.
Where the analogy stops: Model capability is not a fixed human qualification. Measured pass rates have uncertainty and vary by task; a passing average cannot guarantee an individual answer.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/model-routing-basics.webp
Picture description: A basic tool roll, a specialist instrument case and a separate waiting tray with a closed divider sit on a repair dispatch table.
Picture caption: Choose among eligible candidates; when none qualifies, waiting is better than a hidden compromise.
What you see: Candidate one. What it represents: The simple kit represents a potentially cheaper model whose task quality still needs evidence.
What you see: Candidate two. What it represents: The specialist case represents another model, not automatically better or permitted.
What you see: No eligible route. What it represents: The waiting tray represents explicit deferral when quality, policy or availability excludes every candidate.
Predict: With a 90% minimum, does the one-unit small model win merely because it is cheaper?
Try: Confirm Route code=2 and Synthetic cost units=5. Raise Small model evaluation pass rate from 80 to 92; the route becomes 1 with cost 1. Restore 80, then set Large model available to 0.
Explain: Initially only the large model clears the quality gate. At 92%, the small model becomes the cheaper eligible choice. With the small score restored and the large model unavailable, route code 0 means abstain and cost zero means no model call, not a free successful answer.
Worked engineering example: Route one repair question without lowering the bar
1. Set a 90% held-out pass-rate floor for the task class. Why: A declared target makes the cost-quality decision auditable.
2. Reject the small model’s 80%; retain the large model’s 95%. Why: Cheapness does not compensate for missing the required evidence threshold.
3. Check both permission and live availability for the remaining candidate. Why: A quality score alone does not make a model usable for the current request.
4. Choose the large model at five synthetic units, or abstain if it is unavailable. Why: The least-cost rule applies only inside the eligible set.
Outcome: The initial request routes to the large model. A measured small-model score of 92% changes the choice; an outage without another eligible model produces no route.
Tradeoff - Route more work to a cheap candidate: Gain: Can reduce average call cost. Cost or risk: Misrouted hard cases and class-specific failures may increase.
Tradeoff - Raise the quality floor: Gain: Restricts routing to stronger measured evidence. Cost or risk: Can increase cost or leave no eligible route.
Tradeoff - Use a learned router: Gain: Can exploit request-specific capability differences. Cost or risk: Adds its own inference cost, drift and evaluation burden.
Before real use: Version evaluation sets and report pass rates by task, policy, language and failure severity.
Before real use: Enforce data residency, permissions and output/tool requirements before cost ranking.
Before real use: Observe availability and explicitly test no-eligible-route behavior during outages.
Before real use: Evaluate the full router-plus-model system against baselines, including uncertainty and misroutes.
Your turn: Set both candidates to 95% with a 90% floor. The small model is available but forbidden; the large is permitted and available. Finish the route and cost.
Hint: Quality is only one gate; filter permission before ranking costs.
Compare your reasoning: Route to the large model, code 2, cost five synthetic units. If the large candidate also becomes unavailable, abstain rather than overriding permission.
Optional math: eligible_j = (q_j >= q_min) AND p_j AND a_j; choose least c_j among eligible models
j: Candidate identity: small or large in this exercise.
q_j: Supplied held-out task-class pass rate for candidate j.
q_min: Minimum evaluated pass rate required by this policy.
p_j: Permission flag for using candidate j with this request.
a_j: Availability flag for candidate j at this decision.
c_j: Invented call cost: one unit for small, five for large.
eligible_j: True only if all conditions pass; AND means all must hold.
An empty eligible set produces abstention, not a forced cheapest fallback. The exercise is a rule-based gate over supplied evidence; it does not learn request difficulty or model capabilities. Real evaluation confidence intervals and per-class results matter near the threshold.
Common misconception: Routing automatically preserves quality while making every request cheaper.
Correction: Savings depend on measured differences and the router’s decisions. Routing errors, evaluation drift, policy restrictions and outages can remove the benefit. A larger model is not inherently better on every task, and an outage does not lower the quality floor.
Check your understanding: The only model meeting the quality floor is unavailable. What does this policy require?
Abstain or defer rather than silently route below the floor. - Correct. Availability is a hard condition. An approved human or deferred path is more honest than calling an ineligible model a successful fallback.
Treat the cheap model’s failing score as passing during outages. - An outage changes availability, not evidence. A degraded mode needs an explicit separately approved policy and user-facing contract.
Ignore permission because the expensive model is probably better. - Quality does not authorize data transfer. Permission and quality are separate required gates, and neither implies availability.
Use it with purpose: A cheaper model passes general chat tests but has never seen your financial extraction cases. Route those cases to it?
Guidance: Obtain held-out, task-specific evidence and check data permissions, output requirements and availability first. Evaluate routing decisions end to end, including misclassified hard cases. Start with an explicit no-route or reviewed path where evidence is missing.
Where to go next:
Model Routing (concept): https://www.openagentschool.org/concepts/model-routing - Extend a transparent eligibility gate into a measured request-routing policy.
Agent Evaluation Methodologies (concept): https://www.openagentschool.org/concepts/agent-evaluation-methodologies - Use held-out task-class evidence and uncertainty rather than assuming model-size superiority.
References:
Ong et al. - RouteLLM: Learning to Route LLMs with Preference Data: https://arxiv.org/abs/2406.18665

LESSON: Rate Limits
URL: https://www.openagentschool.org/ai-ml-foundations/rate-limits
Also called: request quota, token rate limit, throttling
Summary: Plan request and token traffic together, and respect the server’s instructions when throttled.
In everyday words: Rate limits bound usage over time, often with separate request and token constraints. Requests per minute (RPM) and tokens per minute (TPM) describe different resources. A server can also limit short bursts, shared model pools or spending. A Retry-After response header tells a client the minimum delay before a retry, not a guaranteed future slot.
Why it matters: A client can stay under its request count while exhausting tokens with long prompts. Immediate retries can worsen the overload and consume additional quota. Admission control and measured pacing are more useful than assuming every throttled request will succeed on the next attempt.
Scenario - A fictional swimming-pool reception: A synthetic help service has room for ten requests and 1,000 accounted tokens in its current window. Twelve equal-size requests each need 100 tokens. For deferred work, the server supplies a three-second Retry-After. You must explain why not all twelve can enter now and why waiting three seconds is not an admission guarantee.
Analogy - Count both visitors and towel bundles: A pool reception has entry wristbands and a separate towel shelf. A group needs enough of both resources; spare wristbands do not create towels. A temporary closed gate signals waiting, not a promise that every later group will fit.
Where the analogy stops: API tokens are not towels and real limits may refill continuously rather than reset as a fixed window. The toy uses already-remaining budgets and equal-size requests only.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/rate-limits.webp
Picture description: A tray of plain wristbands and a separate towel shelf sit near a closed pool entry gate with a sand timer.
Picture caption: Requests and tokens are separate allowances; a server delay does not reserve future capacity.
What you see: Request allowance. What it represents: Wristbands represent how many requests may be admitted.
What you see: Token allowance. What it represents: The separate towel supply represents another budget that can run out first.
What you see: Wait and recheck. What it represents: The gate and timer represent deferred work subject to a server minimum and later eligibility.
Predict: Ten request slots and 1,000 tokens admit ten 100-token requests. If each request grows to 250 tokens, do spare request slots help?
Try: Confirm Requests admitted=10, Requests deferred=2, Tokens left=0 and server minimum wait=3 seconds. Change Accounted tokens per request to 250. Then raise Requests remaining from 10 to 20.
Explain: At 250 tokens each, only four requests fit and eight are deferred. Raising request room to twenty still admits four because tokens are now binding. The server minimum stays three seconds for deferred work; the exercise does not model refill or promise later success.
Worked engineering example: Admit the pool reception’s pending work
1. Read ten remaining requests and 1,000 remaining tokens. Why: Current remaining capacity matters more than a nominal daily or monthly number.
2. Divide 1,000 by 100 tokens per request: ten whole requests. Why: Both resources currently allow the same number of admissions.
3. Admit ten of twelve and retain two as deferred work. Why: An explicit queue or rejection is safer than pretending every request was sent successfully.
4. Honor the three-second server minimum, then recheck eligibility. Why: The delay is not a reservation and retry safety still applies.
Outcome: Ten admitted, two deferred, no tokens left. At 250 tokens per request the same budgets admit four and defer eight.
Tradeoff - Queue bursts: Gain: Smooths temporary spikes without immediate rejection. Cost or risk: Adds waiting and requires deadlines, cancellation and bounded queue size.
Tradeoff - Reduce tokens per request: Gain: May admit more work under token limits. Cost or risk: Removing necessary evidence or output capacity can lower task quality.
Before real use: Read limits and error details for the specific organization, project, model and current tier.
Before real use: Track request, input/output-token and shared-pool budgets separately where applicable.
Before real use: Parse and honor Retry-After; do not treat billing/quota failures as transient overload.
Before real use: Use bounded queues and retry budgets, and observe burst behavior rather than only minute averages.
Your turn: There are twelve pending requests, twenty request slots, 900 tokens and 250 tokens per request. Fill in admissions, deferred requests and leftover tokens.
Hint: Round the token quotient down before subtracting usage.
Compare your reasoning: Three admitted, nine deferred and 150 tokens left. Those remaining tokens cannot admit a fourth complete request.
Optional math: A = min(P, R, floor(T / k)); deferred = P - A
A: Number admitted by both remaining budgets.
P: Pending equal-size requests.
R: Remaining request allowance in the modeled window.
T: Remaining accounted-token allowance.
k: Positive accounted tokens per request.
floor: Round down to whole requests; min chooses the smallest constraint.
deferred: Pending requests that cannot be admitted now.
This is a planning calculation, not the provider’s live algorithm. Actual services can separate input and output budgets, use token buckets or apply acceleration limits. Cached-token and output-reservation rules vary by model/provider; verify current headers and documentation.
Common misconception: A 429 response always means retry immediately until it succeeds.
Correction: HTTP 429 means a limit was reached, but the cause may require slowing down, waiting or fixing billing. Inspect the error body and Retry-After, bound retries and avoid replaying unsafe operations. A server limit is not a guaranteed minimum capacity.
Check your understanding: With 1,000 tokens remaining and 250 per request, increasing request allowance from ten to twenty does what?
Admits all twelve pending requests because request room doubled. - Twelve requests would need 3,000 tokens. Increasing one allowance does not increase the other resource budget.
Still admits only four because the token allowance binds. - Yes. The token budget permits four whole requests; the extra request slots cannot manufacture additional token capacity.
Guarantees eight retries will succeed three seconds later. - Retry-After supplies a minimum wait, not a reservation. Other traffic and refill rules still determine future capacity.
Use it with purpose: A burst of long document requests triggers throttling while RPM remains low. What changes?
Guidance: Inspect token accounting, remaining budgets and short-interval or shared limits. Pace or queue work, reduce unnecessary input if task-safe, and honor server delays. Distinguish transient throttling from exhausted spend rather than repeatedly retrying everything.
Where to go next:
Agent Deployment (concept): https://www.openagentschool.org/concepts/agent-deployment - Set admission and pacing behavior against the deployed provider’s current limits.
Agent Operations (concept): https://www.openagentschool.org/concepts/agent-ops - Observe throttling causes and manage user-visible delay instead of multiplying requests.
References:
OpenAI - Rate limits, shared pools and Retry-After handling: https://developers.openai.com/api/docs/guides/rate-limits
Anthropic - Rate limits, token buckets and token accounting: https://platform.claude.com/docs/en/api/rate-limits

LESSON: Retries & Backoff
URL: https://www.openagentschool.org/ai-ml-foundations/retries-and-backoff
Also called: exponential backoff, retry jitter, retry budget
Summary: Retry safe transient failures with bounded waiting, not an endless loop that amplifies overload.
In everyday words: A retry repeats an eligible operation after a failure. Exponential backoff increases local waiting between attempts; a cap bounds that local growth. Jitter varies waits across clients so they do not all retry together. Safety, error type, attempt budget, cancellation and a total deadline decide whether another attempt is allowed at all.
Why it matters: Retries can recover from temporary faults, but they also add load and can duplicate side effects. A request timeout does not prove nothing happened. Retry only operations known to be safe under their contract; duplicate-effect prevention belongs to the separate Idempotency lesson.
Scenario - A fictional apartment intercom: A synthetic read-only building-directory lookup suffered a retryable temporary error. You are considering retry number three, with at most three additional attempts, a two-second base, ten-second local cap and five-second server minimum. Twenty seconds remain overall. The illustrative jitter fraction is one half; no real timer or random draw runs.
Analogy - Wait before pressing the bell again: An apartment intercom sits beside an hourglass and a short row of attempt tokens. Waiting longer and varying the pause reduces a crowd’s synchronized ringing. When the tokens or waiting time run out, the caller stops instead of continuing forever.
Where the analogy stops: Ringing an intercom is not an HTTP operation. Repeating a payment or tool action may cause duplicate effects, so a timing rule cannot decide whether replay is safe.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/retries-and-backoff.webp
Picture description: An unmarked apartment intercom button, a sand timer and a finite row of wooden attempt tokens sit in an entry alcove.
Picture caption: Pause, spread attempts and stop at a bound; waiting does not make every action safe to repeat.
What you see: Check replay safety. What it represents: The call button represents an operation whose repeatability must be decided before retrying.
What you see: Wait before retrying. What it represents: The timer represents bounded backoff that must respect any server minimum.
What you see: Finite attempt budget. What it represents: The limited tokens represent a stop condition, not a guarantee of eventual success.
Predict: At retry three, the local exponential ceiling is eight seconds. With the server’s five-second minimum and half of the four-second jitter span, is the candidate delay seven seconds?
Try: Confirm Local backoff ceiling=8, Candidate delay=7, Schedule retry=1 and Time left=13 seconds. Set Remaining total deadline to 7, then restore 20 and set Safe operation and retryable error to 0. Finally restore safety and set Server minimum delay to 30.
Explain: A delay equal to the deadline leaves no time for an attempt, so it stops. An unsafe or non-retryable operation also stops. A thirty-second server minimum gives a 32-second candidate delay and deferral, not an early retry at the ten-second local cap.
Worked engineering example: Admit one safe directory retry
1. Confirm a safe read-only operation and a transient error. Why: Timing is irrelevant if the operation or error is not eligible for retry.
2. Compute min(10, 2 * 2^(3 - 1)) = eight seconds. Why: The third additional attempt uses a capped exponential envelope.
3. Compute max(5, 4) + 0.5 * 4 = seven seconds. Why: Respect the server minimum while illustrating a spread in client retry times.
4. Check retry three is within budget three and seven is below twenty. Why: Thirteen seconds remain for the attempt, which must still honor cancellation and the total deadline.
Outcome: One candidate retry is eligible after seven seconds. No real request is retried, and success is neither simulated nor promised.
Tradeoff - Allow additional retries: Gain: Can recover more transient failures. Cost or risk: Adds load, cost and user waiting; can amplify an outage.
Tradeoff - Increase delay and jitter: Gain: Spreads traffic and gives a service recovery time. Cost or risk: May consume the user’s remaining deadline before useful work can start.
Tradeoff - Stop on ambiguous side effects: Gain: Avoids blind duplicate actions. Cost or risk: Requires reconciliation or an explicit safe-replay mechanism instead of instant recovery.
Before real use: Classify error type and operation safety; do not automatically replay partial streams or side-effecting tools.
Before real use: Honor valid Retry-After values and defer rather than shorten unsupported long delays.
Before real use: Bound additional attempts and total elapsed time; include SDK retries in the same accounting.
Before real use: Use independent jitter in real clients, propagate cancellation and record explicit exhausted/deferred outcomes.
Your turn: Use the initial third retry, but reduce Maximum additional attempts to two. The seven-second delay still fits. Should it run?
Hint: Time and attempt count are separate gates; passing one cannot override the other.
Compare your reasoning: No. Retry number three exceeds the two-additional-attempt budget. Report exhaustion; do not reset the counter or start a second retry loop.
Optional math: e = min(c, b * 2^(n - 1)); w = max(s, e / 2) + j * e / 2
e: Capped local exponential delay envelope in seconds.
c: Maximum local backoff envelope, not a cap on server instructions.
b: Base delay in seconds.
n: Retry number: one is the first additional attempt after the original.
s: Parsed server minimum delay in seconds.
j: Illustrative jitter fraction between zero and one.
w: Candidate wait; min/max select smaller/larger values and ^ denotes exponentiation.
This example chooses a lower bound of the larger of the server delay and half the local envelope, then adds jitter across half the envelope. Real clients draw jitter independently; the slider is deterministic. Schedule only when safety passes, n is within the attempt budget and w is strictly below the remaining deadline.
Common misconception: A backoff cap permits retrying earlier than a longer Retry-After, and retries make any action safe.
Correction: Never shorten a valid server minimum to fit a local cap. Defer if it exceeds the allowed time. Backoff only schedules attempts; it cannot prevent duplicate side effects or turn authentication, validation or billing errors into transient faults.
Check your understanding: The server asks for thirty seconds, but only twenty remain. What should this client do?
Retry at ten seconds because that is the local cap. - The cap bounds locally chosen exponential growth, not the server’s minimum. Retrying at ten seconds would ignore the thirty-second instruction.
Start another retry loop inside the existing loop. - Nested loops multiply attempts and undermine both delay and deadline limits. SDK and application retries must share one budget.
Stop or defer rather than scheduling an attempt before the server allows it. - Correct. The request cannot fit this deadline while honoring the server. Surface the failure or approved deferred path instead of retrying early.
Use it with purpose: A tool call times out after it may have created an order. Should the generic backoff helper replay it?
Guidance: Not without an explicit safe-replay contract. Determine whether the effect occurred and use the system’s duplicate-prevention mechanism where supported. For known safe transient failures, bound attempts and total time, honor Retry-After and avoid nested retries.
Where to go next:
Agent Operations (concept): https://www.openagentschool.org/concepts/agent-ops - Manage retry load, cancellation and failure visibility under operational limits.
Transactional Agent Workflows (concept): https://www.openagentschool.org/concepts/transactional-agent-workflows - Decide replay safety and duplicate-effect handling separately from retry timing.
References:
OpenAI - Rate limits, bounded backoff, jitter and server delays: https://developers.openai.com/api/docs/guides/rate-limits

LESSON: Guardrails
URL: https://www.openagentschool.org/ai-ml-foundations/guardrails
Also called: input and output filters, safety checks, content moderation layer, policy guards
Summary: Use guards to catch risky content, and keep permission decisions in deterministic server checks.
In everyday words: A guardrail is a check placed before or after a model step that detects unwanted input or output and can block, rewrite, or escalate it. Most guards are classifiers or rules that estimate risk, so they make mistakes in both directions. A guard is not authorization (deciding whether this user may perform this action) and not sandboxing (limiting what an approved action can touch).
Why it matters: Teams often add a moderation step and assume the system is now safe. A guard tuned to block more will also block more harmless requests, and a guard tuned to be friendly will let some risky requests through. Permission must still be enforced by code that knows the signed-in user, because a request that slips past the guard should not gain new powers.
Scenario - A fictional tool-library kiosk: An invented neighborhood tool library runs a chat kiosk that answers questions and can unlock lending lockers. Ten synthetic requests arrive: five harmless questions, two unsafe-advice requests, and three polite requests to unlock a locker that belongs to another member. Decide which layer should stop each kind of problem.
Analogy - A lobby screener and a key-card door: The lobby screener glances at every bag and stops anything that looks dangerous. Separately, each room has a key-card lock that opens only for people on that room's list. A friendly visitor with a clean bag still cannot enter a room they are not allowed in.
Where the analogy stops: A human screener can ask follow-up questions and remember faces. A model guard scores text it sees right now, can be fooled by rephrasing, and has no idea who owns which locker unless your system tells it.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/guardrails.webp
Picture description: A lobby counter with a bag-check tray, a closed interior door with a key-card reader, and a blank access card on a side table between them.
Picture caption: Screening what comes in is a different job from deciding who may open which door.
What you see: Content guard. What it represents: The bag-check tray stands for a guard that estimates risk from what a request contains.
What you see: Server authorization. What it represents: The key-card door represents a permission check tied to who the user is and what they own.
What you see: Identity. What it represents: The blank access card stands for the session identity the server checks, not text the model writes.
Predict: At a guard threshold of 0.5 with server authorization off, how many unauthorized locker unlocks do you expect to execute?
Try: Start at threshold 0.5 and authorization 0. You should see 4 risky requests stopped, 1 benign request wrongly blocked, and 1 unauthorized unlock executed. Turn authorization on, then slide the threshold down to 0.3 and up to 0.9.
Explain: The politely worded unlock request scores only 0.35, so the guard lets it through and, without a server check, it executes. Turning authorization on denies it no matter the threshold. Lowering the threshold to 0.3 stops all five risky requests but wrongly blocks two harmless ones: guards trade misses against false blocks, while permission checks do not.
Worked engineering example: Layering checks for the kiosk unlock tool
1. Run an input guard on each message and log its score and decision. Why: Catches clearly unsafe requests early and gives you data to tune false blocks.
2. Expose the unlock tool only with a locker ID argument, never a member ID. Why: The server derives the member from the session, so the model cannot claim to be someone else.
3. In the unlock handler, check that the session member owns the locker. Why: Deterministic authorization stops requests that sound harmless but are not permitted.
4. Run an output guard on the reply before showing it. Why: Catches unsafe advice the model produced even from a harmless-looking question.
Outcome: A polite request for someone else's locker passes the guard but is refused by the ownership check, and both decisions appear in the logs.
Tradeoff - Lower guard threshold: Gain: Catches more risky requests. Cost or risk: Blocks more harmless requests and frustrates members.
Tradeoff - Blocking guard before the model runs: Gain: Nothing unsafe reaches the model or tools. Cost or risk: Adds latency to every request.
Tradeoff - Parallel guard alongside the model: Gain: Lower latency. Cost or risk: The model may already have spent tokens or called tools before the guard trips.
Before real use: Every state-changing tool enforces authorization on the server using the session identity.
Before real use: Guard false-block and miss rates are reviewed on a labeled sample at least each release.
Before real use: Tools run with least privilege so an approved call cannot touch other members' data.
Before real use: Guard decisions are logged with redacted content for later audit.
Your turn: The kiosk adds a "transfer my borrowed drill to a friend" tool. The guard is in place. Name the check that must run inside the transfer handler.
Hint: Ask which fact the guard cannot know from the text alone.
Compare your reasoning: Verify on the server that the signed-in member currently holds that drill, and that the friend is an active member, before changing the loan record.
Optional math: block when s ≥ τ;  unauthorized executed = count(s < τ and u) × (1 − A)
s: Illustrative risk score the guard gives one request, from 0 to 1.
τ: Blocking threshold you choose; requests at or above it are blocked.
u: True when the request asks for an action the member is not permitted to do.
A: Server authorization switch: 1 when ownership is checked, 0 when it is not.
The guard term depends on a score that can be wrong. The authorization term is a yes-or-no rule against real account data, so turning A on drives unauthorized executions to zero regardless of τ. This is an illustrative accounting model with fixed scores, not a real classifier.
Common misconception: A strong enough moderation guard makes server permission checks unnecessary.
Correction: Guards estimate risk from text and will miss some cases. Permission is a fact about the user and the resource, so check it deterministically on the server, and sandbox tools so even approved actions have limited reach.
Check your understanding: A request to unlock someone else's locker passes the guard because it sounds polite. What should stop it?
Lower the guard threshold until the polite request is blocked. - That may catch this one, but it also blocks more harmless requests, and the next rewording may still pass. Permission is not a text-risk question.
A server-side ownership check before the unlock runs. - Right. The server knows who is signed in and who owns the locker, so it can refuse the action regardless of how the request was phrased.
A stronger system prompt telling the model to refuse unlocks. - Prompts influence behavior but are not enforcement. A rephrased or injected request can still steer the model, so the unlock needs a check outside the model.
Use it with purpose: Your team proposes one powerful guard model in front of an agent that can issue refunds. Is that enough?
Guidance: Keep the guard for unsafe content, but put refund limits and ownership checks in server code, sandbox the refund tool to the caller's own orders, and log guard decisions so you can tune false blocks and misses with real data.
Where to go next:
Agent Security (concept): https://www.openagentschool.org/concepts/agent-security - See how guards fit into a layered defense with identity, authorization, and monitoring.
Agent Execution Sandboxing (concept): https://www.openagentschool.org/concepts/agent-execution-sandboxing - Limit what an approved action can reach so a missed guard decision has a small blast radius.
Guardrails Layer (pattern): https://www.openagentschool.org/patterns/guardrails-layer - Study a concrete pre- and post-model guard pipeline and where its decisions are logged.
Guardrails & Structured Outputs (course): https://www.openagentschool.org/courses/guardrails-structured-outputs - Practice wiring guards, schemas, and server checks together in a guided build.
References:
OpenAI Agents SDK - Guardrails (input, output, and tool guardrails): https://openai.github.io/openai-agents-python/guardrails/
OWASP Gen AI Security Project - LLM06:2025 Excessive Agency: https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

LESSON: LLM-as-a-Judge
URL: https://www.openagentschool.org/ai-ml-foundations/llm-as-judge
Also called: model-graded evaluation, AI grader, automated rater, pairwise judge
Summary: Let a model grade outputs only after a clear rubric, human calibration, order swaps, and honest error bars.
In everyday words: LLM-as-a-judge means using a language model to score or compare outputs, usually against a written rubric. It is a measuring instrument, so it needs the same care as any instrument: a definition of what is being measured, a check against trusted human labels (calibration), known biases such as preferring whichever answer is shown first (position bias), and an estimate of uncertainty.
Why it matters: Judges make it cheap to score thousands of outputs, which is exactly why their errors spread quietly. A judge that agrees with people only 75% of the time, measured on 100 pairs, can easily hide a real regression. Teams that skip calibration end up optimizing for the judge's taste instead of their users' needs.
Scenario - A fictional recipe-reply contest: An invented cooking app compares two draft replies to each synthetic home-cook question and wants a model to pick the better one. A food editor has already labeled 100 pairs by hand. You notice the judge seems to favor whichever reply appears first. Decide how to measure and report the judge's reliability.
Analogy - A baking-contest judge with a scorecard: A good baking judge scores against a published scorecard, tastes some pies alongside a panel of experienced bakers to confirm they agree, and swaps the plate order to avoid always favoring the first slice. Reported results include how sure the panel is.
Where the analogy stops: A human judge can explain a sensation and be questioned. A model judge can sound confident while being inconsistent, and its biases come from training data rather than taste buds, so they must be measured rather than assumed.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/llm-as-judge.webp
Picture description: Two similar pies on a lazy Susan turntable, a clipboard with a blank scorecard, and a glass holding several tasting spoons on a hall table.
Picture caption: A fair judge uses a shared scorecard, swaps the order, and checks against a human panel.
What you see: Swap the order. What it represents: The turntable represents judging both answer orders to expose position bias.
What you see: Rubric. What it represents: The blank scorecard stands for written criteria the judge and humans both apply.
What you see: Human calibration. What it represents: The tasting spoons represent human labels used to measure how often the judge agrees.
Predict: With 80% base agreement and a 10% order-flip rate, will a single-order judge show 80% agreement with the human labels?
Try: Start with agreement 80, flip rate 10, swap off, and 100 calibration pairs. You should see 75.0% agreement and a margin of about ±8.49 points. Turn swap on, then raise the pairs to 400.
Explain: Half of the order-flipped verdicts land on the wrong side, so single-order agreement drops to 75%. Running both orders sends 10 flipped pairs to people and restores 80% on the 90 pairs the judge decides. More human-labeled pairs shrink the margin; changing the rubric would require recalibrating.
Worked engineering example: Calibrating a pairwise reply judge
1. Write a rubric with three concrete criteria and an example of a pass and a fail for each. Why: The judge and the humans need the same definition of better.
2. Have two people label 100 pairs and resolve their disagreements first. Why: Human labels are the reference, so their own consistency must be known.
3. Run the judge on each pair in both orders. Why: Order-dependent verdicts reveal position bias directly.
4. Report agreement, the flip rate, and a 95% interval on the decided pairs. Why: Decision makers need to know how far the score can be trusted.
Outcome: The team learns the judge agrees with people on about 80% of decided pairs, routes order-flipped pairs to humans, and only trusts score differences larger than the margin.
Tradeoff - Judge both orders: Gain: Exposes and removes position-biased verdicts. Cost or risk: Doubles judge calls and sends some pairs to people.
Tradeoff - Larger calibration set: Gain: Tighter uncertainty on agreement. Cost or risk: More human labeling time and cost.
Tradeoff - Pass/fail instead of 1-10 scores: Gain: Usually more consistent judgments. Cost or risk: Less detail about how close an answer was.
Before real use: Rubric, judge model, and prompt are versioned together, and any change triggers recalibration.
Before real use: A frozen human-labeled calibration set is rescored on every judge change.
Before real use: Order-flip and length-preference rates are tracked over time.
Before real use: Dashboards show the margin of error next to every judge-based score.
Your turn: Your judge agrees with people 82% on 50 pairs, and a new prompt raises the judge score by 2 points. Should you ship on that evidence?
Hint: Compute roughly 1.96 × √(0.82 × 0.18 / 50) and compare it with 2 points.
Compare your reasoning: No. The margin is about ±10.6 points, far larger than the 2-point change. Label more pairs or run a paired comparison with humans before deciding.
Optional math: single order: p = a − f/2;  margin ≈ 1.96 × √(p(1 − p) / n_decided)
a: Agreement rate with human labels when order effects are removed.
f: Share of pairs whose verdict flips when the two answers swap position.
p: Measured agreement used for the estimate, as a fraction.
n_decided: Number of human-labeled pairs the judge actually decided.
The 1.96 multiplier gives an approximate 95% interval using a normal approximation, which is rough for small samples. This is an illustrative accounting model of a judge; it does not call any model, and the base agreement is an input you would measure on your own labeled pairs.
Common misconception: A powerful model judge is objective, so its scores need no human checking.
Correction: Judges show position, length, and self-preference biases and can misread a rubric. Calibrate against human labels, swap orders, report uncertainty, and recheck whenever the rubric, judge model, or task changes.
Check your understanding: Your judge prefers reply A when shown first and reply B when shown first. What is the most useful next step?
Average its scores and report the result without comment. - Averaging hides the inconsistency. A verdict that depends on order is not evidence about quality and should be flagged, not blended away.
Switch to a larger judge model and assume the bias is gone. - Larger judges can still show position bias. Any new judge needs the same order-swap test and human calibration before you trust it.
Judge both orders and send disagreements to human review. - Yes. Requiring agreement across orders removes position-dependent verdicts from the automatic path and keeps people in the loop for the uncertain cases.
Use it with purpose: Leadership wants the judge score on a dashboard as the release gate. What must accompany it?
Guidance: Show the rubric version, the judge-human agreement on a frozen calibration set, the margin of error, and the share of cases sent to people. Gate releases only on differences larger than that uncertainty.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Place model-graded scoring among human review, code checks, and task-level metrics.
Agent Evaluation Methodologies (concept): https://www.openagentschool.org/concepts/agent-evaluation-methodologies - Compare pairwise, rubric, and reference-based grading methods and when each fits.
Rubric Rater (pattern): https://www.openagentschool.org/patterns/rubric-rater - See a rubric-driven scoring pattern that makes criteria explicit before grading.
Agent Evals & Benchmarking Lab (course): https://www.openagentschool.org/courses/agent-evals-benchmarking-lab - Practice calibrating a judge against human labels in a hands-on evaluation lab.
References:
Zheng et al. - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685
Wang et al. - Large Language Models are not Fair Evaluators: https://arxiv.org/abs/2305.17926
OpenAI API docs - Evaluation best practices (judge biases and recommendations): https://developers.openai.com/api/docs/guides/evaluation-best-practices

LESSON: Structured Outputs
URL: https://www.openagentschool.org/ai-ml-foundations/structured-outputs
Also called: JSON schema output, constrained decoding, typed responses, JSON mode
Summary: A schema guarantees the shape of a reply, not its truth; still handle refusals, truncation, and business rules.
In everyday words: Structured outputs ask a model to reply in a fixed format, usually described by a JSON Schema (a standard way to state required fields, types, and allowed values). With strict enforcement the reply matches the schema when it completes normally. It can still refuse, stop early when it hits a token limit, or fill a perfectly shaped field with a wrong value.
Why it matters: Parsing free text is fragile, so schemas remove a whole class of bugs. The danger is believing a valid object is a correct one. A receipt total can be well formed and wrong, a refusal is a different kind of response, and a reply cut off by a token limit is incomplete and must not be parsed as if it were finished.
Scenario - A fictional expense-receipt reader: An invented small office scans ten synthetic receipts and asks a model to return merchant, line items, and total as JSON. Two receipts have smudged totals, one request is refused, and a few long receipts need more output tokens than allowed. Decide what your code must check before saving a record.
Analogy - An egg carton: An egg carton guarantees each cup holds one egg-shaped thing and that the lid closes. It cannot tell you whether an egg is fresh. A half-filled carton with the lid propped open is not ready to ship, and an empty carton returned on purpose is a different message altogether.
Where the analogy stops: A carton is a physical container that never changes its contents. A model chooses tokens to fit the schema, so constraining the shape can sometimes nudge the content too, and it cannot add facts the input never contained.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/structured-outputs.webp
Picture description: Three egg cartons on a counter: one closed, one half-filled with its lid propped open, and one full carton containing a single wooden egg.
Picture caption: The carton fixes the shape, not the freshness; a half-filled carton is not ready to use.
What you see: Complete shape. What it represents: The closed carton represents a reply that finished and matches the schema.
What you see: Truncated. What it represents: The half-filled open carton stands for an incomplete reply that must not be parsed as finished.
What you see: Valid but wrong. What it represents: The wooden egg represents a schema-valid field whose value is not true.
Predict: With a 60-token output budget, one refusal, and schema enforcement on, how many of the ten receipts produce schema-valid records, and is every valid record correct?
Try: Start at 60 tokens, 1 refusal, schema on, and business rule off. You should see 4 schema-valid records, 5 truncated responses, 1 refusal, and 1 valid record with a wrong total accepted. Turn on the business rule, then raise the budget to 200.
Explain: Receipts needing more than 60 tokens stop early and must be retried, not parsed. One of the four valid records has a misread total that passes the schema. Only the line-item sum rule catches it. Turning schema enforcement off adds a schema-invalid response instead.
Worked engineering example: Saving receipts safely
1. Check the response status for refusal or incomplete output before parsing. Why: A refusal or a truncated object is not data and must take a different path.
2. Parse into the typed object defined by the schema. Why: Shape is now guaranteed, so code can rely on required fields.
3. Compare the total with the sum of line items and check the date is not in the future. Why: Cross-field rules catch values that are well formed but wrong.
4. Send flagged receipts to a review queue with the original image. Why: A person can resolve smudges the model misread.
Outcome: Only complete, schema-valid, rule-passing receipts are saved automatically; refusals, truncations, and suspicious totals are visible instead of silently stored.
Tradeoff - Higher output token limit: Gain: Fewer truncated long receipts. Cost or risk: Higher worst-case cost and latency.
Tradeoff - Strict schema enforcement: Gain: No missing keys or invalid enum values. Cost or risk: Supported schema features vary by provider; business-rule checks are still your responsibility.
Before real use: Refusal and incomplete statuses are handled before any parsing.
Before real use: Business rules validate cross-field meaning such as sums and date ranges.
Before real use: Schema versions are stored with each saved record.
Before real use: A sample of saved records is compared against source documents each week.
Your turn: A long receipt keeps returning an incomplete response at 60 tokens. List two reasonable fixes.
Hint: One changes the budget; the other changes how much one request must produce.
Compare your reasoning: Raise the output token limit for that request, or split long receipts into pages and merge the line items afterward, then re-run the sum check.
Optional math: complete when t_i ≤ T;  wrong accepted = count(valid and wrong) × (1 − B)
t_i: Output tokens receipt i needs to finish its JSON object.
T: Maximum output tokens allowed for the response.
valid and wrong: Records that match the schema but contain a misread total.
B: Business-rule switch: 1 when line items are summed and compared with the total.
The first rule decides whether a response finished. The second shows that shape checks and truth checks are separate: only B, a rule about meaning, reduces wrong records. This is an illustrative accounting model with fixed receipt lengths, not a provider call.
Common misconception: If the response matches the schema, the extracted data is correct.
Correction: Valid schema is not factual truth. A schema guarantees fields and types; correctness needs source checks, business rules, or human review. Also check the response status for refusals and incomplete output before parsing.
Check your understanding: A receipt reply fits the schema, but its total does not equal the sum of its line items. What should happen?
Flag the record with a business-rule check before saving it. - Correct. The schema confirmed the shape; the sum check tests meaning. Flagged records can be retried or sent to a person.
Save it, because strict schema mode guarantees correctness. - Strict mode guarantees that fields and types match. It cannot know that the smudged total was misread, so this would save wrong data.
Tighten the schema so totals must be positive numbers. - Useful as a basic constraint, but a wrong positive number still passes. Cross-field rules about meaning need code outside the schema.
Use it with purpose: A teammate removes all post-processing because the provider now enforces the schema. Agree?
Guidance: Keep checks for refusal and incomplete status, cross-field business rules, and source verification for high-stakes fields. Remove only the brittle text-parsing code that the schema truly replaced.
Where to go next:
Tool Use & Function Calling (concept): https://www.openagentschool.org/concepts/tool-use-function-calling - See how the same schemas describe tool arguments that your code must still validate.
Hallucination & Grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Separate well-formed output from content that is actually supported by the input.
Guardrails & Structured Outputs (course): https://www.openagentschool.org/courses/guardrails-structured-outputs - Build a schema-first extractor with refusal handling and business-rule checks.
PromptOps & Tooling (skill): https://www.openagentschool.org/ai-skills#module/promptops - Version schemas and prompts together so changes are tested before release.
References:
OpenAI API docs - Structured model outputs (refusals and incomplete responses): https://developers.openai.com/api/docs/guides/structured-outputs
JSON Schema - What is JSON Schema? Official overview: https://json-schema.org/overview/what-is-jsonschema

LESSON: Hybrid Search
URL: https://www.openagentschool.org/ai-ml-foundations/hybrid-search
Also called: BM25 plus vectors, lexical and semantic retrieval, reciprocal rank fusion, RRF
Summary: Combine exact-word search with meaning-based vectors, and fuse their rankings by position, not raw scores.
In everyday words: Hybrid search runs two kinds of retrieval and merges the results. Lexical search such as BM25 (a classic formula that rewards matching words, especially rare ones) is excellent for part codes and names. Dense vector search compares embeddings (lists of numbers that place similar meanings near each other) and finds paraphrases. Reciprocal rank fusion (RRF) merges the two lists using rank positions.
Why it matters: Each retriever fails differently. Vectors can miss an exact product code; lexical search misses "stopping feels weak" when the manual says "reduced braking". Combining them helps, but BM25 scores and cosine similarities live on different scales, so simply adding them lets one retriever drown out the other without anyone noticing.
Scenario - A fictional bike-shop manual: An invented bike shop indexes six synthetic manual passages. A rider asks about brake pads with part code 7Q that squeal and feel weak in wet weather. One passage contains the exact code; another describes weak stopping after rain without sharing the rider's words. Plan how to rank both near the top.
Analogy - Two library helpers and a merged shortlist: One library helper searches the card catalog for your exact words. Another knows the shelves by topic and pulls books that mean the same thing. To merge their picks fairly, you look at where each helper ranked a book, not at how loudly each helper praised it.
Where the analogy stops: Real helpers can talk and agree on a combined view. Retrievers never consult each other; fusion is a fixed formula, and it cannot rescue a document that neither list included.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/hybrid-search.webp
Picture description: An open card-catalog drawer on the left, a shelf of similar books on the right, and a small merged stack of books on a reading table between them.
Picture caption: Exact-word lookup and meaning-based picks merge by position into one shortlist.
What you see: Lexical (BM25). What it represents: The card-catalog drawer represents exact-word search that excels at codes and names.
What you see: Dense vectors. What it represents: The topic-grouped shelf stands for embedding search that finds paraphrases.
What you see: Rank fusion. What it represents: The merged stack represents combining both lists by rank rather than by raw scores.
Predict: If you add raw BM25 scores to cosine similarities, will the paraphrase-only passage D2 rise or stay buried?
Try: Start with k 60, 4 results kept from each retriever, and naive sum off. You should see D1 at fused rank 2, D2 at rank 4, and 5 candidates in the pool. Switch the naive sum on, then set depth to 6 with the naive sum off again.
Explain: BM25 scores here reach 18 while cosine stays under 1, so the raw sum simply repeats the BM25 order and pushes D2 to rank 5. RRF adds 1/(k + rank) from each list, so both retrievers count equally. Keeping 6 results from each list lifts D2 to rank 3 because it now earns lexical credit too.
Worked engineering example: Adding hybrid retrieval to the bike manual
1. Index each passage for BM25 and store its embedding in a vector index. Why: Both retrievers need the same passages and IDs to be fused.
2. Retrieve the top results from each retriever for the rider's question. Why: Each list captures a different kind of match.
3. Fuse the lists with RRF using k = 60 as a starting point. Why: Rank-based fusion avoids comparing BM25 and cosine values directly.
4. Check recall separately on code queries and paraphrase queries. Why: An average can hide that one query type got worse.
Outcome: The exact part-code passage and the paraphrase passage both reach the fused shortlist, and the team can see which query type each change helps.
Tradeoff - Keep more results per retriever: Gain: More documents earn credit from both lists. Cost or risk: More candidates to fuse and possibly rerank, adding latency.
Tradeoff - Smaller k: Gain: Top positions dominate, rewarding strong single-list matches. Cost or risk: Documents found modestly by both lists can be overtaken.
Tradeoff - Weighted score blending: Gain: Can outperform plain RRF when well tuned. Cost or risk: Requires normalization and labeled queries, and drifts when the corpus changes.
Before real use: Recall is reported per query type, including exact-identifier queries.
Before real use: Both retrievers index the same document versions with shared IDs.
Before real use: Fusion constants and kept-list sizes are versioned and tested on a labeled set.
Your turn: With k = 60, document X is rank 1 in BM25 and absent from the vector list; document Y is rank 3 in both. Which ranks higher?
Hint: Compute 1/61 for X and 1/63 + 1/63 for Y.
Compare your reasoning: Y ranks higher: about 0.0317 versus about 0.0164 for X, because appearing in both lists earns two contributions.
Optional math: RRF(d) = Σ_r 1 / (k + rank_r(d))
d: One document or passage being scored.
r: One retriever, such as BM25 or the vector index.
rank_r(d): Position of d in retriever r's kept list, starting at 1.
k: Ranking constant; larger values flatten the advantage of top positions.
A document missing from a retriever's kept list contributes zero from that retriever. Because only positions are used, BM25 values near 18 and cosine values near 0.8 never get added together. Scores in this lesson are fixed synthetic numbers rather than output from a real search engine.
Common misconception: You can blend lexical and vector results by adding their scores together.
Correction: The scores measure different things on different scales. Use rank-based fusion such as RRF, or normalize scores carefully and validate the weights on labeled queries.
Check your understanding: Your vector search keeps missing passages that contain an exact error code. What is the best first change?
Replace the vector index with a larger embedding model. - A bigger embedding model may still treat rare codes poorly. Exact tokens are a strength of lexical search, so add it rather than only swapping models.
Add BM25 retrieval and fuse both lists with RRF. - Yes. Lexical search finds the exact code, vectors find paraphrases, and rank fusion lets both contribute without one score scale dominating.
Multiply cosine scores by 20 so they match BM25 values. - A fixed multiplier is a guess that changes with every query and corpus. Rank fusion or validated normalization is more robust.
Use it with purpose: Your support search mostly gets natural-language questions, but 15% include product codes. Hybrid or vectors only?
Guidance: Use hybrid retrieval with rank fusion, then measure recall separately for code queries and paraphrase queries on a labeled set so neither group is hidden in an average.
Where to go next:
Retrieval Engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Tune retrievers, fusion, and evaluation sets as one measured system.
RAG Basics (concept): https://www.openagentschool.org/concepts/rag-basics - See where retrieval quality sets the ceiling for generated answers.
Agentic RAG (pattern): https://www.openagentschool.org/patterns/agentic-rag - Watch an agent choose and combine retrieval steps for a multi-part question.
Graph RAG & Knowledge Ops (course): https://www.openagentschool.org/courses/graph-rag-knowledge-ops - Extend lexical and vector retrieval with structured knowledge for harder questions.
References:
Elastic docs - Reciprocal rank fusion (RRF) formula and retriever: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion
Elastic docs - Similarity settings including the default BM25 similarity: https://www.elastic.co/docs/reference/elasticsearch/index-settings/similarity

LESSON: Prompt Injection
URL: https://www.openagentschool.org/ai-ml-foundations/prompt-injection
Also called: indirect prompt injection, instruction hijacking, untrusted content steering, LLM01
Summary: Untrusted text can steer a model; limit what steered output can do instead of trusting one system prompt.
In everyday words: Prompt injection happens when text the model reads changes its behavior in ways the developer did not intend. Direct injection comes from the user's own message. Indirect injection hides in content the model processes, such as a web page, email, or file. Models read instructions and data in the same stream of tokens, so data can act like instructions.
Why it matters: Assistants that read outside content and also hold tools turn a wording problem into a security problem. OWASP lists prompt injection first among risks for language-model applications and notes it is unclear whether fool-proof prevention exists. Strong system prompts help, but the dependable controls sit outside the model: least privilege, server-side policy, and human approval.
Scenario - A fictional garden-club inbox assistant: An invented garden club uses an assistant to summarize ten shared documents each day and, if asked, send replies. Some synthetic documents contain a harmless-looking line addressed to "the assistant" asking it to email the member list elsewhere. Decide which defenses limit the damage when the assistant follows such a line.
Analogy - A mail clerk and a note inside a parcel: A mail clerk takes instructions from the manager. One day a parcel contains a note saying "clerk, send the customer list to this address". A careful clerk treats the note as part of the parcel's contents, and the mailroom only lets outgoing post go to approved addresses anyway.
Where the analogy stops: A human clerk easily knows who the manager is. A model has no built-in way to tell instructions from data in the same text stream, which is why external controls matter more than the clerk's good judgment.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/prompt-injection.webp
Picture description: A clipboard with a sealed envelope on the wall, an opened parcel holding a folded blank slip, and an outgoing-mail tray beside a closed gate latch.
Picture caption: A note inside the parcel is content to handle, not an order from the manager.
What you see: Trusted instructions. What it represents: The clipped envelope represents developer instructions the system is meant to follow.
What you see: Untrusted content. What it represents: The folded slip stands for text inside data that may try to steer the model.
What you see: Privileged action. What it represents: The latched outgoing tray represents tools guarded by server policy outside the model.
Predict: With three injected documents per batch and an assumed 20% chance each is followed, roughly how many of 100 batches would produce a steered response?
Try: Start with 3 injected documents, a 20% assumed follow rate, the send tool on, and the allowlist off. You should see 48.8 steered batches and 48.8 unintended sends per 100. Turn on the server allowlist, then turn off the send tool.
Explain: Steering grows quickly with more untrusted documents: 1 − 0.8³ = 0.488. The allowlist blocks all 48.8 sends without changing the model, and removing the send tool removes the outbound path. Neither stops the summaries from being steered, so outputs still need checks. The 20% rate is an assumption for exploration, not a measured property.
Worked engineering example: Hardening the inbox assistant
1. Mark retrieved documents as untrusted data and keep them separate from developer instructions in the prompt. Why: Clear separation lowers, but does not remove, the chance of steering.
2. Give the send tool credentials only for the club mailbox and enforce a recipient allowlist on the server. Why: Least privilege and server policy cap what steered output can do.
3. Require a member to approve each outbound reply. Why: A person sees the draft and recipient before anything leaves.
4. Log the source documents behind every proposed send. Why: Investigators can trace a strange request back to the content that caused it.
Outcome: A steered summary can still occur, but it cannot email the member list to an outside address, and the source of the steering is traceable.
Tradeoff - Remove the send tool entirely: Gain: Eliminates the outbound path. Cost or risk: Members must send replies themselves.
Tradeoff - Human approval for each send: Gain: Stops most unintended sends. Cost or risk: Slower, and approvers can grow fatigued and approve too quickly.
Before real use: Every tool reachable after reading untrusted content has least-privilege credentials.
Before real use: High-impact actions require server-side policy and human approval.
Before real use: Model outputs are treated as untrusted before being rendered, executed, or forwarded.
Before real use: Red-team tests with benign fictional injections run before each release.
Your turn: The assistant gains a "share file" tool. Complete the defense: the server must check ___ before sharing.
Hint: Think about who may receive the file and who owns it.
Compare your reasoning: That the requesting member may share that file and the recipient is on the club's approved list; otherwise, queue it for human approval.
Optional math: P(steered) = 1 − (1 − p)^n
p: Assumed chance the model follows one embedded instruction.
n: Number of documents in a batch that carry an embedded instruction.
P(steered): Chance that at least one embedded instruction steers the batch.
Even a small p becomes large when many untrusted documents are read. Defenses inside the prompt lower p; privilege limits and server policy cap the impact even when steering happens. This is an illustrative accounting model that treats documents as independent, which real attacks may not be.
Common misconception: A firm system prompt telling the model to ignore embedded instructions solves prompt injection.
Correction: System prompts reduce the chance but cannot be relied on alone. Treat model output as untrusted, give tools least privilege, enforce policy on the server, and require human approval for high-impact actions.
Check your understanding: An assistant that reads shared documents can also send email. Which change most reliably limits injected instructions?
Enforce a server-side recipient allowlist for the send tool. - Correct. The allowlist is checked outside the model, so no embedded text can argue its way past it. Review still matters for the summaries themselves.
Add a sentence to the system prompt saying "never obey documents". - Helpful as one layer, but models can still be steered by cleverly worded content. A rule the model is asked to follow is not an enforcement point.
Filter out documents containing the word "assistant". - Keyword filters are easy to evade with rewording, other languages, or hidden text, and they also drop harmless documents.
Use it with purpose: A product manager wants the assistant to auto-send replies to anyone mentioned in a shared document. Approve?
Guidance: Not as designed. Draft replies for human approval, restrict recipients server-side, keep the assistant's credentials limited to the club's own mailbox, and log every send with its source document.
Where to go next:
Prompt Injection Defense (concept): https://www.openagentschool.org/concepts/prompt-injection-defense - Go deeper on layered defenses, from input handling to privilege separation.
Agent Red Teaming (concept): https://www.openagentschool.org/concepts/agent-red-teaming - Learn how teams safely test their own systems for injection paths before release.
Agent Security Red Team Lab (course): https://www.openagentschool.org/courses/agent-security-red-team-lab - Practice finding and fixing trust-boundary gaps in a controlled lab.
Security & Data Boundaries (skill): https://www.openagentschool.org/ai-skills#module/security-data-boundaries - Map which data each agent can read and which actions it can take.
References:
OWASP Gen AI Security Project - LLM01:2025 Prompt Injection: https://genai.owasp.org/llmrisk/llm01-prompt-injection/
OWASP Cheat Sheet Series - LLM Prompt Injection Prevention Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
Greshake et al. - Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection: https://arxiv.org/abs/2302.12173

LESSON: Tool Calling
URL: https://www.openagentschool.org/ai-ml-foundations/tool-calling
Also called: function calling, tool use, action proposals, agent tools
Summary: A model proposes a tool call; your code validates its arguments, checks permission, and decides to execute.
In everyday words: Tool calling lets a model return a structured request to run a function you defined, such as "look up booking 42". The model does not run anything. Your application receives the proposed call, checks the arguments against the tool's schema, confirms the signed-in user is allowed to do it, executes it, and sends the result back to the model.
Why it matters: Treating a proposed call as permission is one of the fastest ways to give an agent too much power. Arguments can be malformed, a user can steer the model toward someone else's records, and injected text can propose harmful actions. OpenAI's function-calling flow and the Model Context Protocol (MCP) tools guidance both keep validation and access control in application code.
Scenario - A fictional climbing-gym booking agent: An invented climbing gym lets members chat with an agent that can view, book, and cancel lesson slots. In one synthetic session the model proposes eight calls: one has a malformed date, and two try to cancel slots belonging to other members. Decide where each bad call should be stopped.
Analogy - A request slip at a rental counter: A customer fills in a slip asking to rent a power washer. The clerk checks the slip is complete, checks the customer's membership card, and only then unlocks the cabinet. Writing the slip never hands over the tool.
Where the analogy stops: A clerk can ask clarifying questions face to face. Software must encode every check ahead of time, and the model writing the slip may have been influenced by text the customer never saw.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/tool-calling.webp
Picture description: A blank request slip with a pen on a rental counter, a card reader with a membership card, and a padlocked tool cabinet with a key on a hook.
Picture caption: Writing the request slip never hands over the tool; the counter checks come first.
What you see: Proposed call. What it represents: The request slip represents a structured tool call the model proposes, not an action.
What you see: Authorization. What it represents: The card reader stands for the server checking the signed-in user's permission.
What you see: Execution. What it represents: The locked cabinet represents the application running the tool only after checks pass.
Predict: With argument validation on but server authorization off, how many of the two unauthorized cancellations will execute?
Try: Start with 1 malformed call, 2 unauthorized calls, validation on, and authorization off. You should see 1 call rejected by validation, 2 unauthorized calls executed, and 5 legitimate calls executed. Turn authorization on, then turn validation off.
Explain: Schema validation catches the malformed date but has no idea who owns a slot, so both cancellations run. Server authorization rejects them by checking the session member. Turning validation off lets the malformed call reach the executor, where it may crash or do the wrong thing.
Worked engineering example: Handling a proposed cancellation
1. Parse the proposed call and validate slot_id and reason against the schema. Why: Rejects malformed or out-of-range arguments before any lookup.
2. Load the slot and compare its owner with the session member ID. Why: Authorization uses trusted records, not the model's claim.
3. Ask the member to confirm the cancellation in the interface. Why: High-impact actions deserve a human yes.
4. Execute, then return a short result or denial as the tool output. Why: The model can explain the outcome without seeing private data.
Outcome: Malformed and unauthorized cancellations are refused with clear tool results, and only confirmed, permitted ones change the schedule.
Tradeoff - Confirmation for every write: Gain: Members see each change before it happens. Cost or risk: More clicks and slower flows.
Tradeoff - Strict tool schemas: Gain: Fewer malformed calls reach your code. Cost or risk: Schema changes need coordinated releases.
Before real use: Tool handlers derive user identity from the session, never from model arguments.
Before real use: Every tool validates arguments server-side even when the provider enforces a schema.
Before real use: Tools are rate-limited and time out cleanly.
Before real use: Denied calls are logged with the reason for later review.
Your turn: A get_booking tool takes booking_id. Name the check that stops one member reading another member's booking.
Hint: Reading is also an action that needs permission.
Compare your reasoning: In the handler, load the booking and return it only if its member ID equals the session member ID; otherwise return a not-found or denied result.
Optional math: executed = L + U × (1 − Z) + M × (1 − V)
L: Legitimate, well-formed, permitted calls the model proposed.
U: Proposed calls the signed-in member is not permitted to make.
Z: Server authorization switch: 1 when each call is checked against the session.
M: Proposed calls with malformed arguments.
V: Argument validation switch: 1 when arguments are checked against the schema.
Each check removes one kind of bad call, and neither replaces the other. In this illustrative accounting model the eight proposals are fixed and the bad subsets do not overlap; real calls can be both malformed and unauthorized.
Common misconception: If the model chose to call the cancel tool, the user must be allowed to cancel.
Correction: A tool call is a proposal, not permission or execution. Validate arguments, authorize on the server with the real session identity, and confirm high-impact actions with the user.
Check your understanding: The model proposes cancel_slot with a valid slot ID that belongs to another member. What should your code do?
Execute it, because the arguments match the schema. - Matching the schema only proves the shape is right. Ownership is a separate fact your server must check before acting.
Ask the model whether the member owns the slot. - The model may guess or have been steered. Ownership should come from your database, not from another model answer.
Deny it after the server checks slot ownership. - Correct. Authorization uses the signed-in member and your records, and the denial can be returned to the model as a tool result.
Use it with purpose: You are adding a refund tool to the booking agent. What must be true before it can execute automatically?
Guidance: Arguments validate, the server confirms the member paid for that booking, the amount is within a policy limit, and larger refunds wait for staff approval. The model never supplies the member identity.
Where to go next:
Tool Use & Function Calling (concept): https://www.openagentschool.org/concepts/tool-use-function-calling - Follow the full propose, execute, and return loop with real tool schemas.
Agent Security (concept): https://www.openagentschool.org/concepts/agent-security - Place tool authorization inside a wider least-privilege design.
Policy-Gated Tool Invocation (pattern): https://www.openagentschool.org/patterns/policy-gated-tool-invocation - See a pattern that inserts policy checks between proposal and execution.
MCP Server Builder (course): https://www.openagentschool.org/courses/mcp-server-builder - Build tool servers that validate inputs and enforce access control themselves.
References:
OpenAI API docs - Function calling (application executes tool calls): https://developers.openai.com/api/docs/guides/function-calling
Model Context Protocol specification - Tools: security considerations: https://modelcontextprotocol.io/specification/2025-06-18/server/tools

LESSON: Reranking
URL: https://www.openagentschool.org/ai-ml-foundations/reranking
Also called: second-stage ranking, cross-encoder reranker, semantic reranking, two-stage retrieval
Summary: A reranker reorders candidates more carefully, but it cannot recover a passage the first search never returned.
In everyday words: Reranking is a second retrieval step. A fast first stage returns a shortlist of candidates (K of them); a slower, more careful model, often a cross-encoder that reads the question and passage together, scores each candidate and reorders them. Only the top few (n) go to the language model.
Why it matters: Rerankers often improve which passages land at the top, so teams sometimes treat them as a fix for weak retrieval. But the reranker only sees what the first stage handed over. If a needed passage is ranked 17th and K is 5, no reranker can bring it back, and each extra candidate costs more scoring time.
Scenario - A fictional houseplant help center: An invented plant shop has 20 synthetic care passages for a question about yellowing leaves. Three passages truly answer it, sitting at first-stage ranks 2, 9, and 17. You must choose how many candidates to rerank and how many to keep for the answer.
Analogy - Picking apples from a crate: A quick picker grabs a few apples from a big crate into a basket. A careful sorter then lines up the basket's apples from best to worst. The sorter can do a great job ordering the basket, but the best apple left in the crate never gets considered.
Where the analogy stops: Apples do not change with the question. In retrieval, which passages are best depends on the query, and real rerankers also make mistakes, so this lesson assumes an ideal sorter as an upper bound.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/reranking.webp
Picture description: A crate of apples with a glossy apple left behind, a small basket of five picked apples, and three apples lined up on a wooden board.
Picture caption: The careful sorter orders the basket but never sees the apple left in the crate.
What you see: Missed candidate. What it represents: The glossy apple left in the crate represents a relevant passage outside the first-stage shortlist.
What you see: Candidate pool. What it represents: The basket stands for the top K candidates handed to the reranker.
What you see: Reranked top n. What it represents: The ordered row represents the few passages kept after careful reranking.
Predict: If you rerank only the first 5 candidates and keep 3, what share of the three relevant passages can the answer use?
Try: Start with K = 5 and n = 3. You should see 1 relevant passage in the pool and a recall at n of 0.3333 both with and without the ideal reranker. Raise K to 10, then to 20.
Explain: Only rank 2 is inside the first 5, so even a perfect reranker reaches one third. At K = 10 the pool holds two relevant passages and reranking lifts recall to 0.6667 while the plain top 3 stays at 0.3333. At K = 20 recall reaches 1.0, but the reranker now scores 20 passages.
Worked engineering example: Choosing K for the plant help center
1. Label relevant passages for 50 real-style questions. Why: You cannot tune K without knowing which passages should be found.
2. Measure first-stage recall at K = 5, 10, 20, and 40. Why: Shows the ceiling each K sets for the reranker.
3. Time the reranker at each K on your hardware. Why: Scoring cost grows with every candidate.
4. Pick the smallest K that captures most relevant passages within the latency budget. Why: Balances answer quality against response time.
Outcome: The team reranks 20 candidates, captures nearly all labeled relevant passages, and stays within its latency target.
Tradeoff - Larger K: Gain: Higher ceiling for recall. Cost or risk: More reranker computation and latency.
Tradeoff - Smaller n kept: Gain: Shorter, cheaper prompts. Cost or risk: Fewer chances to include all needed evidence.
Before real use: First-stage recall at K is tracked separately from final answer quality.
Before real use: Reranker latency is measured at the production K.
Before real use: Questions where relevant passages fall outside K are logged for retrieval fixes.
Your turn: Relevant passages sit at ranks 3, 6, and 25. With K = 10 and n = 2, what is the ideal-reranker recall?
Hint: Count relevant passages within 10, then cap at n.
Compare your reasoning: Two relevant passages are within K, and n = 2 keeps both, so recall is 2/3 ≈ 0.6667. Rank 25 stays unreachable.
Optional math: recall@n (ideal rerank) = min(R_K, n) / R
R_K: Relevant passages inside the first-stage top K.
n: Passages kept after reranking for the answer.
R: Total relevant passages for the question; three here.
The reranker can raise recall up to the ceiling R_K / R set by the first stage, never above it. Real rerankers fall below this ideal upper bound. Positions here are fixed synthetic values; nothing is retrieved or scored by a real model.
Common misconception: Adding a strong reranker fixes poor first-stage retrieval.
Correction: A reranker cannot restore missing candidates. Measure first-stage recall at K, improve it with hybrid search or better chunking, and then use reranking to order the shortlist.
Check your understanding: After adding a reranker, answers still miss a key passage that the first stage ranks 40th. What should you change?
Swap in a larger reranker model. - A larger reranker still never sees the 40th passage if K is smaller. The ceiling comes from the first stage.
Raise K or improve first-stage retrieval. - Yes. The passage must enter the candidate pool before any reranker can promote it. Watch the extra scoring cost as K grows.
Keep more passages after reranking. - Keeping more of the reranked list does not help when the passage was never in it, and it adds context cost.
Use it with purpose: Reranking 100 candidates is too slow for your chat latency target. What do you trade?
Guidance: Measure first-stage recall at several K values on labeled questions. Pick the smallest K that keeps most relevant passages, and invest in better first-stage retrieval rather than reranking everything.
Where to go next:
Retrieval Engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Tune first-stage recall and second-stage ordering as separate, measured steps.
Agentic RAG (pattern): https://www.openagentschool.org/patterns/agentic-rag - See how an agent can retry retrieval when the shortlist lacks evidence.
Context & Memory Engineering (course): https://www.openagentschool.org/courses/context-memory-engineering - Decide how many passages earn a place in a limited context window.
References:
Nogueira and Cho - Passage Re-ranking with BERT: https://arxiv.org/abs/1901.04085
Elastic docs - Semantic reranking on a small top-k result set: https://www.elastic.co/docs/solutions/search/ranking/semantic-reranking

LESSON: Evaluation Sets
URL: https://www.openagentschool.org/ai-ml-foundations/eval-sets
Also called: eval datasets, golden sets, held-out test sets, regression suites
Summary: Build eval sets that match real traffic, freeze a held-out set, block leakage, and report slices.
In everyday words: An evaluation set is a fixed collection of inputs with expected behavior used to measure a system. A development set is used freely while building. A frozen held-out set is kept separate and touched rarely, so it can give an honest final reading. Slices are labeled subgroups, such as non-English requests, reported separately.
Why it matters: One average score can hide that a system fails a group of users. If the eval mix differs from real traffic, the average misleads. If development examples leak into the held-out set, scores rise for the wrong reason. Thin slices produce noisy numbers that tempt teams to celebrate or panic over chance.
Scenario - A fictional city-library help bot: An invented library builds 190 synthetic eval cases: 140 common questions, 30 about fines and refunds, 10 accessibility requests, and 10 in languages other than English. Real traffic is 20% non-English. A teammate copied ten development examples into the held-out set. Decide what to report.
Analogy - A spice rack and a sealed tasting box: A cook keeps practice jars on an open rack, grouped by cuisine, and a sealed box of jars used only for the final tasting. If one cuisine has two jars, any verdict about it is shaky. If a practice jar slips into the sealed box, the final tasting is no longer a fair test.
Where the analogy stops: Spices do not change with the season the way user requests do. Eval sets need refreshing as real traffic shifts, and even a sealed set loses value if the team checks it after every tweak.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/eval-sets.webp
Picture description: A spice rack with grouped jars and one sparse section, a sealed wooden box tied with twine below, and a practice jar tipped toward the box.
Picture caption: Group cases by slice, notice thin groups, and keep practice jars out of the sealed box.
What you see: Slices. What it represents: The grouped jar sections represent eval cases tagged by user group or task type.
What you see: Thin slice. What it represents: The sparse section stands for a group with too few cases to report reliably.
What you see: Frozen held-out. What it represents: The sealed box represents a held-out set kept apart from development work.
What you see: Leakage risk. What it represents: The tipping practice jar represents development examples slipping into the held-out set.
Predict: With 10 leaked development cases, will the reported eval-set score be higher or lower than the clean score, and which estimate is closer to real traffic?
Try: Start with 10 non-English cases, 10 leaked cases, and a minimum reported slice of 20. You should see a reported score of 0.8768, a clean score of 0.8726, 0.42 points of leakage inflation, a production-weighted estimate of 0.827, and 2 slices below the minimum. Set non-English cases to 0, then raise leakage to 40.
Explain: Leaked cases score as if memorized, adding 0.42 points here and 1.68 at 40 leaked cases. Weighting slices by real traffic lowers the estimate because non-English cases score worst and are underrepresented. With zero non-English cases, the slice is unmeasured and only 80% of traffic is covered.
Worked engineering example: Repairing the library eval set
1. Deduplicate held-out cases against development cases using exact and near-duplicate matching. Why: Removes leakage that inflates scores.
2. Tag every case with a slice and compare slice shares with real traffic. Why: Reveals coverage gaps such as non-English requests.
3. Add new cases until each important slice meets a minimum size. Why: Small slices give noisy readings.
4. Freeze the held-out set with a version ID and limit who can run it. Why: Repeated peeking turns the held-out set into another development set.
Outcome: The team reports a traffic-weighted score plus per-slice results with counts, and the held-out reading is trustworthy again.
Tradeoff - Larger slices for rare groups: Gain: Stable per-slice readings. Cost or risk: More cases to write and review.
Tradeoff - Refresh the held-out set often: Gain: Tracks changing traffic. Cost or risk: Harder to compare results across versions.
Before real use: Held-out sets are versioned and deduplicated against development and training data.
Before real use: Every report lists slice scores with case counts.
Before real use: Slice shares are compared with recent real traffic each quarter.
Before real use: Access to run the held-out set is logged so overuse is visible.
Your turn: Slices score 0.9 (70% of traffic) and 0.5 (30% of traffic). Your eval set is 95% from the first slice. Estimate the traffic-weighted score.
Hint: Weight by traffic, not by eval counts.
Compare your reasoning: 0.7 × 0.9 + 0.3 × 0.5 = 0.78, well below the 0.88 the eval mix would report.
Optional math: production estimate = Σ w_s × q_s / Σ w_s  (measured slices only)
w_s: Share of real traffic belonging to slice s.
q_s: Score of the system on eval cases in slice s.
Σ: Sum over the slices that have at least one eval case.
The eval-set average weights slices by how many cases you happened to write; this formula weights them by how often users actually arrive. Slice scores here are fixed synthetic values in an illustrative accounting model, and leaked cases are assumed to score 1.0.
Common misconception: A high average score on a large eval set proves the system works for everyone.
Correction: Check coverage against real traffic, keep held-out cases frozen and free of development leakage, and report slices with their sample sizes so weak groups stay visible.
Check your understanding: Your eval average rose after copying several tuning examples into the held-out set. What does that rise tell you?
Little; leaked cases inflate the score, so remove them and re-score. - Correct. Development examples in the held-out set reward memorization. Deduplicate against development data and rerun the frozen set.
The system improved, because the eval set got larger. - A larger set only helps when new cases are independent of tuning. These cases were already seen during development.
Nothing, as long as the average is above 0.85. - Thresholds do not fix contamination. A contaminated score above 0.85 can still hide a real drop on unseen requests.
Use it with purpose: Your non-English slice has only 10 cases but 20% of real traffic. Ship based on the overall average?
Guidance: No. Add enough non-English cases to report that slice on its own, show slice scores with counts, and weight the headline by real traffic before deciding.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Connect eval sets to the broader practice of measuring agent behavior.
Agent Testing & Benchmarks (concept): https://www.openagentschool.org/concepts/agent-testing-benchmarks - Compare public benchmarks with your own frozen, traffic-matched suites.
Benchmarking & Evaluating Agents (course): https://www.openagentschool.org/courses/benchmarking-evaluating-agents - Design slices, held-out sets, and regression gates in a guided course.
Observability & EvalOps (skill): https://www.openagentschool.org/ai-skills#module/observability - Turn production traces into new eval cases without leaking them into tuning.
References:
OpenAI API docs - Evaluation best practices (datasets matching production): https://developers.openai.com/api/docs/guides/evaluation-best-practices
scikit-learn User Guide - Cross-validation: evaluating estimator performance: https://scikit-learn.org/stable/modules/cross_validation.html

LESSON: Chunking
URL: https://www.openagentschool.org/ai-ml-foundations/chunking
Also called: text splitting, document segmentation, chunk overlap, passage windows
Summary: Split documents so key passages stay whole, and count the duplicate text and context that overlap adds.
In everyday words: Chunking splits long documents into smaller passages before they are embedded and indexed for retrieval. Chunk size sets how many tokens each passage holds. Overlap repeats some tokens from the end of one chunk at the start of the next so that sentences near a boundary are not cut in half.
Why it matters: Retrieval returns chunks, not whole documents. A key instruction split across two chunks may never be retrieved whole, so the model sees only half. Overlap helps, but every overlapped token is stored twice and may be retrieved twice, using context budget on repeated text.
Scenario - A fictional espresso-machine manual: An invented appliance company has a 1,200-token synthetic manual. The descaling warning sits at tokens 580 to 640. Customers ask about descaling, and the assistant retrieves three chunks per question. Choose a chunk size and overlap so the warning stays intact without wasting too much context.
Analogy - Cutting a long map into overlapping tiles: To carry a long trail map, you cut it into tiles that overlap a little at the edges. A river bend near a cut still appears whole on one tile. But every overlap strip is paper you carry twice.
Where the analogy stops: Map tiles are cut by position only. Good text splitters can also respect headings, paragraphs, and tables, so fixed windows are the simplest case rather than the recommended default.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/chunking.webp
Picture description: Overlapping watercolor map tiles in a row, a single tile with a whole river bend, and a stack of extra tiles beside scissors.
Picture caption: Overlap keeps a key feature whole on one tile, but every overlap strip is carried twice.
What you see: Overlap. What it represents: The shingled strips represent tokens repeated between neighboring chunks.
What you see: Passage kept whole. What it represents: The river bend inside one tile stands for a key passage that is retrieved intact.
What you see: Duplicate cost. What it represents: The extra stack represents more indexed and retrieved text created by overlap.
Predict: With 300-token chunks and 50 tokens of overlap, will any single chunk hold the whole 580–640 warning?
Try: Start with size 300, overlap 50, and 3 retrieved chunks. You should see 5 chunks, 1,400 indexed tokens, 200 duplicate tokens, 1 chunk holding the whole key passage, and 900 context tokens reserved. Set overlap to 0, then try size 100 with overlap 50.
Explain: With a 250-token stride the third chunk spans 500 to 800, covering the warning. With no overlap, chunks break at 600, splitting the warning in two, so no chunk holds it whole. Smaller chunks with overlap rescue it again but create many more chunks and duplicates.
Worked engineering example: Chunking the espresso manual
1. Split at section headings so the descaling section starts its own chunk. Why: Natural boundaries keep related sentences together.
2. Cap long sections at 300 tokens with 50 tokens of overlap. Why: Keeps chunks retrievable while protecting boundary sentences.
3. Prefix each chunk with its section title. Why: Gives the embedding context about where the passage came from.
4. Test 30 labeled questions and check that each key passage arrives whole. Why: Confirms the settings work on the questions that matter.
Outcome: The descaling warning is retrieved intact, and retrieved context holds little duplicated text.
Tradeoff - Larger chunks: Gain: Fewer boundary splits and more surrounding context. Cost or risk: Less precise matches and more context per retrieved chunk.
Tradeoff - More overlap: Gain: Fewer split passages. Cost or risk: More duplicates in the index and in retrieved context.
Before real use: Chunk settings are versioned with the index so results are reproducible.
Before real use: Labeled questions confirm key passages are retrieved whole after any change.
Before real use: Near-duplicate retrieved chunks are merged before building the prompt.
Your turn: A 1,000-token document is chunked at size 400 with overlap 100. How many chunks are created, and how many tokens are duplicated?
Hint: Stride is 300; chunks start at 0, 300, and 600.
Compare your reasoning: Three chunks: 0–400, 300–700, and 600–1000, totaling 1,200 tokens, so 200 tokens are duplicated.
Optional math: stride = size − overlap;  duplicates = Σ chunk lengths − D
size: Maximum tokens in one chunk.
overlap: Tokens shared between neighboring chunks; must be smaller than size.
stride: Distance between the starts of neighboring chunks.
D: Total tokens in the original document, 1,200 here.
Overlap shrinks the stride, so more chunks cover the same document and more tokens are stored twice. The experiment rejects overlap at or above chunk size because the windows would never advance. Context reserve is top-k × size, an upper bound for budgeting.
Common misconception: More overlap is always safer, so set it as high as possible.
Correction: Overlap protects boundary passages but adds duplicate index entries, near-identical retrieved chunks, and context cost. Pick it by measuring retrieval on labeled questions, and prefer structure-aware splits first.
Check your understanding: Retrieved context often contains two chunks that are mostly the same text. What is the likely cause?
The embedding model is broken and should be replaced. - Neighboring chunks with heavy overlap really are nearly the same text, so similar embeddings are expected rather than a sign of failure.
Chunk size is too small to hold any sentence. - Very small chunks cause fragments, but repeated near-identical text points more directly at overlap between neighbors.
Large overlap makes neighbors share much of their text. - Right. Reduce overlap, deduplicate neighbors at retrieval time, or split at headings so boundaries fall in natural places.
Use it with purpose: Your manuals have clear headings, short sections, and some long tables. How should you chunk them?
Guidance: Split at headings first, keep each table with its caption, apply a size cap with modest overlap only for long sections, and verify on labeled questions that key passages are retrieved whole.
Where to go next:
RAG Basics (concept): https://www.openagentschool.org/concepts/rag-basics - See where chunking sits between ingestion, embedding, and retrieval.
Retrieval Engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Measure how chunk choices change recall on labeled questions.
Context & Memory Engineering (course): https://www.openagentschool.org/courses/context-memory-engineering - Budget retrieved chunks inside a finite context window.
References:
Microsoft Learn - Chunk large documents for vector search solutions in Azure AI Search: https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents
Lewis et al. - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
Anthropic - Contextual Retrieval: adding context to chunks before embedding: https://www.anthropic.com/research/contextual-retrieval

LESSON: Grounding & Citations
URL: https://www.openagentschool.org/ai-ml-foundations/grounding
Also called: attribution, citation verification, evidence support, faithfulness
Summary: A citation is only useful when the cited passage actually supports the claim; check support, not just presence.
In everyday words: Grounding means an answer's claims are supported by specific evidence, such as retrieved passages. A citation points to that evidence. Attribution checking asks a harder question: does the cited passage actually entail the claim, meaning a careful reader would agree the passage says it? Citation presence and evidence support are different measurements.
Why it matters: It is easy to make every sentence carry a citation, and dashboards love a 100% citation rate. But a citation to a related-but-different passage gives readers false confidence. Research on attribution and cited generation measures whether citations support claims, because coverage alone rewards the wrong behavior.
Scenario - A fictional pet-clinic answer bot: An invented pet clinic answers owner questions using synthetic care leaflets. One answer contains eight claims: six cite a leaflet, two cite nothing. A reviewer finds that one cited leaflet discusses a different medicine dose. Decide how to measure and improve the answer's grounding.
Analogy - Sticky tabs in a reference book: A student marks each claim in an essay with a sticky tab in a reference book. A teacher who only counts tabs is impressed. A teacher who opens each tabbed page notices that one tab points to a picture of an apple when the essay is talking about pears.
Where the analogy stops: A teacher can read the whole book and judge nuance. Automatic support checkers estimate entailment and also make mistakes, so they need calibration against human reviewers.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/grounding.webp
Picture description: A wooden pear on a blank card, an open book with a tab marking an apple illustration, and a magnifying glass angled toward the tabbed page.
Picture caption: A tab shows where to look; only reading the page shows whether it supports the claim.
What you see: Claim. What it represents: The wooden pear stands for a statement in the answer that needs evidence.
What you see: Citation present. What it represents: The tabbed apple page represents a citation that points to evidence about something else.
What you see: Check support. What it represents: The magnifying glass represents verifying that the cited passage actually entails the claim.
Predict: At a support-check threshold of 0.6, will every cited claim that passes the check truly be supported?
Try: Start with threshold 0.6 and cite-anything off. You should see citation coverage of 0.75, two thirds of cited claims truly supported, 4 claims accepted, 1 unsupported claim accepted, and 1 supported claim flagged. Raise the threshold to 0.7, then turn cite-anything on at 0.6.
Explain: Claim C4 cites a related leaflet with a support score of 0.68, so it slips through at 0.6, while supported claim C3 at 0.58 is flagged. At 0.7, C4 is caught but C3 is still flagged. Attaching the nearest passage to uncited claims raises coverage to 1.0 yet lowers the truly supported share to 0.5.
Worked engineering example: Verifying the pet-clinic answer
1. Split the answer into individual factual claims. Why: Support is judged claim by claim, not for the whole answer.
2. Pair each claim with its cited passage and run a support check. Why: Separates claims backed by evidence from those that only look backed.
3. For low-scoring or uncited claims, retrieve again or soften the wording. Why: Avoids presenting unsupported statements as fact.
4. Have a reviewer label a weekly sample to calibrate the checker threshold. Why: The checker itself makes mistakes in both directions.
Outcome: The dose claim with a mismatched leaflet is caught and rewritten, and the team reports both coverage and verified support.
Tradeoff - Higher support threshold: Gain: Fewer unsupported claims shown. Cost or risk: More supported claims flagged for rework.
Tradeoff - Abstain when unsupported: Gain: Owners see fewer confident errors. Cost or risk: Some answers become shorter or less helpful.
Before real use: Grounding dashboards show verified support rate next to citation coverage.
Before real use: Support-check thresholds are calibrated on human-labeled samples.
Before real use: Citations link to the exact passage version used at answer time.
Your turn: Five claims all have citations; reviewers find three are supported. State coverage and verified support rate.
Hint: Coverage counts citations; support counts verified claims among cited ones.
Compare your reasoning: Coverage is 5/5 = 1.0, but verified support is 3/5 = 0.6, so two claims need new evidence or softer wording.
Optional math: coverage = cited / claims;  accept claim when support ≥ τ
cited: Number of claims with at least one attached citation.
claims: Total factual claims in the answer, eight here.
support: Illustrative checker score estimating whether the passage entails the claim.
τ: Acceptance threshold for the support check.
Coverage counts citations regardless of content, so it can rise while support falls. The threshold trades unsupported claims let through against supported claims sent for review. Scores are fixed synthetic values standing in for an entailment checker; human labels are the reference.
Common misconception: An answer where every sentence has a citation is well grounded.
Correction: Citation presence does not equal supporting evidence. Check that each cited passage entails its claim, calibrate the checker against human review, and remove or soften unsupported claims.
Check your understanding: Citation coverage jumped from 75% to 100% after a change, but reviewers report more wrong answers. What likely happened?
Reviewers became stricter, so the change is fine. - Possible, but coverage alone cannot tell you. Check whether the new citations actually support their claims before blaming reviewers.
Citations were attached without checking that they support the claims. - Yes. Attaching the nearest passage raises coverage without adding evidence. Measure support, not just presence.
The retriever now returns too many passages. - More passages can add noise, but the specific pattern of higher coverage with more errors points at unsupported citations.
Use it with purpose: Your team wants to report "percentage of sentences with citations" as the grounding metric. Accept it?
Guidance: Report it only next to a support rate measured on a human-reviewed sample, and flag or soften claims whose citations fail the support check. Track both, because they can move in opposite directions.
Where to go next:
Hallucination & Grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Explore why models produce unsupported claims and how evidence reduces them.
RAG Basics (concept): https://www.openagentschool.org/concepts/rag-basics - See how retrieved passages become the evidence that citations point to.
Agentic RAG (pattern): https://www.openagentschool.org/patterns/agentic-rag - Watch an agent re-retrieve when evidence does not support its draft.
Fine-Tuning vs. RAG vs. Prompting (course): https://www.openagentschool.org/courses/fine-tuning-rag-prompting - Decide when retrieval with verified citations beats other approaches.
References:
Rashkin et al. - Measuring Attribution in Natural Language Generation Models: https://arxiv.org/abs/2112.12870
Gao et al. - Enabling Large Language Models to Generate Text with Citations: https://arxiv.org/abs/2305.14627

LESSON: Observability & Tracing
URL: https://www.openagentschool.org/ai-ml-foundations/observability-basics
Also called: tracing, telemetry, OpenTelemetry, spans
Summary: Trace each AI request step by step, keep rare failures, and redact sensitive data instead of logging secrets.
In everyday words: Observability is the ability to understand what a system did from the signals it emits. A trace follows one request through its steps; each step is a span with timing and attributes. OpenTelemetry (OTel) is an open standard for these signals, and its generative AI conventions name attributes such as model and token counts while marking message content as opt-in.
Why it matters: AI requests pass through guards, retrieval, model calls, and tools, and any of them can fail quietly. Traces show where. But capturing prompts and tool arguments can copy personal data and access tokens into a system many people can read. Good tracing records enough to debug while keeping secrets out.
Scenario - A fictional bike-rental support assistant: An invented bike-rental service handles 10,000 synthetic assistant requests a day, about 200 of which fail. Each request includes a guard, retrieval, a model call, and a tool call with the rider's email and an access token. Decide what to sample, what to capture, and what to redact.
Analogy - A parcel tracking log with window envelopes: A courier records each checkpoint a parcel passes, so a late delivery can be traced. Window envelopes show the address the courier needs while hiding the letter inside, and the house keys never travel in the parcel at all.
Where the analogy stops: A parcel log records only location. Software traces can capture anything the code attaches, so nothing hides content automatically; the team must decide and enforce what is captured.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/observability-basics.webp
Picture description: Five kraft envelopes clipped along a cord on a corkboard, a desk envelope with its window covered by a grey strip, and a closed steel lockbox.
Picture caption: Record each checkpoint, mask what readers do not need, and keep keys out of the log.
What you see: Trace spans. What it represents: The clipped envelopes represent ordered steps recorded for one request.
What you see: Redaction. What it represents: The covered window stands for masking personal data before telemetry is exported.
What you see: Secrets stay out. What it represents: The closed lockbox represents access tokens that are never captured in telemetry.
Predict: With 10% sampling and content capture on but no redaction, how many access tokens end up in telemetry each day?
Try: Start with 10% sampling, error tail rule off, content capture on, and redaction off. You should see 1,000 traces, 20 of 200 errors traced, 3,000 sensitive values, and 1,000 access tokens stored. Turn on redaction, then the error tail rule, then turn content capture off.
Explain: Every sampled trace copies three sensitive fields, including the token. Redaction removes the email and token patterns but misses a free-text personal detail, leaving 1,000 values. The tail rule keeps all 200 errors for debugging, at the cost of 180 extra traces. Turning content capture off keeps timing and counts without message content.
Worked engineering example: Instrumenting the rental assistant
1. Create one trace per request with spans for guard, retrieval, model, and tool. Why: Shows which step failed or slowed down.
2. Record model name, token counts, latency, and retrieved document IDs as attributes. Why: Low-risk metadata answers most debugging questions.
3. Pass tokens through headers the tracer never captures, and redact emails before export. Why: Keeps secrets out and reduces personal data exposure.
4. Sample 10% of requests and keep every error trace. Why: Controls volume while preserving rare failures.
Outcome: Engineers can trace any failure to a step without anyone being able to read riders' tokens in telemetry.
Tradeoff - Capture message content: Gain: Easier to reproduce bad answers. Cost or risk: Personal data in telemetry and stricter access needs.
Tradeoff - Higher sampling rate: Gain: More complete picture of normal behavior. Cost or risk: Higher storage cost and more data to protect.
Tradeoff - Tail rule for errors: Gain: Every failure is traceable. Cost or risk: Tail decisions need buffering, adding pipeline complexity.
Before real use: Secrets are blocked at the source and scanned for in exported telemetry.
Before real use: Content capture is off by default and enabled only with documented need.
Before real use: Redaction rules are tested with synthetic personal data before release.
Before real use: Telemetry access is role-restricted and retention is limited.
Your turn: At 5% head sampling with 10,000 requests and 200 errors, how many traces are stored with the error tail rule on?
Hint: Sampled traces plus errors not already sampled.
Compare your reasoning: 500 sampled traces plus 190 unsampled errors equals 690 traces per day.
Optional math: traces = N × s + E × (1 − s) × T
N: Requests per day, 10,000 in this scenario.
s: Head sampling rate as a fraction.
E: Failed requests per day, 200 in this scenario.
T: Tail rule switch: 1 when every error trace is kept.
Head sampling keeps a random share of all requests; a tail rule adds failed ones that sampling missed. Sensitive values grow with traces times fields captured. This is an illustrative accounting model with fixed counts, not a real telemetry pipeline.
Common misconception: Logging full prompts and tool arguments is fine because only engineers can see the logs.
Correction: Telemetry is often widely accessible and retained for a long time. Never send secrets; capture content only when needed, redact before export, and restrict and expire access.
Check your understanding: A tool span records the rider's access token in its arguments. What is the right fix?
Stop attaching the token and redact secret patterns before export. - Correct. Secrets should never be captured; redaction is defense in depth for anything that slips through. Rotate any token already exposed.
Keep it, because traces are only kept for 30 days. - A token in telemetry can be misused within minutes. Short retention limits exposure time but does not prevent it.
Lower the sampling rate so fewer tokens are stored. - Fewer traces means fewer exposed tokens, but every sampled trace still leaks one. Remove the secret instead.
Use it with purpose: Support wants full message content in every trace to debug complaints. What do you offer?
Guidance: Keep metadata on all sampled traces, capture redacted content only for errors or consented sessions, block secrets at the source, and limit who can read content with expiring access.
Where to go next:
Agent Observability (concept): https://www.openagentschool.org/concepts/agent-observability - Design traces, metrics, and dashboards specifically for multi-step agents.
Agent Ops (concept): https://www.openagentschool.org/concepts/agent-ops - Turn traces into alerts, incident reviews, and reliability improvements.
AgentOps Reliability Engineer (course): https://www.openagentschool.org/courses/agentops-reliability-engineer - Practice instrumenting and debugging agents in a reliability-focused course.
Observability & EvalOps (skill): https://www.openagentschool.org/ai-skills#module/observability - Connect traces to evaluations so failures become new test cases.
References:
OpenTelemetry docs - Traces: spans, attributes, and context propagation: https://opentelemetry.io/docs/concepts/signals/traces/
OpenTelemetry GenAI semantic conventions - Spans (content attributes are opt-in): https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md
OWASP Cheat Sheet Series - Logging Cheat Sheet (data to exclude or mask): https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html

LESSON: Idempotency
URL: https://www.openagentschool.org/ai-ml-foundations/idempotency
Also called: idempotency keys, safe retries, request deduplication, exactly-once effects
Summary: Make retries safe with a stable key per intent, a request fingerprint, and a durable record that expires deliberately.
In everyday words: An operation is idempotent when doing it twice has the same effect as doing it once. For actions such as bookings or payments, servers achieve this with an idempotency key: the client sends a unique key per intended action, the server stores the first result with a fingerprint (hash) of the request, and later requests with the same key replay that stored result instead of acting again.
Why it matters: Agents and networks retry. A timeout does not mean the action failed; it may have succeeded just before the connection dropped. Without idempotency, each retry creates another booking or charge. Comparing whether two model responses look the same does not help, because model output can vary and the side effect already happened.
Scenario - A fictional community-room booking agent: An invented community center lets an agent book meeting rooms. The booking call times out, and the agent retries twice an hour apart. Sometimes a member double-clicks, sending two identical requests at once. Decide how the server should make sure only one booking exists.
Analogy - A coat-check ticket: When you hand in a coat, you get a numbered ticket. Showing the same ticket again gets you the same coat, not a second one. Tickets from yesterday are cleared at closing, and showing a ticket with a different description gets questioned rather than honored.
Where the analogy stops: A coat check has one attendant handling one ticket at a time. Servers can receive the same key simultaneously on different machines, so the key record must be written atomically in a durable store.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/idempotency.webp
Picture description: Two matching coat-check stub halves, a single coat on a brass hook, a plain wall clock without numerals, and a spike holder of stubs.
Picture caption: The same ticket brings back the same coat; records are kept for a set time.
What you see: Stable key. What it represents: The matching stub halves represent one idempotency key reused on every retry.
What you see: One effect. What it represents: The single coat stands for exactly one booking no matter how many retries arrive.
What you see: Expiry window. What it represents: The plain clock represents how long the server keeps the key record.
What you see: Durable record. What it represents: The spike holder represents an atomic, shared store of keys and stored results.
Predict: With a stable key, a 24-hour record, and two retries one hour apart, how many bookings will be created?
Try: Start with 2 retries, a 1-hour gap, stable key strategy 2, a 24-hour record, atomic insert on, 1 simultaneous copy, and payload unchanged. You should see 1 booking and 2 replayed responses. Try strategy 1, then 2 copies with atomic insert off, then a 24-hour gap, then a changed final payload.
Explain: A stable key lets the server replay the first result, so retries do no harm. A new key per attempt creates 3 bookings. Without atomic insert, two simultaneous copies both see no record and create 2 bookings. A 24-hour gap outlasts the record, creating duplicates. A changed payload under the same key is rejected as a conflict.
Worked engineering example: Making room bookings safe to retry
1. Generate one random key when the agent decides to book, and store it in the task state. Why: Every retry of the same intent must reuse the same key.
2. On the server, atomically insert the key with a hash of member ID, endpoint, and normalized body. Why: Only one concurrent request wins, and the hash scopes the key to one caller and payload.
3. Store the booking result with the key once the action completes. Why: Later retries replay the stored result instead of booking again.
4. Reject reuse with a different hash, and expire records after the retry window. Why: Prevents silent mismatches and bounds storage.
Outcome: Timeouts, retries, and double-clicks all produce one booking, and a retry with altered details gets a clear conflict error.
Tradeoff - Longer key retention: Gain: Covers slow retry schedules. Cost or risk: More storage and older records to manage.
Tradeoff - Wait for in-flight duplicates: Gain: Callers receive the real result. Cost or risk: Holding connections open adds latency and load.
Before real use: Every state-changing tool accepts an idempotency key and requires it for retries.
Before real use: Key records are written atomically to a durable store shared by all servers.
Before real use: Fingerprints include caller identity so keys cannot be replayed across members.
Before real use: Key retention exceeds the longest client retry window.
Your turn: Retries arrive 13 and 26 hours after the original with a 24-hour key record and a stable key. How many bookings exist?
Hint: Check each retry against the record time: 13 is inside the window; 26 is outside.
Compare your reasoning: Two bookings: the 13-hour retry replays, but the 26-hour retry arrives after the record expires and books again. Extend retention or stop retrying sooner.
Optional math: replay when key matches, fingerprint matches, and t − t₀ < TTL
key: Client-generated unique value for one intended action.
fingerprint: Hash of the normalized request, scoped to the caller and endpoint.
t − t₀: Time since the key record was first stored.
TTL: Time to live: how long the server keeps the key record.
If the key matches but the fingerprint differs, the server rejects the request as a conflict rather than guessing. After the TTL the record is gone, so a late retry acts again. The experiment runs this rule on a fixed timeline, using an in-memory record to stand in for a durable store.
Common misconception: If the agent's retry produces the same model response as before, the action will not repeat.
Correction: Response similarity says nothing about side effects. Use a stable key per intent, store it atomically in a durable store with a scoped request fingerprint, and set an expiry longer than your retry window.
Check your understanding: Two identical booking requests with the same key arrive at the same instant on different servers. What prevents two bookings?
Checking whether the key exists, then inserting it if not. - Check-then-insert has a race: both servers can check before either inserts. You need a single atomic step.
Comparing the two model responses and keeping one. - Model responses do not control the database. Both bookings could already exist by the time you compare text.
An atomic insert-if-absent on a durable key store. - Correct. One request wins the insert and proceeds; the other waits for or replays the stored result.
Use it with purpose: Your agent retries failed bookings for up to 48 hours, but keys expire after 24. What should change?
Guidance: Align them: keep key records longer than the longest retry window, or stop retrying before expiry and ask the member. Keep the stable key in the agent's task state so every retry reuses it.
Where to go next:
Agentic Workflow Control (concept): https://www.openagentschool.org/concepts/agentic-workflow-control - Design retries, checkpoints, and resumption so agents never repeat side effects.
Tool Use & Function Calling (concept): https://www.openagentschool.org/concepts/tool-use-function-calling - Add idempotency keys to tools that change state.
Agent Platform Operations (course): https://www.openagentschool.org/courses/agent-platform-operations - Operate agent platforms where retries and failures are routine.
References:
IETF HTTPAPI Working Group - The Idempotency-Key HTTP Header Field (Internet-Draft): https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
Stripe API Reference - Idempotent requests (parameter comparison and key pruning): https://docs.stripe.com/api/idempotent_requests
RFC 9110 HTTP Semantics - Section 9.2.2 Idempotent Methods: https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2

LESSON: Working Memory: The Active Workspace
URL: https://www.openagentschool.org/ai-ml-foundations/working-memory
Also called: agent working memory, capacity based memory, active context, scratchpad
Summary: Choose what an agent needs for its next step within a limited active workspace.
In everyday words: An agent’s working memory is the information actively available for the current step: its goal, recent results, relevant evidence and temporary notes. Capacity matters: a model can only receive a bounded context, and useful attention is not guaranteed just because text fits. External state may be larger than what is supplied to one model call.
Why it matters: A long-running task can lose a critical constraint while retaining pages of irrelevant history. Decide what must stay active and what can be stored and retrieved later.
Scenario - Keep the constraint active: The agent is comparing train tickets, not buying one. Old search pages can be stored elsewhere. Message: "Goal: compare two options. Budget: 80 units. Do not purchase."
Analogy - A desk beside a filing cabinet: The desk holds the few things needed now. The cabinet can hold much more, but an item in the cabinet is not automatically on the desk.
Where the analogy stops: An agent does not have human short-term cognition. Token limits, application state and selection policies determine what is actually available.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/working-memory.webp
Picture description: Three selected task cards occupy a small desk tray, while a separate portable file box stores additional folders.
Picture caption: What is on the active desk can guide this step; what is stored elsewhere is not automatically present.
What you see: Active information. What it represents: The selected cards represent the current goal, restrictions and useful evidence.
What you see: Limited capacity. What it represents: The bounded desk space represents the context budget, not a human memory model.
What you see: Stored elsewhere. What it represents: The file box represents records that must be selected or retrieved before use.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Keep the constraint active. Context: The agent is comparing train tickets, not buying one. Old search pages can be stored elsewhere.
Text: Goal: compare two options. Budget: 80 units. Do not purchase.
Notice "compare two options" - Current goal: Keeps the immediate deliverable in view.
Notice "Budget: 80 units" - Decision constraint: Needs a defined currency in real use.
Notice "Do not purchase" - Action boundary: Must survive compaction and also be enforced by tools.
Proposed next step: Use the active workspace for goal, constraints and selected results; retrieve other material only when relevant.
Case: A lossy handoff. Context: A summary replaced the current task note, dropping the purchase restriction.
Text: The person wants train tickets. Several options were discussed.
Notice "wants train tickets" - Overgeneralized goal: Comparison became an open-ended desire.
Notice "Several options" - Low-information history: Does not preserve the budget, chosen evidence or action boundary.
Proposed next step: Repair the handoff from the original task before continuing; do not infer permission from the summary.
Guided practice: A document is too large for the active context. What should you retain?
Hint: Keep decision-changing passages and their source identifiers, not arbitrary first pages.
Compare your reasoning: Select relevant excerpts with provenance, preserve the goal and restrictions, reserve output space, and leave a way to retrieve omitted sections.
Predict: Which three pieces of information should survive a shrinking context budget?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Keep the current goal, decision-changing evidence and constraints. Compaction must not silently expand authority.
Optional math: active context = task + constraints + selected evidence + recent state, within a token budget
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Everything saved in the application is already known to the model.
Correction: Only information supplied or retrieved into the current call is available through that context. Storage capacity and active context capacity are different.
Check your understanding: What primarily constrains working memory for a model call?
The number of days since a note was written. - A new note can be absent from context and an old note can be active. Check selection and capacity.
The active context budget and what the application selects. - Yes. Capacity and availability define the active workspace; age can affect relevance but is not the definition.
Use it with purpose: What must remain visible when your agent resumes after a pause?
Guidance: Persist a structured checkpoint, then construct a minimal current context. Recheck permissions; a saved checkpoint is not renewed consent.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-working-memory-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-working-memory-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-working-memory-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-working-memory-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Memory & State (concept): https://www.openagentschool.org/concepts/memory-state?tab=why-memory - Take this distinction into Memory & State and check how it changes a real system decision.
References:
LangChain - Agent memory: working state, semantic, episodic and procedural memory: https://docs.langchain.com/oss/python/concepts/memory

LESSON: Episodic Memory: What Happened
URL: https://www.openagentschool.org/ai-ml-foundations/episodic-memory
Also called: agent episodic memory, event memory, experience memory, memory duration
Summary: Keep useful records of past events with time, outcome and a deliberate retention policy.
In everyday words: Episodic memory records particular events or experiences: what was attempted, when, under which conditions and with what result. Duration is a separate design choice. An event record may expire after an hour or remain for years; “episodic” describes its event-based content, not a mandatory short lifetime.
Why it matters: An agent can avoid repeating a failed attempt by retrieving what happened. But one past success is not a universal rule, and retaining every interaction indefinitely creates privacy and storage risks.
Scenario - A useful event record: A fictional agent attempted an upload during a service incident. Message: "At 09:10, upload U7 timed out; no completion receipt was found."
Analogy - A dated service log: The log says what happened on a particular visit, rather than declaring how every future visit will go.
Where the analogy stops: Logs can be incomplete or incorrect. Recalling an event is not re-experiencing it or proving its cause.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/episodic-memory.webp
Picture description: A coiled bicycle inner tube sits beside a blank visit record, a small clock and a removable archive sleeve.
Picture caption: An episode records what happened in particular circumstances; its retention period is a separate choice.
What you see: Specific event. What it represents: The removed tube represents one past attempt or service event, not every future case.
What you see: Time and context. What it represents: The visit sheet and clock represent metadata needed to interpret the event.
What you see: Retention choice. What it represents: The removable sleeve represents a deliberate storage and deletion policy.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A useful event record. Context: A fictional agent attempted an upload during a service incident.
Text: At 09:10, upload U7 timed out; no completion receipt was found.
Notice "At 09:10" - Event time: A real record needs date and time zone as well.
Notice "U7 timed out" - Observed event: Describes one attempt, not the service’s permanent behavior.
Notice "no completion receipt" - Uncertainty: A timeout alone does not prove the server did nothing.
Proposed next step: Reconcile the upload status before retrying; retain the event only as long as policy requires.
Case: An unsafe generalization. Context: The only evidence is the one timeout described above.
Text: Uploads to this service always fail.
Notice "always fail" - Unsupported general rule: One episode cannot establish an all-time property.
Proposed next step: Keep the event-specific evidence instead of promoting it into a universal fact.
Guided practice: A prior agent attempt succeeded. What should you verify before reusing it?
Hint: Compare conditions, not just the task label.
Compare your reasoning: Check tool version, input conditions, permissions and outcome evidence. Use the episode as a clue, not a guarantee.
Predict: Does retaining an episode for a year turn it into semantic memory?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: No. Its contents still describe a particular event. Retention duration, retrieval scope and memory type are distinct design axes.
Optional math: episode = event + time + context + outcome + source; retention = separate policy
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Episodic memory is defined by a short duration.
Correction: It is defined by event-based content. Both short-lived and durable stores can contain episodes.
Check your understanding: Which record is episodic?
“This service always fails.” - That is a generalized claim, and the single event does not support it.
“On this date, this attempt timed out under these conditions.” - Correct. It describes a bounded event with context.
Use it with purpose: What event history would help your agent, and when should it be deleted?
Guidance: Store minimal relevant records with provenance, access controls and an expiry/review rule. Do not retain personal conversations merely because storage is cheap.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-episodic-memory-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-episodic-memory-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-episodic-memory-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-episodic-memory-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Memory Systems (concept): https://www.openagentschool.org/concepts/agent-memory-systems?tab=types - Take this distinction into Agent Memory Systems and check how it changes a real system decision.
References:
LangChain - Agent memory: working state, semantic, episodic and procedural memory: https://docs.langchain.com/oss/python/concepts/memory

LESSON: Semantic Memory: Reusable Knowledge
URL: https://www.openagentschool.org/ai-ml-foundations/semantic-memory
Also called: agent semantic memory, durable facts, knowledge memory
Summary: Retain reusable facts and meanings while making correction and expiry possible.
In everyday words: Semantic memory stores reusable facts, concepts or explicitly chosen preferences rather than one event’s full story. In agents, this often means durable records outside the model that can be retrieved across tasks. Durable means retained beyond one interaction—not permanent, automatically true or exempt from deletion.
Why it matters: A useful remembered fact can become stale. Keep its source, scope and review date so an agent can update it instead of treating it as timeless authority.
Scenario - A scoped reusable fact: An authorized product specification defines the fictional device’s input. Message: "Device K accepts 5 V input under specification version 3."
Analogy - A maintained reference card: A reference card keeps a useful fact ready for later tasks, with enough information to know when it needs checking.
Where the analogy stops: A stored statement is a claim. Writing it down does not verify it or update a model’s weights.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/semantic-memory.webp
Picture description: A workshop component, a reusable reference card and a replacement card beside an open sleeve.
Picture caption: A reusable fact needs scope and maintenance; durable does not mean permanently correct.
What you see: A defined subject. What it represents: The component represents the specific thing to which a fact applies.
What you see: Reusable knowledge. What it represents: The reference card represents a sourced fact available across tasks.
What you see: Correct and update. What it represents: The replacement card represents review, expiry and correction rather than permanence.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A scoped reusable fact. Context: An authorized product specification defines the fictional device’s input.
Text: Device K accepts 5 V input under specification version 3.
Notice "Device K" - Scope: The fact belongs to this device, not every product.
Notice "5 V" - Value and unit: The unit must remain attached.
Notice "version 3" - Provenance: Check the applicable specification before reuse.
Proposed next step: Store a sourced, scoped fact with review rules; retrieve it when relevant.
Case: A stale claim challenged. Context: A new authorized specification changes the supported input; the old record remains in storage.
Text: Always use the remembered 5 V value.
Notice "Always" - Unjustified permanence: Durability does not remove the need to check applicability.
Notice "remembered 5 V" - Potentially stale fact: Resolve the new specification and supersede the old record explicitly.
Proposed next step: Reconcile versions and mark obsolete information, rather than letting conflicting facts coexist without explanation.
Guided practice: A person corrects a saved preference. What should happen?
Hint: Keeping two contradictory “current” values is not an update.
Compare your reasoning: Confirm the intended scope, replace or supersede the previous value, and respect the person’s deletion or retention choice.
Predict: What metadata makes a durable fact safer to reuse?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Scope, source, version and review rules help distinguish an applicable fact from an old assertion. Retrieval is not automatic truth checking.
Optional math: reusable claim + scope + source + version + review/deletion policy
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Semantic memory is permanent truth learned into the model.
Correction: An application memory store is usually external to model weights. Its contents can be wrong and should support correction and deletion.
Check your understanding: What makes a reusable fact dependable?
It has been stored for a long time. - Old information can become less reliable. Check its current applicability.
Appropriate provenance, scope, review and correction—not durability alone. - Yes. Retention age is not evidence of correctness.
Use it with purpose: Which facts actually merit cross-session retention in your agent?
Guidance: Keep only needed, permitted facts. Define who may write, read, correct and delete them; exclude sensitive inferences and unsupported guesses.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-semantic-memory-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-semantic-memory-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-semantic-memory-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-semantic-memory-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Memory Systems (concept): https://www.openagentschool.org/concepts/agent-memory-systems?tab=types - Take this distinction into Agent Memory Systems and check how it changes a real system decision.
References:
LangChain - Agent memory: working state, semantic, episodic and procedural memory: https://docs.langchain.com/oss/python/concepts/memory

LESSON: Procedural Memory: How to Do the Work
URL: https://www.openagentschool.org/ai-ml-foundations/procedural-memory
Also called: agent procedural memory, skills, runbooks, learned procedures
Summary: Reuse a reviewed method, while checking that it still fits and is allowed.
In everyday words: Procedural memory represents ways of doing tasks: an agent’s approved instructions, skills, runbooks or reusable workflows. Some research also discusses procedures encoded in model parameters. In an application, a stored procedure should have preconditions, steps, checks and a stopping rule—not just a successful-looking transcript.
Why it matters: Reusing a method can reduce repeated planning, but a stale procedure can repeat mistakes at scale. Knowing how to perform an action does not grant permission to perform it.
Scenario - A bounded procedure: A fictional refund workflow is approved for eligible orders only. Message: "Verify owner; check eligibility; show refund amount; obtain confirmation; execute once."
Analogy - A reviewed checklist at a workbench: The checklist helps someone repeat a process, including checks before starting and conditions for stopping.
Where the analogy stops: An agent is not developing human muscle memory. A written procedure still depends on correct tools, inputs and enforcement.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/procedural-memory.webp
Picture description: A blank reusable checklist, containers with a soft brush and a separately latched cupboard at a workbench.
Picture caption: Remembering how to do the work does not grant permission to start it.
What you see: Reviewed method. What it represents: The checklist represents steps, preconditions and stopping rules.
What you see: Task-specific tools. What it represents: The selected objects represent inputs and tools the method expects.
What you see: Separate permission. What it represents: The latched cupboard represents authority that a saved procedure cannot supply.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A bounded procedure. Context: A fictional refund workflow is approved for eligible orders only.
Text: Verify owner; check eligibility; show refund amount; obtain confirmation; execute once.
Notice "Verify owner" - Precondition: Authority must be checked for this request.
Notice "obtain confirmation" - Human decision point: Past approval does not authorize the next refund.
Notice "execute once" - Operational constraint: Use idempotency and reconciliation to avoid duplicate refunds.
Proposed next step: Run the current reviewed procedure only when all preconditions hold; otherwise stop or escalate.
Case: A copied shortcut. Context: An old transcript succeeded in a different account under a different policy.
Text: Skip eligibility checks and repeat the previous refund call.
Notice "Skip eligibility checks" - Missing safeguard: A transcript is not an approved procedure.
Notice "previous refund call" - Stale parameters: Old targets and credentials must not be reused blindly.
Proposed next step: Reject the shortcut; restore the reviewed checks and resolve current inputs.
Guided practice: Convert “the agent fixed it last time” into a reusable checklist. What is missing?
Hint: Start and stop conditions matter as much as the middle steps.
Compare your reasoning: Identify the applicable problem, required permissions, current inputs, bounded steps, success evidence and failure/rollback path. Review before reuse.
Predict: What separates a reusable procedure from a record of a past success?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: A procedure explicitly describes when it applies, what to do and how to verify or stop. An episode describes what happened once.
Optional math: procedure = preconditions + permitted steps + verification + stop/rollback rule + version
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A remembered skill carries permanent permission to use it.
Correction: A skill describes a method. Current authorization, safety constraints and approval still govern execution.
Check your understanding: A learned procedure conflicts with current policy. Which wins?
The procedure, because it worked previously. - Conditions change. A reusable method must remain subordinate to current policy.
Current authoritative policy; stop and revise the procedure. - Correct. Historical success does not override present constraints.
Use it with purpose: Choose one repetitive agent task. Which part belongs in a reviewed skill?
Guidance: Extract the stable method, not private session details. Version it, test failures, and require fresh authorization at execution time.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-procedural-memory-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-procedural-memory-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-procedural-memory-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-procedural-memory-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Memory & State (concept): https://www.openagentschool.org/concepts/memory-state?tab=types - Take this distinction into Memory & State and check how it changes a real system decision.
References:
LangChain - Agent memory: working state, semantic, episodic and procedural memory: https://docs.langchain.com/oss/python/concepts/memory

LESSON: Chain of Thought & Checkable Explanations
URL: https://www.openagentschool.org/ai-ml-foundations/chain-of-thought
Also called: CoT, chain-of-thought prompting, reasoning steps, reasoning summary
Summary: Understand intermediate reasoning without treating a convincing explanation as proof.
In everyday words: Chain of thought refers to intermediate reasoning steps generated or demonstrated while solving a task. The original prompting method used worked examples with intermediate steps. This can help some tasks, but a visible explanation is not guaranteed to faithfully expose a model’s internal computation or to be correct.
Why it matters: For a learner or agent operator, the useful output is checkable: assumptions, evidence, important calculations and a concise explanation. You do not need a private internal reasoning transcript to verify a result.
Scenario - A checkable explanation: Three passes cost 4 units each, with a single 2-unit fee. Message: "3 × 4 + 2 = 14 units."
Analogy - Showing the arithmetic on a bill: A short calculation lets another person check the total instead of accepting a confident number.
Where the analogy stops: A generated explanation can be a plausible story after an error. It is not an audit log of all internal model computation.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/chain-of-thought.webp
Picture description: Purchased containers, grouped counters showing intermediate work and an unmarked calculator beside a blank total slip.
Picture caption: Visible working can make an answer easier to check, without revealing or proving a model’s internal reasoning.
What you see: Given information. What it represents: The containers represent inputs that a short explanation should remain tied to.
What you see: Checkable steps. What it represents: The counters represent external intermediate work that can be independently inspected.
What you see: Verify the result. What it represents: The calculator represents checking the answer instead of trusting a plausible reasoning story.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A checkable explanation. Context: Three passes cost 4 units each, with a single 2-unit fee.
Text: 3 × 4 + 2 = 14 units.
Notice "3 × 4" - Line-item calculation: Three passes at four units each gives twelve.
Notice "+ 2" - One fixed fee: Apply the stated fee once.
Proposed next step: Verify the assumptions and arithmetic independently. This is an authored worked example, not hidden model reasoning.
Case: A longer but wrong explanation. Context: The same fee is charged once, not per pass.
Text: Each pass costs 4 + 2, so 3 × 6 = 18 units.
Notice "Each pass" - Wrong assumption: The fixed fee was incorrectly applied per pass.
Notice "18 units" - Incorrect answer: Correct arithmetic on the wrong setup still gives the wrong result.
Proposed next step: Repair the assumption; a longer explanation did not guarantee correctness.
Guided practice: Ask an assistant to compare two options. What useful explanation should you request?
Hint: Specify what another person can verify.
Compare your reasoning: Ask for the key criteria, assumptions, relevant sources, calculation if needed and a short rationale. Verify these rather than demanding a private reasoning trace.
Predict: Which part must you check before trusting the arithmetic?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Check how the task was represented, not just the final operation. Ask for concise justification and use calculators, source checks or tests where appropriate.
Optional math: task -> candidate solution -> concise evidence and calculation -> independent verification
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A detailed chain of thought proves why a model answered and that it is right.
Correction: Generated reasoning can be incorrect or unfaithful. Evaluate the outcome and independently check important claims; do not rely on access to private internal reasoning.
Check your understanding: Which evidence is stronger than a long confident explanation?
More reasoning text, regardless of verification. - Extra prose can repeat the same mistaken assumption. Check the result and its support.
A correct result supported by independent checks and applicable evidence. - Yes. Explanation length is not evidence quality.
Use it with purpose: What independent verifier could your agent use for a high-impact step?
Guidance: Use task-specific tools, tests or human review. Reasoning-model prompting practices vary; do not assume “think step by step” always improves them.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-chain-of-thought-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-chain-of-thought-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-chain-of-thought-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-chain-of-thought-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agentic Prompting Fundamentals (concept): https://www.openagentschool.org/concepts/agentic-prompting-fundamentals - Take this distinction into Agentic Prompting Fundamentals and check how it changes a real system decision.
References:
Wei et al. - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: https://arxiv.org/abs/2201.11903

LESSON: Retrieval-Augmented Generation (RAG)
URL: https://www.openagentschool.org/ai-ml-foundations/rag
Also called: RAG, retrieval augmented generation, retrieve then generate
Summary: Find relevant evidence before drafting an answer—and check what the evidence actually supports.
In everyday words: Retrieval-augmented generation supplies retrieved information to a generative model so it can answer with evidence beyond the immediate question. A typical system retrieves permitted sources, selects useful passages, drafts an answer and checks its support. RAG does not automatically retrain the model or guarantee factual accuracy.
Why it matters: An agent answering about a changing policy needs the right version and access rules. Retrieval quality and answer quality are separate places where a system can fail.
Scenario - Relevant current evidence: Question: “Can I return this item after 20 days?” The permitted current policy says 30 days, with conditions. Message: "Returns are accepted within 30 days if the item is unused."
Analogy - Answering with the relevant handbook open: Instead of answering from memory alone, you find the applicable passage and use it to explain the rule.
Where the analogy stops: Opening a handbook does not guarantee the right page, current edition or faithful reading. An agent can still invent details.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/rag.webp
Picture description: A small shelf of manuals, one open tabbed handbook and a separate blank response card on a shared table.
Picture caption: Find relevant material, supply it, then check whether the answer faithfully uses it.
What you see: Available collection. What it represents: The manuals represent sources that may contain relevant information.
What you see: Retrieved passage. What it represents: The tabbed page represents selected evidence, still needing relevance and freshness checks.
What you see: Generated answer. What it represents: The separate card represents an answer that must stay supported by the supplied source.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Relevant current evidence. Context: Question: “Can I return this item after 20 days?” The permitted current policy says 30 days, with conditions.
Text: Returns are accepted within 30 days if the item is unused.
Notice "30 days" - Time-window evidence: Twenty days fits this part of the rule.
Notice "if the item is unused" - Condition to preserve: The answer still needs to address item condition and applicability.
Proposed next step: Explain the window and condition; do not promise approval without the missing facts.
Case: Retrieved, but not sufficient. Context: The same question is paired with an old policy from another product category.
Text: Returns are accepted within 14 days.
Notice "14 days" - Wrong-context evidence: This passage is real but does not establish the current rule for the item.
Proposed next step: Find applicable evidence or say what is missing. Do not treat retrieval success as answer success.
Guided practice: The search returns nothing that answers the question. What should the agent do?
Hint: Absence of evidence is not permission to invent a policy.
Compare your reasoning: Explain the missing evidence, ask a focused question or route to an authoritative source. Do not manufacture an answer or citation.
Predict: Can a retrieved passage be genuine but still be the wrong evidence?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Yes. Check relevance, authority, version and permissions, then verify each answer claim. Keyword, vector and hybrid search can all be part of RAG.
Optional math: question -> permitted retrieval -> evidence selection -> generation -> support check
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Adding RAG eliminates hallucinations and always requires a vector database.
Correction: RAG can use several retrieval methods. It reduces some information gaps but can still retrieve the wrong evidence or generate unsupported claims.
Check your understanding: What should you measure separately in a RAG system?
Only whether the answer sounds natural. - Fluency cannot establish either evidence recall or faithful use.
Whether retrieval found the right evidence and whether the answer used it correctly. - Correct. Retrieval and generation have distinct failure modes.
Use it with purpose: Choose a question whose answer changes over time. What sources should your agent retrieve?
Guidance: Define authority, freshness, permissions, chunk boundaries and a no-evidence response. Test stale and contradictory sources too.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-rag-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-rag-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-rag-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-rag-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Retrieval-Augmented Generation (concept): https://www.openagentschool.org/concepts/rag-basics?tab=how-it-works - Take this distinction into Retrieval-Augmented Generation and check how it changes a real system decision.
References:
Lewis et al. - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401

LESSON: Vector Databases
URL: https://www.openagentschool.org/ai-ml-foundations/vector-databases
Also called: vector database, vector index, ANN, nearest neighbor search, metadata filtering
Summary: Store and search numerical representations without confusing similarity with truth or permission.
In everyday words: A vector database stores vectors, usually with record IDs and metadata, and supports similarity search over them. A vector index accelerates that search; it may be part of a dedicated system or a general database. Many indexes use approximate nearest-neighbor methods to trade some retrieval recall for speed.
Why it matters: An agent may need passages that use different words from the question. Similarity can find candidates, but access control, freshness, exact identifiers and source support still need explicit handling.
Scenario - A useful candidate: Question: “Are pushchairs allowed?” The passage is current and visible to this user. Message: "Strollers are permitted in the building."
Analogy - A library shelf arranged by resemblance: Nearby items may be useful neighbors even when their titles use different words. A catalogue still records which item each one is and who may access it.
Where the analogy stops: Vector dimensions are not neat human categories. Nearness does not prove agreement, truth, identity or access rights.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/vector-databases.webp
Picture description: Texture-grouped material samples, one selected sample with its own blank inventory tag and a separately latched compartment.
Picture caption: Nearness finds candidates; identity, truth and access still need their own checks.
What you see: Similarity neighbors. What it represents: Nearby samples represent candidates retrieved by similarity rather than exact wording.
What you see: Record and metadata. What it represents: The inventory sleeve represents the distinct source record behind a vector.
What you see: Access still applies. What it represents: The latch represents permission filters that similarity alone cannot satisfy.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A useful candidate. Context: Question: “Are pushchairs allowed?” The passage is current and visible to this user.
Text: Strollers are permitted in the building.
Notice "Strollers" - Different wording: May be close to “pushchairs” under a suitable embedding.
Notice "permitted" - Claim to read: The retrieved text supplies the actual rule; similarity alone does not.
Proposed next step: Retrieve the passage as a candidate, then verify its applicability and support.
Case: Similar but inaccessible. Context: A highly similar passage belongs to another customer’s private collection.
Text: Strollers are permitted in the private facility.
Notice "private facility" - Access boundary: Similarity does not authorize exposing another customer’s data.
Proposed next step: Enforce tenant and document permissions before exposing content to the model or user.
Guided practice: You switch embedding models. Can old and new vectors always be compared?
Hint: Equal dimensions do not guarantee a shared coordinate space.
Compare your reasoning: No. Use compatible document/query embeddings, reindex as needed, version the index and re-evaluate retrieval quality.
Predict: Should the most similar result always be shown?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: No. Filter by permissions and applicability, then inspect the actual evidence. Re-embedding or changing a distance metric also changes what “near” means.
Optional math: query -> compatible embedding -> authorized candidate search -> ranking -> evidence inspection
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A vector database understands and verifies the stored documents.
Correction: It searches representations under configured rules. Meaning, correctness, permissions and deletion need additional system design.
Check your understanding: What does a high similarity score establish?
The passage is true and may be shared. - Check source support and access separately from similarity.
Closeness under that representation and metric, not factual support or authorization. - Correct. Geometry is candidate-selection evidence, not a permission or truth verdict.
Use it with purpose: Before adding a vector database, what should you compare?
Guidance: Compare keyword, vector and hybrid retrieval on real query patterns. Measure evidence recall, latency, update/delete behavior and permission enforcement.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-vector-databases-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-vector-databases-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-vector-databases-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-vector-databases-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Retrieval Engineering (concept): https://www.openagentschool.org/concepts/retrieval-engineering - Take this distinction into Retrieval Engineering and check how it changes a real system decision.
References:
Qdrant - How vector search works: https://qdrant.tech/documentation/overview/vector-search/

LESSON: Machine Translation
URL: https://www.openagentschool.org/ai-ml-foundations/machine-translation
Also called: translation, MT, NMT, neural machine translation, localization
Summary: Carry meaning across languages without silently losing conditions, tone or units.
In everyday words: Machine translation automatically renders content from one language into another. The goal is to preserve the intended meaning for a purpose, not replace each word independently. Localization goes further by adapting conventions such as dates, units and interface wording for an audience.
Why it matters: A fluent translation can omit a negation, change an obligation or mishandle a name. For consequential use, review with people competent in the relevant languages and domain.
Scenario - Keep the condition: Spanish source: “No envíes el paquete hasta mañana.” This fictional instruction concerns sending a parcel. Message: "Do not send the parcel until tomorrow."
Analogy - Relaying instructions across a language boundary: The recipient needs the same practical meaning, even if the sentence structure changes.
Where the analogy stops: There may be no exact word-for-word equivalent. Cultural, legal and domain distinctions can require clarification rather than a neat substitution.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/machine-translation.webp
Picture description: Two differently folded blank instruction cards surround the same type of container and measuring cup.
Picture caption: The wording may change across languages; the important practical meaning and uncertainty must survive.
What you see: Source expression. What it represents: The first card represents the original wording in its context.
What you see: Meaning to preserve. What it represents: The shared objects represent quantities, conditions and intent that should remain intact.
What you see: Target expression. What it represents: The second card represents a new expression, not necessarily a word-for-word mapping.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Keep the condition. Context: Spanish source: “No envíes el paquete hasta mañana.” This fictional instruction concerns sending a parcel.
Text: Do not send the parcel until tomorrow.
Notice "Do not send" - Negation preserved: Sending now is prohibited.
Notice "until tomorrow" - Time boundary preserved: A real agent must still resolve the date and time zone.
Proposed next step: The prepared translation preserves the key prohibition and time boundary. It does not itself authorize a later tool call.
Case: Fluent meaning loss. Context: The source is still “No envíes el paquete hasta mañana.”
Text: Send the parcel today.
Notice "Send" - Negation lost: A prohibition became an instruction to act.
Notice "today" - Time reversed: The translation conflicts with the original boundary.
Proposed next step: Reject this translation even though the English is grammatical.
Guided practice: A translated message changes “may” to “must”. What should the reviewer check?
Hint: Permission and obligation are different.
Compare your reasoning: Compare the source’s intended force and domain context. Restore permission versus obligation accurately rather than choosing the stronger-sounding word.
Predict: Which parts need special attention before a translated instruction reaches an agent?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Verify negation, conditions, quantities, named entities and dates. Fluency and automatic overlap scores alone do not establish meaning preservation.
Optional math: source meaning + context -> target-language wording -> bilingual task-specific review
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A fluent translation preserves every important meaning.
Correction: Fluency can hide omissions and reversals. Even back-translation can repeat an error; use independent review for consequential content.
Check your understanding: Which translation check is most consequential for an action instruction?
Whether every source word has a same-position translated word. - Languages arrange meaning differently. Literal alignment can be misleading.
Whether the action, conditions and negations retain their meaning. - Yes. Preserve meaning and task boundaries, not word order.
Use it with purpose: Which content in your product needs bilingual human review?
Guidance: Prioritize instructions, consent, safety, legal obligations and financial amounts. Offer clarification and preserve the original text where appropriate.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-machine-translation-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-machine-translation-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-machine-translation-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-machine-translation-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
What Is a Language Model? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Take this distinction into What Is a Language Model? and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Natural Language Processing (NLP)
URL: https://www.openagentschool.org/ai-ml-foundations/nlp
Also called: NLP, computational linguistics, natural language
Summary: See the family of tools that helps software work with human language.
In everyday words: Natural language processing is the field of building systems that work with human language: reading, classifying, searching, translating, summarizing or generating it. Natural language means language people use, not a programming language. NLP is broader than chatbots and does not require a large language model.
Why it matters: A small, well-tested routing rule may be enough for one task. A generative model may help with another. Start with the person’s need and what a mistake would cost, not with the most powerful tool.
Scenario - Route a delivery message: A fictional help desk needs the right queue, not a written reply. Message: "My parcel has not arrived."
Analogy - A message desk with different jobs: One person sorts messages, another finds a record, another writes a reply. NLP covers these different kinds of language work.
Where the analogy stops: Software does not share a person’s lived experience. Useful language processing does not establish human understanding or empathy.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/nlp.webp
Picture description: Blank incoming message cards, three separated sorting trays and an open reply card with a pen on a modern service desk.
Picture caption: Language work can mean sorting, finding or replying; it is not one single task.
What you see: Incoming language. What it represents: The message cards represent human requests arriving in different forms.
What you see: Different tasks. What it represents: The trays represent separate jobs such as classification and extraction.
What you see: Possible reply. What it represents: The reply card represents generation, needed only when the workflow calls for it.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Route a delivery message. Context: A fictional help desk needs the right queue, not a written reply.
Text: My parcel has not arrived.
Notice "parcel" - Topic: The message concerns a delivery.
Notice "has not arrived" - Reported situation: An expected event has not happened; check delivery status.
Proposed next step: Route to delivery support. A classifier or rule could do this without generating prose.
Case: Write a supported reply. Context: The verified tracking record says delayed; no arrival date is available.
Text: My parcel has not arrived.
Notice "parcel" - Record needed: Find the correct tracking record with permission.
Notice "has not arrived" - Response need: Acknowledge the issue without inventing an arrival date.
Proposed next step: Draft a short reply explaining the known delay and the available next step.
Guided practice: A clinic needs appointment requests separated from feedback. What output is enough?
Hint: Does the next worker need a paragraph or a queue label?
Compare your reasoning: Start with a routing label and an uncertainty path. Do not generate medical advice just because the model can.
Predict: Can the same message require different kinds of NLP work?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: The input stayed the same; the goal changed from categorization to a supported response. Choose the output your workflow actually needs.
Optional math: message + task definition -> label, extracted fields, retrieved evidence or generated text
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: NLP is just another name for an LLM.
Correction: NLP includes rules, statistical models, neural models and whole applications. An LLM is one possible component.
Check your understanding: What should determine the choice of NLP system?
Always choose a chatbot. - A chatbot can add unnecessary generation and risk. Define the task first.
The needed output, users, error costs and evidence. - Yes. Fit the language tool to the decision and its risks.
Use it with purpose: Pick a language task you do weekly. What is the smallest useful output?
Guidance: Name the input, the person using the output, a useful result and one harmful error. Only then choose a technique.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-nlp-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-nlp-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-nlp-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-nlp-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
What Is a Language Model? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Take this distinction into What Is a Language Model? and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Natural Language Understanding (NLU)
URL: https://www.openagentschool.org/ai-ml-foundations/nlu
Also called: NLU, meaning representation
Summary: Turn wording into a tentative interpretation, while keeping uncertainty visible.
In everyday words: Natural language understanding is the task of interpreting what language means for a purpose: who is involved, what happened, what is requested and what remains unclear. In software, “understanding” describes a capability we evaluate; it is not evidence of human-like awareness.
Why it matters: Similar words can request opposite actions. An agent must distinguish checking whether a booking changed from being asked to change it.
Scenario - Ask for information: A traveler checks an existing reservation. Message: "Did you change my booking?"
Analogy - Repeating back an instruction: Before acting, a careful colleague says what they think you mean and asks about the missing parts.
Where the analogy stops: A model can produce a convincing paraphrase while misunderstanding the request. Test the interpretation against context and outcomes.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/nlu.webp
Picture description: A request card, two distinguishable appointment cards and an unsigned confirmation card laid out on a booking counter.
Picture caption: An interpretation is something to check before treating it as the person’s intended request.
What you see: What was said. What it represents: The first card stands for the actual request rather than an assumed action.
What you see: Possible meanings. What it represents: The two appointments represent interpretations that context must distinguish.
What you see: Confirm meaning. What it represents: The unsigned card represents checking an interpretation before acting on it.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Ask for information. Context: A traveler checks an existing reservation.
Text: Did you change my booking?
Notice "Did you" - Question: Asks whether an event happened.
Notice "change my booking" - Event mentioned: Mentioning a change is not authorizing one.
Proposed next step: Inspect permitted records and answer the status question; do not modify the booking.
Case: Request a change. Context: A traveler wants a different time but has not named it.
Text: Can you change my booking?
Notice "Can you" - Possible indirect request: Often asks for help, but capability and intention depend on context.
Notice "my booking" - Unresolved target: Identify the booking and desired change before execution.
Proposed next step: Ask which booking and what time; check permissions and confirmation requirements.
Guided practice: Interpret “I thought this was cancelled.” What needs checking?
Hint: Separate an expectation from a command.
Compare your reasoning: Check the current status and what “this” refers to. Do not cancel something solely from the statement.
Predict: Does recognizing the word “change” settle what the assistant should do?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: One message asks about history; the other may request an operation. Interpretation needs sentence form, context and a clarification path.
Optional math: wording + context -> candidate meaning + unresolved questions
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A fluent restatement proves the system understood.
Correction: Test important distinctions and ambiguous cases. Fluency alone cannot verify interpretation.
Check your understanding: “Did you cancel it?” should trigger what first?
Run the cancellation tool. - That changes the world instead of answering the question. Inspect first.
Resolve “it” and check whether cancellation happened. - Correct. It asks about an event, not permission for a new one.
Use it with purpose: What misunderstood phrase could cause harm in your application?
Guidance: Create contrasting examples with nearly identical wording and different intended actions. Review them with actual users.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-nlu-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-nlu-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-nlu-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-nlu-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Human-Agent Delegation (concept): https://www.openagentschool.org/concepts/human-agent-delegation - Take this distinction into Human-Agent Delegation and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Natural Language Generation (NLG)
URL: https://www.openagentschool.org/ai-ml-foundations/nlg
Also called: NLG, text generation, verbalization
Summary: Express information clearly without adding facts that were never supplied.
In everyday words: Natural language generation turns information, instructions or a representation into language people can read or hear. A template filling in a delivery date and a model composing a paragraph are both examples. Good generation preserves meaning while fitting the audience and purpose.
Why it matters: A polished sentence can quietly turn an estimate into a promise. Generation quality includes what was omitted or invented, not just how smooth the writing sounds.
Scenario - Preserve uncertainty: The record says “estimated arrival Friday; not guaranteed”. Message: "Your parcel is estimated to arrive Friday."
Analogy - Writing a notice from a verified record: You can make a notice shorter or warmer while keeping the same facts and uncertainty.
Where the analogy stops: Changing wording can change meaning. A friendly tone does not guarantee care, truth or a successful outcome.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/nlg.webp
Picture description: An open maintenance folder, two differently sized draft sheets and an empty notice holder on a shared-building desk.
Picture caption: A reply can change its wording and length without inventing facts missing from the record.
What you see: Source information. What it represents: The folder stands for facts and uncertainty that the wording must preserve.
What you see: Different wording. What it represents: The draft sheets represent alternative ways to express the same supported content.
What you see: Check before use. What it represents: The empty holder represents reviewing a generated notice before publication.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Preserve uncertainty. Context: The record says “estimated arrival Friday; not guaranteed”.
Text: Your parcel is estimated to arrive Friday.
Notice "estimated" - Uncertainty preserved: Signals that the date is not a guarantee.
Notice "Friday" - Supported date: Matches the supplied record.
Proposed next step: A concise draft preserves the important limitation.
Case: A fluent overpromise. Context: The same record still says “estimated arrival Friday; not guaranteed”.
Text: Your parcel will definitely arrive Friday.
Notice "definitely" - Unsupported certainty: Adds a guarantee absent from the record.
Notice "Friday" - Correct detail, wrong claim: A matching date does not rescue the overpromise.
Proposed next step: Revise the certainty before showing the message.
Guided practice: Write a short update from “request received; review pending; no decision date”.
Hint: Do not fill in the missing date.
Compare your reasoning: “We received your request. It is awaiting review; a decision date is not available yet.”
Predict: Can a reply keep the right date but still misrepresent the record?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Generation must preserve qualifications, not just named facts. “Estimated” and “definitely” create different expectations.
Optional math: supported information + audience + format -> wording -> meaning check
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Better writing means adding more detail.
Correction: Useful detail must be supported and needed. Extra prose can overwhelm a person or invent a commitment.
Check your understanding: Which revision makes an uncertain update safer?
Remove uncertainty so the reply sounds confident. - Confidence is not a substitute for support. Preserve the qualification.
Keep the estimate and make the next step clear. - Yes. Clarity can coexist with honest uncertainty.
Use it with purpose: Which facts and qualifications must survive a summary in your work?
Guidance: Write a checklist before generating: dates, units, exceptions, uncertainty and commitments. Compare the draft with its source.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-nlg-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-nlg-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-nlg-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-nlg-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Hallucination & Grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Take this distinction into Hallucination & Grounding and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Part-of-Speech Tagging
URL: https://www.openagentschool.org/ai-ml-foundations/pos-tagging
Also called: POS, POS tagging, noun, verb, morphology, lemma
Summary: Notice the job a word is doing in this sentence, not just its dictionary entry.
In everyday words: Part-of-speech tagging assigns grammatical roles such as noun, verb or adjective to words in context. A noun can name a thing; a verb can describe an action or state. The same written word may take different roles. Tag sets and grammatical categories vary across languages and tools.
Why it matters: Tagging can support extraction, search and linguistic analysis. A word list alone misses how a word is being used. Lemmatization is related but different: it maps inflected forms such as “running” to a base form such as “run”.
Scenario - Book as an action: A person requests a room reservation. Message: "Book a room."
Analogy - The same person wearing different work badges: A person may act as organizer in one meeting and participant in another. A word’s role also depends on its surroundings.
Where the analogy stops: Grammar is richer than a fixed badge, and a predicted tag can be wrong. Not every language organizes words like English.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/pos-tagging.webp
Picture description: A single folded workshop apron between two differently shaped blank clip-on badges on a clean community table.
Picture caption: The same word can take a different grammatical role when its surroundings change.
What you see: One role. What it represents: The round badge represents a word functioning as a noun in one sentence.
What you see: Same word. What it represents: The single apron represents the same spelling rather than a new word.
What you see: Another role. What it represents: The other badge represents that word functioning as a verb in another context.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Book as an action. Context: A person requests a room reservation.
Text: Book a room.
Notice "Book" - Verb: Here it requests an action.
Notice "room" - Noun: Names what should be reserved.
Proposed next step: The word “book” is used as a verb in this prepared annotation.
Case: Book as a thing. Context: A person describes an object on a desk.
Text: The book is open.
Notice "book" - Noun: Names an object.
Notice "open" - Adjective here: Describes the state of the book in this reading.
Proposed next step: The same spelling “book” now names a thing, not an instruction.
Guided practice: Compare “Send the report” with “Report the error”. What is “report” doing?
Hint: Which use names a thing and which requests an action?
Compare your reasoning: The first is a noun naming the item to send. The second is a verb requesting an action.
Predict: Will “book” have the same tag in both sentences?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Context changes the grammatical role. These are hand-authored annotations, not a tagger being run on your text.
Optional math: tokens + sentence context -> predicted grammatical tags
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Each word has one permanent part of speech.
Correction: Many words change role in context. A tag is not a complete meaning or a user intention.
Check your understanding: Can a noun/verb tag authorize a tool call?
Yes, any verb should become an action. - A verb may appear in a question, warning or quotation. Check intent and authority separately.
No. Grammar alone does not establish intent or permission. - Correct. Tags describe language; authority comes from the workflow.
Use it with purpose: Would tags help you extract product descriptions?
Guidance: Try a small labeled sample, including unusual names and different languages. Compare the errors with a simpler baseline.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-pos-tagging-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-pos-tagging-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-pos-tagging-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-pos-tagging-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
What Is a Language Model? (concept): https://www.openagentschool.org/concepts/what-is-an-llm - Take this distinction into What Is a Language Model? and check how it changes a real system decision.
References:
spaCy - Linguistic features: tags, dependencies and entities: https://spacy.io/usage/linguistic-features

LESSON: Parsing: How Words Connect
URL: https://www.openagentschool.org/ai-ml-foundations/parsing
Also called: syntax, dependency parsing, constituency parsing, subject, object
Summary: Find who did what to whom, and see why word order changes meaning.
In everyday words: Parsing builds a representation of sentence structure. A dependency parse connects words by grammatical relationships; a constituency parse groups words into phrases. These structures help identify subjects, objects and modifiers, but grammar alone does not settle truth or every intended meaning.
Why it matters: A system can recognize every word yet swap who performed an action and who received it. That matters when extracting approvals, responsibilities or events.
Scenario - One direction: Two people are involved in a support exchange. Message: "Mina helped Leo."
Analogy - Arrows on a handoff note: A handoff note names both the sender and receiver. A parse similarly makes relationships visible rather than leaving a bag of words.
Where the analogy stops: Grammatical relationships are not always real-world causes. Passive voice, omitted actors and ambiguous modifiers need care.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/parsing.webp
Picture description: One sealed parcel rests between a dispatch tray and a distinct receiving tray on a modern pickup counter.
Picture caption: Recognizing the objects is not enough; the relationship tells us who did what to whom.
What you see: Actor. What it represents: The dispatch side represents the participant performing the described action.
What you see: Action relationship. What it represents: The parcel handoff represents the relationship that connects sentence parts.
What you see: Recipient. What it represents: The receiving side represents who is affected, not simply the next name in the sentence.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: One direction. Context: Two people are involved in a support exchange.
Text: Mina helped Leo.
Notice "Mina" - Subject / helper: In this active sentence, Mina performs the helping.
Notice "helped" - Main verb: Connects the participants through an action.
Notice "Leo" - Object / recipient: Leo receives help.
Proposed next step: Record helper=Mina, recipient=Leo.
Case: Roles reversed. Context: The same words occur, but the order changes.
Text: Leo helped Mina.
Notice "Leo" - Subject / helper: Leo now performs the helping.
Notice "helped" - Main verb: The action word is unchanged.
Notice "Mina" - Object / recipient: Mina now receives help.
Proposed next step: Record helper=Leo, recipient=Mina. Word overlap alone misses this reversal.
Guided practice: In “Leo was helped by Mina”, who helped whom?
Hint: The grammatical subject is not always the actor.
Compare your reasoning: Mina helped Leo. Passive voice puts the recipient first without changing the real-world roles.
Predict: Would a bag of the words distinguish these events?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Relationships, not just word presence, decide the roles. A parser predicts structure; validate role extraction on the sentence forms you expect.
Optional math: subject -> verb -> object (one active-sentence pattern, not a universal language rule)
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Knowing all the keywords means knowing the sentence.
Correction: Order, attachment, negation and grammatical relationships can change the meaning without changing the keywords.
Check your understanding: What should an extractor preserve?
Just the names, sorted alphabetically. - A name list loses who did what. Preserve relationships as well.
Participant roles and the evidence for their relationship. - Yes. The roles distinguish different events.
Use it with purpose: Your workflow extracts who approved a request. Which sentence forms need testing?
Guidance: Include active and passive voice, quoted approvals, negated approvals and omitted actors. Unknown should remain unknown.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-parsing-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-parsing-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-parsing-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-parsing-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
AI-Ready Data (concept): https://www.openagentschool.org/concepts/ai-ready-data - Take this distinction into AI-Ready Data and check how it changes a real system decision.
References:
spaCy - Linguistic features: tags, dependencies and entities: https://spacy.io/usage/linguistic-features

LESSON: Named Entities & Recognition
URL: https://www.openagentschool.org/ai-ml-foundations/named-entities
Also called: NER, named entity recognition, person, organization, location, entity linking
Summary: Mark a name and its type without pretending you know which real-world record it is.
In everyday words: A named entity is a mention of a particular person, organization, place or other item in text. Named entity recognition (NER) locates mentions and assigns types under a chosen annotation scheme. Some schemes also include dates and quantities. Entity linking is a separate task: connecting a mention to a specific record.
Why it matters: Recognizing “Jordan” does not tell you whether it names a person, a country, or which person in a directory. Mistaken identity can send information to the wrong place.
Scenario - A person mention: A message refers to a colleague. Message: "Jordan will join the meeting."
Analogy - Highlighting names before opening a directory: First circle a mention and ask what kind of thing it names. Only then look for the matching entry.
Where the analogy stops: A highlight is not identity proof. Names are shared, context can be sparse, and categories are task-dependent.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/named-entities.webp
Picture description: One tabbed blank name badge sits separately from two similar candidate badges and a small inspection lens.
Picture caption: Finding a name mention does not tell you which real-world record it belongs to.
What you see: Recognized mention. What it represents: The tabbed badge represents locating something that appears to be a name.
What you see: Candidate identities. What it represents: The two badges represent different records that can share the same name.
What you see: Resolve separately. What it represents: The lens represents checking permitted evidence before choosing a unique record.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A person mention. Context: A message refers to a colleague.
Text: Jordan will join the meeting.
Notice "Jordan" - Person in this reading: The context suggests a person; it does not identify a unique account.
Proposed next step: Mark a person mention. Use additional verified details before linking a directory record.
Case: A place mention. Context: A travel note describes a journey.
Text: We travelled to Jordan.
Notice "Jordan" - Country in this reading: Travel context supports a place interpretation.
Proposed next step: Mark a location mention, not the colleague’s account.
Guided practice: A message says “Alex approved it”, and your directory has three Alexes. What can you record?
Hint: Separate observed text from resolved identity.
Compare your reasoning: Record the Alex mention and approval claim with its source. Do not assign a unique account until it is resolved.
Predict: Can identical spelling refer to different kinds of entities?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Mention detection, type assignment and record linking are separate decisions. Preserve the text span and uncertainty.
Optional math: text -> mention span + type; mention + evidence -> candidate record
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: NER identifies the exact person behind a name.
Correction: NER usually identifies a span and category. Disambiguation and verified identity require more evidence.
Check your understanding: Two records share a recognized name. What next?
Choose the first search result. - Search order is not identity evidence. Keep ambiguity visible.
Resolve the identity using permitted context or ask. - Correct. A recognized name is not enough to select a unique record.
Use it with purpose: Which entity types would help in your documents?
Guidance: Define types and span boundaries before labeling. Include unfamiliar names and privacy-sensitive cases in review.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-named-entities-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-named-entities-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-named-entities-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-named-entities-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
AI-Ready Data (concept): https://www.openagentschool.org/concepts/ai-ready-data - Take this distinction into AI-Ready Data and check how it changes a real system decision.
References:
spaCy - Linguistic features: tags, dependencies and entities: https://spacy.io/usage/linguistic-features

LESSON: Entity & Field Extraction
URL: https://www.openagentschool.org/ai-ml-foundations/entity-extraction
Also called: information extraction, slot filling, relation extraction, structured extraction
Summary: Turn a message into useful fields while preserving what is missing or uncertain.
In everyday words: Entity and field extraction turns text into structured information such as a destination, item, quantity or relationship. It can include named entities, ordinary noun phrases and task-specific slots. Extracting “tomorrow” is different from resolving it to a date, and extracting a request is different from executing it.
Why it matters: A valid-looking record can hide a guessed date or missing unit. Keep the original span, normalized value, source and unresolved fields distinguishable.
Scenario - A complete request: A fictional room-booking form expects room, date and time in the office time zone. Message: "Book room B on 12 May at 14:00."
Analogy - Filling a form from someone’s message: Copy what was actually said into the right fields, then leave a visible question where information is missing.
Where the analogy stops: Real messages do not always match a form. The form itself can impose the wrong assumptions.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/entity-extraction.webp
Picture description: A blank message sheet, a three-compartment organizer with one empty space, and an unfinished shipping card.
Picture caption: Extract what the message supplies; leave missing fields visible rather than filling them with guesses.
What you see: Source message. What it represents: The original sheet represents the wording from which fields must be supported.
What you see: Known and missing. What it represents: The partly filled organizer represents extracted values and unresolved fields.
What you see: Not ready to submit. What it represents: The unfinished card represents waiting for required details and authorization.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A complete request. Context: A fictional room-booking form expects room, date and time in the office time zone.
Text: Book room B on 12 May at 14:00.
Notice "room B" - Room field: Retain the room identifier.
Notice "12 May" - Date mention: The year still needs the conversation’s explicit context.
Notice "14:00" - Time field: Interpret under the stated office time zone.
Proposed next step: Extract the fields; resolve the year and check availability before any booking.
Case: Missing details. Context: No earlier booking or location is available.
Text: Book a room tomorrow.
Notice "a room" - Unspecified room: Do not invent a room identifier.
Notice "tomorrow" - Relative date: Requires a reference date and relevant time zone.
Proposed next step: Ask for the missing time/location and resolve the date. Extraction alone is not a completed booking.
Guided practice: Extract “Send three boxes to the usual address”. What remains unresolved?
Hint: A quantity can be known while the destination is not.
Compare your reasoning: Quantity=3 and item=boxes are present. “Usual address” needs an authorized, current reference; do not guess it.
Predict: Which fields should remain unresolved instead of receiving a plausible default?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: A schema can require a field, but it cannot supply missing evidence. Use explicit unknowns or clarification rather than fabricated values.
Optional math: span -> field -> normalization -> validation -> clarification if needed
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Well-formed JSON means the extracted facts are correct.
Correction: Schema validity checks shape and types. Evidence, identity, units and meaning still need validation.
Check your understanding: An extraction omits the currency for “500”. What should happen?
Assume the application’s most common currency. - A silent default can change the amount’s meaning. Make the assumption explicit or ask.
Mark the unit unresolved and clarify or find explicit evidence. - Yes. A common value is not necessarily the user’s value.
Use it with purpose: Design three fields for a workflow you know. Which need source spans?
Guidance: Keep provenance for consequential fields, validate units and identities, and define what happens when information conflicts.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-entity-extraction-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-entity-extraction-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-entity-extraction-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-entity-extraction-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Tool Use & Function Calling (concept): https://www.openagentschool.org/concepts/tool-use-function-calling - Take this distinction into Tool Use & Function Calling and check how it changes a real system decision.
References:
spaCy - Linguistic features: tags, dependencies and entities: https://spacy.io/usage/linguistic-features

LESSON: Ambiguity & Word Meaning
URL: https://www.openagentschool.org/ai-ml-foundations/language-ambiguity
Also called: polysemy, word sense disambiguation, lexical ambiguity, semantic ambiguity
Summary: Recognize when more than one interpretation fits, and ask the useful question.
In everyday words: Ambiguity means a word, phrase or sentence supports more than one interpretation. Word-sense disambiguation selects a meaning using context. Sometimes the evidence does not justify selecting one; a good system can preserve alternatives and ask a focused question.
Why it matters: Guessing may be harmless in a casual search and costly when changing a reservation or moving money. The cost of the wrong interpretation matters as much as its likelihood.
Scenario - A financial clue: The conversation concerns a transfer. Message: "Meet me at the bank to discuss the transfer."
Analogy - A sign with two possible destinations: A nearby clue may tell you which direction is intended. Without it, asking beats confidently walking the wrong way.
Where the analogy stops: Meanings are not always a small fixed menu. Communities and contexts create new uses.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/language-ambiguity.webp
Picture description: A blank two-part direction plate beside two equally plausible meeting-room doorways in a modern corridor.
Picture caption: When more than one meaning fits, a focused clarification can matter more than a confident guess.
What you see: Missing context. What it represents: The incomplete direction plate represents a request that does not identify one meaning.
What you see: Plausible meaning A. What it represents: One doorway represents a possible interpretation, not a proven destination.
What you see: Plausible meaning B. What it represents: The other doorway represents an alternative that may require clarification.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A financial clue. Context: The conversation concerns a transfer.
Text: Meet me at the bank to discuss the transfer.
Notice "bank" - Likely financial institution: The transfer context supports this reading, but the location still needs confirmation.
Proposed next step: Use the financial reading for a draft; clarify the actual meeting location before navigation.
Case: A landscape clue. Context: The conversation concerns a river walk.
Text: Meet me at the bank beside the river.
Notice "bank" - Land beside water: The river phrase supports a different sense of the same word.
Proposed next step: Do not search only for financial institutions.
Guided practice: Someone says “Move it to the next one.” What is your first question?
Hint: Both the object and destination are vague.
Compare your reasoning: Ask what should move and which destination they mean. Use existing context if it already answers one part.
Predict: What evidence changes the likely meaning of “bank”?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Nearby words and the broader situation constrain meaning. Context can reduce ambiguity without eliminating all uncertainty.
Optional math: possible meanings + relevant context -> supported interpretation or clarification
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: The most frequent meaning is always the intended one.
Correction: Frequency is a starting guess, not evidence about this person’s particular situation.
Check your understanding: A consequential request has two plausible meanings. What is safest?
Pick one and hide the uncertainty. - Hidden uncertainty becomes hidden risk. Clarify the decision-changing detail.
Ask a short question that distinguishes the meanings. - Yes. A targeted clarification prevents a costly guess without overwhelming the person.
Use it with purpose: Where should your assistant ask instead of guessing?
Guidance: List ambiguous phrases and the consequences of a wrong guess. Use tighter clarification rules for irreversible actions.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-language-ambiguity-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-language-ambiguity-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-language-ambiguity-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-language-ambiguity-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Take this distinction into Context Engineering and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Coreference: Who Does “They” Mean?
URL: https://www.openagentschool.org/ai-ml-foundations/coreference
Also called: anaphora, pronoun resolution, reference resolution
Summary: Connect repeated mentions without guessing who owns an action or decision.
In everyday words: Coreference is the relationship between expressions that refer to the same entity. Resolving it connects a pronoun or phrase, such as “they” or “the customer”, to the intended referent. Some sentences remain genuinely ambiguous.
Why it matters: A summary or agent can assign consent, responsibility or a request to the wrong person if it resolves a pronoun incorrectly.
Scenario - Clearer reference: There is one explicitly named requester. Message: "Ari requested a refund. Ari wants an update."
Analogy - Following name tags through a conversation: Different expressions can point to the same participant, like a name tag and “the organizer”.
Where the analogy stops: A conversation may not contain enough evidence to connect them. Do not infer identity or gender from stereotypes.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/coreference.webp
Picture description: Two distinct participant badges sit by empty meeting chairs, with an unattached follow-up note between them.
Picture caption: A reference like “they” needs evidence; the closest name is not automatically the right one.
What you see: One participant. What it represents: The first badge represents one possible referent in the conversation.
What you see: Unresolved reference. What it represents: The loose note represents an expression whose referent is not yet established.
What you see: Another participant. What it represents: The second badge represents an alternative; identity must not be guessed from stereotypes.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Clearer reference. Context: There is one explicitly named requester.
Text: Ari requested a refund. Ari wants an update.
Notice "Ari" - Repeated name: Both mentions are intended to identify the requester in this example.
Proposed next step: Connect both mentions to the same fictional requester.
Case: Unclear reference. Context: Ari and Sam are both participants; nothing else identifies the requester.
Text: Ari told Sam they needed a refund.
Notice "Ari" - Candidate referent: Could be the person needing the refund.
Notice "Sam" - Another candidate: Could also be the intended referent.
Notice "they" - Unresolved pronoun: The sentence alone does not settle who needs the refund.
Proposed next step: Ask who needs the refund before modifying an account.
Guided practice: “The package reached the locker, but it was damaged.” What could “it” mean?
Hint: Both named things can be damaged.
Compare your reasoning: The package or the locker. Ask or inspect explicit evidence before filing the damage report.
Predict: Does the nearest name always tell us who “they” refers to?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Proximity is a heuristic, not proof. Preserve unresolved references rather than silently assigning an action to someone.
Optional math: mentions + discourse context -> referent links or unresolved alternatives
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A pronoun always refers to the nearest noun.
Correction: Grammar, context and discourse all matter; sometimes no unique answer is available.
Check your understanding: How should an ambiguous pronoun appear in an extracted record?
As the nearest person’s account ID. - That converts a grammatical guess into an identity decision.
As unresolved, with a clarification or review step. - Correct. The extraction must not claim more than the text supports.
Use it with purpose: Review a summary containing “it”, “they” or “this”. Can another reader identify each referent?
Guidance: Replace ambiguity with a verified name or explicit uncertainty. Do not remove important uncertainty merely to make prose smoother.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-coreference-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-coreference-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-coreference-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-coreference-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
AI-Ready Data (concept): https://www.openagentschool.org/concepts/ai-ready-data - Take this distinction into AI-Ready Data and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Negation, Conditions & Scope
URL: https://www.openagentschool.org/ai-ml-foundations/negation-and-scope
Also called: negation, scope, modality, conditional instructions
Summary: Keep “not”, “only” and “unless” attached to the right action.
In everyday words: Negation changes whether something is asserted; scope tells us which part of the statement it affects. Conditions say when an action may happen. Small words such as “not”, “only” and “unless” can change an instruction’s entire effect.
Why it matters: Removing small words during preprocessing can erase a safety boundary. Mentioning an action does not mean it was requested, allowed or completed.
Scenario - Permission withheld: A person asks for a preview only. Message: "Draft the email, but do not send it."
Analogy - The condition printed beside a switch: “Turn on after inspection” is not the same instruction as “turn on”. The condition belongs with the action.
Where the analogy stops: Natural-language conditions still need explicit runtime enforcement; a prompt is not a lock.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/negation-and-scope.webp
Picture description: A protected draft, an unfinished inspection checklist and an unplugged desktop shredder arranged without any action underway.
Picture caption: “Only after checking” is part of the instruction, not an optional phrase to discard.
What you see: Object of the action. What it represents: The protected draft represents the specific material governed by the instruction.
What you see: Required condition. What it represents: The unfinished checklist represents verification that must actually occur first.
What you see: Action stays blocked. What it represents: The disconnected device represents an enforced boundary, not merely careful wording.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Permission withheld. Context: A person asks for a preview only.
Text: Draft the email, but do not send it.
Notice "Draft" - Allowed preparation: Creating a preview is requested.
Notice "do not send" - Prohibition: Sending is explicitly outside the request.
Proposed next step: Produce a draft and stop before sending.
Case: Conditional permission. Context: Approval must come from the person after reviewing the draft.
Text: Send the email only after I approve the draft.
Notice "only after" - Condition: Sending depends on a later approval event.
Notice "I approve" - Named authority: The agent cannot approve on the person’s behalf.
Proposed next step: Wait for recorded approval; then recheck recipient and final content before sending.
Guided practice: Interpret “Do not cancel the replacement, only the original order.”
Hint: Track the two targets separately.
Compare your reasoning: The original may be cancelled under the workflow; the replacement must not be cancelled. Resolve both IDs first.
Predict: Why is finding the keyword “send” insufficient?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: The first instruction forbids sending; the second makes it conditional. Preserve scope and check the condition in the system.
Optional math: action + negation + condition + authority -> permitted operation set
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Small function words can always be removed as noise.
Correction: Words such as “not” and “unless” carry essential meaning, especially in rules and requests.
Check your understanding: Can the assistant send because it predicts the person will approve?
Yes, if its confidence is high. - A prediction cannot substitute for consent or an enforced permission check.
No. Predicted approval is not actual authorization. - Correct. The required event is a person’s approval, not a model score.
Use it with purpose: Which conditions in your workflow must be checked outside the prompt?
Guidance: Put approval, identity, spending and safety constraints into application checks. Keep the original instruction as evidence.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-negation-and-scope-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-negation-and-scope-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-negation-and-scope-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-negation-and-scope-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Human-Agent Delegation (concept): https://www.openagentschool.org/concepts/human-agent-delegation - Take this distinction into Human-Agent Delegation and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Language, Dialect & Context
URL: https://www.openagentschool.org/ai-ml-foundations/multilingual-language
Also called: multilingual NLP, code switching, dialect, pragmatics, language variation
Summary: Design for varied ways of speaking instead of treating one language pattern as universal.
In everyday words: People express meaning through different languages, dialects, scripts, registers and mixtures of languages. Pragmatics studies how a situation and social context help convey meaning beyond literal wording. A system should not treat unfamiliar wording as lesser intelligence or assume every person uses the same conventions.
Why it matters: Word boundaries, politeness, dates and indirect requests vary. Aggregate accuracy can hide worse outcomes for particular language communities.
Scenario - Mixed-language request: A fictional bilingual message asks about an invoice. The author has not requested a language switch. Message: "Please send la factura."
Analogy - Listening before completing someone’s sentence: A considerate listener checks what the person means rather than forcing their words into a familiar pattern.
Where the analogy stops: An AI does not have lived cultural understanding. Language and location do not reliably reveal a person’s preferences or identity.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/multilingual-language.webp
Picture description: Two different personal notebooks surround a shared blank planning sheet and unused pen at a community table.
Picture caption: Check the person’s meaning and preferences rather than filling them in from cultural assumptions.
What you see: One expression. What it represents: One notebook represents a person’s wording, dialect or language choice.
What you see: Clarify together. What it represents: The shared sheet represents confirming the decision-relevant meaning.
What you see: Another expression. What it represents: The second notebook represents variation without ranking one form as inherently correct.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Mixed-language request. Context: A fictional bilingual message asks about an invoice. The author has not requested a language switch.
Text: Please send la factura.
Notice "Please send" - Requested action: The request is understandable across the mixed wording.
Notice "la factura" - Invoice mention: Spanish wording for the invoice; this does not identify which invoice.
Proposed next step: Clarify the invoice if needed. Do not infer the person’s nationality or proficiency.
Case: A locale-dependent date. Context: The sender’s date convention and time zone are unknown.
Text: Schedule it for 03/04.
Notice "03/04" - Ambiguous date: Could mean 3 April or March 4; the year and time are also missing.
Proposed next step: Ask for a date with the month written out and the relevant time zone before scheduling.
Guided practice: A system performs well overall but misses requests in a local dialect. What should change?
Hint: An average can hide a group’s experience.
Compare your reasoning: Review consented examples with relevant speakers, measure errors by language variety and improve the task policy or model. Do not blame the users.
Predict: Which details cannot safely be inferred from the language of a message?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Language is evidence about the wording, not a reliable identity profile. Ask for decision-relevant preferences and test varied language use.
Optional math: language variation + local convention + task context -> interpretation with explicit unknowns
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Translate everything to English and all meaning is preserved.
Correction: Translation can lose ambiguity, social meaning, terminology or important distinctions. Evaluate the end task with the communities using it.
Check your understanding: How should the assistant handle an ambiguous numeric date?
Infer the date convention from the person’s name. - A name does not establish a date convention. Ask about the date itself.
Ask a concise clarification using an explicit month name. - Yes. Clarify the convention without stereotyping identity.
Use it with purpose: Whose language is absent from your evaluation examples?
Guidance: Include relevant scripts, dialects, accessibility needs and mixed-language messages. Obtain permission and protect identifying information.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-multilingual-language-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-multilingual-language-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-multilingual-language-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-multilingual-language-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
AI-Ready Data (concept): https://www.openagentschool.org/concepts/ai-ready-data - Take this distinction into AI-Ready Data and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Intent Detection
URL: https://www.openagentschool.org/ai-ml-foundations/intent-detection
Also called: intent classification, user intent, out-of-scope intent
Summary: Identify the likely purpose of a message without turning a guess into an action.
In everyday words: Intent detection identifies the kind of outcome a person appears to want, such as checking a delivery, changing a date or reporting a problem. Intent labels belong to an application’s task design; they are not universal categories. One message may contain several intents or none of the supported ones.
Why it matters: Detecting a cancellation topic is not permission to cancel. A useful detector also recognizes uncertainty and requests outside its supported scope.
Scenario - Ask about an option: No cancellation has been requested yet. Message: "What happens if I cancel my ticket?"
Analogy - Choosing the right help desk queue: A receptionist routes you to a team based on your purpose, without performing the team’s consequential action.
Where the analogy stops: A label compresses a richer human need and may be mistaken. Routing must allow correction.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/intent-detection.webp
Picture description: A request card waits in front of two differently shaped routing trays at a contemporary reception counter.
Picture caption: Choosing the likely help queue is not the same as carrying out a consequential action.
What you see: Expressed purpose. What it represents: The card represents the person’s request before a category is assigned.
What you see: Information route. What it represents: One tray represents a question that needs an answer rather than an action.
What you see: Action-request route. What it represents: The other tray represents a possible operation that still needs resolution and checks.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Ask about an option. Context: No cancellation has been requested yet.
Text: What happens if I cancel my ticket?
Notice "What happens if" - Information-seeking intent: The person asks about consequences.
Notice "cancel" - Topic, not authorization: The action is discussed hypothetically.
Proposed next step: Explain the applicable cancellation policy; do not cancel.
Case: Request an operation. Context: The ticket identity still needs verification.
Text: Please cancel my ticket.
Notice "Please cancel" - Cancellation request: A candidate operation intent.
Notice "my ticket" - Target needed: A label does not identify the ticket or prove authority.
Proposed next step: Route to the cancellation workflow, which must resolve the target and enforce permissions.
Guided practice: Classify “Where is my order, and can I change the address?”
Hint: There may be more than one purpose.
Compare your reasoning: It contains a status question and a possible address-change request. Handle each with the right context and checks.
Predict: Should both messages lead directly to the same cancellation tool?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: They share a topic but request different outcomes. Evaluate intent distinctions that change what the system does.
Optional math: message -> candidate intent(s) + uncertainty + out-of-scope path
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: An intent label is the user’s complete request.
Correction: Labels omit details, constraints and authority. Treat them as routing evidence, not a full execution plan.
Check your understanding: A message fits none of the supported intent labels. What is useful?
Force it into the closest label and act. - The closest label may still be wrong. Support an uncertainty or out-of-scope outcome.
Ask for clarification or route to an appropriate human. - Correct. An explicit unsupported path prevents silent misrouting.
Use it with purpose: Which two intents in your application are easy to confuse?
Guidance: Write minimal pairs that share vocabulary but need different outcomes. Measure those confusions separately.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-intent-detection-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-intent-detection-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-intent-detection-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-intent-detection-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Human-Agent Delegation (concept): https://www.openagentschool.org/concepts/human-agent-delegation - Take this distinction into Human-Agent Delegation and check how it changes a real system decision.
References:
Jurafsky & Martin - Speech and Language Processing (open textbook): https://web.stanford.edu/~jurafsky/slp3/

LESSON: Intent Resolution for Agents
URL: https://www.openagentschool.org/ai-ml-foundations/intent-resolution
Also called: agent intent, clarification, slot resolution, action authorization
Summary: Turn a tentative request into a specific, permitted next step—or a good question.
In everyday words: Here, intent resolution means turning a candidate interpretation into a specific next step: resolve the target, missing details, constraints and authority. The phrase is used differently across systems; this lesson treats it as a workflow responsibility, not one standard NLP algorithm.
Why it matters: “Move my appointment” is not executable safely until the system knows which appointment, which time, who may change it and whether confirmation is needed.
Scenario - Intent, but not a resolved action: There are two appointments and no destination time. Message: "Move my appointment."
Analogy - Confirming a booking at a service desk: A careful assistant finds the right booking, asks the missing question and confirms the change before committing it.
Where the analogy stops: An agent must enforce permissions through application controls; sounding careful is not enough.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/intent-resolution.webp
Picture description: A request slip, two possible booking cards and an unsigned confirmation beside a closed key drawer.
Picture caption: A usable intent needs the right target, missing details and current authority—not only a category.
What you see: Requested change. What it represents: The request slip represents the broad goal that intent detection identified.
What you see: Resolve the target. What it represents: The two booking cards represent ambiguity that must be settled before acting.
What you see: Confirm authority. What it represents: The closed drawer represents execution staying gated until the required checks pass.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Intent, but not a resolved action. Context: There are two appointments and no destination time.
Text: Move my appointment.
Notice "Move" - Candidate intent: Rescheduling appears to be wanted.
Notice "my appointment" - Ambiguous target: Two records match, and the new time is absent.
Proposed next step: Ask which appointment and the preferred new time. Do not call a mutation tool yet.
Case: Specific request, checks still needed. Context: The signed-in person owns appointment A17. A move requires availability and confirmation.
Text: Move appointment A17 to 10:00 on 5 June.
Notice "A17" - Resolved target candidate: Verify the account-to-record relationship.
Notice "10:00 on 5 June" - Requested slot: Resolve year and time zone, check availability and confirm the final change.
Proposed next step: Prepare a proposed change; execute only after the workflow’s required checks and confirmation.
Guided practice: “Refund the last purchase.” The account has two purchases on the same day. What should the agent do?
Hint: The sorting rule may not reflect the person’s intent.
Compare your reasoning: Present a minimal, privacy-safe choice between the purchases. Resolve the target and refund policy before requesting confirmation.
Predict: What remains to be checked even when the message contains an appointment ID?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: A specific request removes some ambiguity, not all risk. Identity, constraints, side effects and authority remain application concerns.
Optional math: detected intent -> target + slots + constraints -> permission check -> confirm, clarify or decline
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: High model confidence is enough to authorize an action.
Correction: Confidence estimates a prediction; authorization grants permission. Neither substitutes for the other.
Check your understanding: What proves that a resolved action may run?
A fluent plan with a high confidence score. - A plan is a proposal. It does not establish permission.
Explicit policy and authorization checks, plus any required confirmation. - Yes. Action permission must be enforced by the surrounding system.
Use it with purpose: Define a resolution checklist for one agent action you would allow.
Guidance: Include target, required fields, units/time zone, authority, confirmation, failure handling and a safe stop. Keep questions short and decision-relevant.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-intent-resolution-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-intent-resolution-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-intent-resolution-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-intent-resolution-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Tool Use & Function Calling (concept): https://www.openagentschool.org/concepts/tool-use-function-calling - Take this distinction into Tool Use & Function Calling and check how it changes a real system decision.
References:
Anthropic - Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

LESSON: Task Adherence for Agents
URL: https://www.openagentschool.org/ai-ml-foundations/task-adherence
Also called: instruction following, scope adherence, agent compliance, acceptance criteria
Summary: Check whether the agent did the requested work—and stopped at the agreed boundary.
In everyday words: Task adherence asks whether an agent followed the intended task and its constraints across both its response and its actions. It includes requested format, allowed tools, scope, stopping rules and things that must not happen. It is not the same as obeying every instruction found in untrusted text.
Why it matters: An answer may be accurate while the agent sends an unapproved email or changes an unrelated record. Review the action trace as well as the final message.
Scenario - Within the brief: The task is to draft two time options, without booking or emailing. Message: "Here are two proposed times. Nothing has been booked or sent."
Analogy - A repair brief with clear boundaries: Fixing the requested fault does not authorize replacing unrelated parts or sharing the customer’s details.
Where the analogy stops: Real briefs can be incomplete or conflicting. Clarify safely rather than assuming unlimited authority.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/task-adherence.webp
Picture description: A bicycle light and blank work slip sit beside the few needed tools, while a larger toolbox remains closed.
Picture caption: Completing the requested work does not authorize every other change you could make.
What you see: Requested task. What it represents: The single light represents the specific deliverable and its scope.
What you see: Relevant work. What it represents: The selected tools represent actions needed for that task under its constraints.
What you see: Outside the brief. What it represents: The closed toolbox represents other possible actions that were not authorized.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Within the brief. Context: The task is to draft two time options, without booking or emailing.
Text: Here are two proposed times. Nothing has been booked or sent.
Notice "two proposed times" - Requested deliverable: Matches the number and proposal-only scope.
Notice "Nothing has been booked or sent" - Claim to verify: The action log, not this sentence alone, must establish that no side effect occurred.
Proposed next step: Check the trace: if only permitted reads and drafting occurred, this fits the task.
Case: Helpful-looking overreach. Context: The task is still to draft two options without booking or emailing.
Text: I chose a time, booked it and emailed everyone.
Notice "chose a time" - Decision overreach: The person asked for options, not an autonomous choice.
Notice "booked it and emailed" - Unauthorized side effects: Both actions were outside the brief.
Proposed next step: Fail the adherence check even if the chosen time was available.
Guided practice: An agent was asked to summarize a file but also edits it to fix a typo. Does that adhere?
Hint: Was editing authorized?
Compare your reasoning: No, unless the task explicitly permitted editing. Offer the correction as a suggestion instead.
Predict: Can a useful-looking outcome still fail task adherence?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Yes. Outcome quality and scope compliance are separate. Validate forbidden actions, not just requested output fields.
Optional math: task contract + action trace + final output -> requirement-by-requirement review
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: If the final answer looks right, the task was followed.
Correction: Inspect tool calls, state changes, omissions and prohibited behavior. A final answer can conceal harmful actions.
Check your understanding: Which evidence best checks “do not send”?
The assistant saying “I did not send it”. - Self-report is not an independent record of side effects.
The execution trace and enforced tool permissions. - Correct. Check observable actions and the control that prevented sending.
Use it with purpose: Write one positive requirement and one forbidden action for a task.
Guidance: Make both observable. Specify how the runtime prevents the forbidden action and how evaluation detects a violation.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-task-adherence-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-task-adherence-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-task-adherence-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-task-adherence-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Take this distinction into Agent Evaluation and check how it changes a real system decision.
References:
Microsoft - AI evaluation metrics and their distinct purposes: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-metrics-built-in

LESSON: Semantic Engineering
URL: https://www.openagentschool.org/ai-ml-foundations/semantic-engineering
Also called: semantics, shared meaning, ontology, controlled vocabulary, semantic contracts
Summary: Give important words shared definitions before connecting data, tools and agents.
In everyday words: Semantic engineering is used here as a design practice: make meaning explicit across people, data and software through definitions, relationships, units, identifiers and rules. It is not one universally standardized NLP technique. An ontology describes concepts and relationships; a vocabulary gives agreed labels and definitions.
Why it matters: Two tools can both return “active customer” while counting different people. Matching field names or embeddings will not repair a disagreement in business meaning.
Scenario - Same label, different rules: Sales means a purchase in 90 days; support means an open account. Message: "Show active customers."
Analogy - Agreeing on the legend before reading a shared map: The legend explains what a symbol means so different readers do not act on incompatible interpretations.
Where the analogy stops: Definitions are human decisions that can change. A vocabulary must be governed and reviewed, not treated as timeless truth.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/semantic-engineering.webp
Picture description: Two simple room-outline plans share a small legend tray containing matching circular, square and triangular pieces.
Picture caption: Shared labels help only when people agree what they mean and keep the agreement current.
What you see: One representation. What it represents: The first plan represents one system describing the world.
What you see: Shared definitions. What it represents: The shape legend represents explicitly agreed meanings and relationships.
What you see: Consistent reuse. What it represents: The second plan represents another system using the same governed definitions.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Same label, different rules. Context: Sales means a purchase in 90 days; support means an open account.
Text: Show active customers.
Notice "active customers" - Undefined business meaning: The two teams’ criteria produce different lists.
Proposed next step: Ask which definition applies. Do not merge the lists as if they measured the same concept.
Case: An explicit definition. Context: The report owner agrees on a purchase-based rule and a reporting date.
Text: Show customers with a completed purchase in the previous 90 days.
Notice "completed purchase" - Event definition: Specify whether refunds and cancellations count.
Notice "previous 90 days" - Time window: Anchor the interval to an explicit reporting date and time zone.
Proposed next step: Version the rule and map tool fields to it before comparing results.
Guided practice: Two systems report “revenue”. What must you ask before adding the values?
Hint: Think timing, units and accounting definition.
Compare your reasoning: Check currency, reporting period, gross versus net, recognition rules and overlap. Document the mapping and its owner.
Predict: Will identical field names guarantee compatible data?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Shared meaning needs definitions and tests. A semantic contract records assumptions so an agent does not quietly bridge incompatible concepts.
Optional math: term -> definition + relationships + units + scope + version + owner
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: An embedding automatically supplies a trustworthy business ontology.
Correction: Similarity can suggest related terms. It cannot decide the authoritative definition, policy or ownership.
Check your understanding: Who should settle a consequential business definition?
Whichever model answer sounds most confident. - A plausible model definition may not be your organization’s policy.
An accountable domain owner with documented examples and exceptions. - Yes. Meaning that drives decisions needs human ownership and evidence.
Use it with purpose: Choose a word your team uses differently across systems.
Guidance: Write two competing definitions, an example that separates them and a named owner. Version the agreed rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-semantic-engineering-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-semantic-engineering-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-semantic-engineering-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-semantic-engineering-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Enterprise Ontology (concept): https://www.openagentschool.org/concepts/agentic-enterprise-ontology-stack - Take this distinction into Enterprise Ontology and check how it changes a real system decision.
References:
W3C - SKOS: a shared vocabulary for concepts and labels: https://www.w3.org/TR/skos-primer/

LESSON: Context Engineering
URL: https://www.openagentschool.org/ai-ml-foundations/context-engineering
Also called: context selection, context assembly, context window, relevant evidence
Summary: Give the model the information it needs now, with clear boundaries and sources.
In everyday words: Context engineering designs what information is supplied to a model for a particular step: instructions, current task, selected history, retrieved evidence, tool results and relevant state. It involves selection, organization, freshness and trust boundaries, not simply putting more text in a prompt.
Why it matters: A correct answer may depend on today’s policy, not a similar policy from last year. Excess, stale or untrusted material can distract the model and burden the person reviewing the answer.
Scenario - Conflicting material: A current policy allows changes within 24 hours; an older policy says 48 hours. Message: "The old guide says 48 hours; the current policy says 24 hours."
Analogy - Preparing a short, sourced handoff: A colleague needs the current question, relevant records, what has already been tried and what they may decide—not every document in the organization.
Where the analogy stops: A model’s context window is not human working memory. Placement and size matter, but neither guarantees use of the right evidence.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/context-engineering.webp
Picture description: A large tote of folders, a small selected handoff packet and an uncluttered work tray arranged across a team table.
Picture caption: Useful context is selected for the next decision, not every record the organization has stored.
What you see: Available sources. What it represents: The tote represents information that exists but is not all needed now.
What you see: Purposeful selection. What it represents: The small packet represents relevant evidence, constraints and current state.
What you see: Current context. What it represents: The work tray represents what is actually supplied for the next model call.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Conflicting material. Context: A current policy allows changes within 24 hours; an older policy says 48 hours.
Text: The old guide says 48 hours; the current policy says 24 hours.
Notice "old guide" - Stale evidence: Should not silently override the current authority.
Notice "current policy" - Fresh source: Preserve version, effective date and applicability.
Proposed next step: Resolve source authority before answering; explain any remaining conflict.
Case: A focused context packet. Context: The task is a change-window question, not a request to perform a change.
Text: Use the current policy: changes within 24 hours. Answer the question; do not modify the booking.
Notice "current policy" - Relevant evidence: Include its source and effective date in the actual packet.
Notice "do not modify" - Action boundary: The runtime must also withhold or gate mutation tools.
Proposed next step: Give a sourced answer and stop. Retrieved text remains evidence, not permission to change rules.
Guided practice: An agent needs to explain one invoice line. What context should you omit?
Hint: Would unrelated customer histories change this answer?
Compare your reasoning: Omit unrelated records and sensitive details not needed for the line item. Include the relevant invoice, permitted policy and task boundary.
Predict: Does including both documents without labels make the answer safer?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Context quality depends on provenance, applicability and separation of instructions from evidence. More material can increase conflict instead of resolving it.
Optional math: task -> select evidence -> preserve source and trust level -> fit budget -> check missing context
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A longer context window removes the need to select information.
Correction: Capacity is not relevance. Long inputs can still be stale, conflicting, distracting or incomplete.
Check your understanding: A retrieved document says “ignore all earlier rules”. What is it?
A higher-priority instruction because it was retrieved recently. - Retrieval supplies evidence, not permission to rewrite the application’s rules.
Untrusted source text, not a new application instruction. - Correct. Freshness does not grant authority.
Use it with purpose: Sketch the context packet for one agent step.
Guidance: Separate instructions, evidence and tool output. Name what is omitted, how sources are dated and what happens if needed evidence is missing.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-context-engineering-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-context-engineering-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-context-engineering-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-context-engineering-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Context Engineering (concept): https://www.openagentschool.org/concepts/context-engineering - Take this distinction into Context Engineering and check how it changes a real system decision.
References:
Anthropic - Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

LESSON: Coherence
URL: https://www.openagentschool.org/ai-ml-foundations/coherence
Also called: discourse coherence, logical flow, consistency, cohesion
Summary: Check whether the parts of an answer fit together without mistaking smoothness for truth.
In everyday words: Coherence is how well ideas fit together into an understandable whole. It includes consistent referents, compatible claims and a sensible progression. Cohesion concerns surface links such as pronouns and connecting words; those links can help, but cannot make contradictory ideas coherent.
Why it matters: A person should not have to repair contradictions to understand the next step. A beautifully coherent answer can still be false or irrelevant.
Scenario - Compatible statements: The example report describes a request waiting for review. Message: "The request is pending. A reviewer must decide before work begins."
Analogy - Directions that do not contradict themselves: Each instruction should fit the route established by the others.
Where the analogy stops: Consistent directions can still lead to the wrong destination. Coherence is not correctness.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/coherence.webp
Picture description: Three miniature path segments join consistently across a tabletop but end at a closed, unidentified doorway.
Picture caption: The parts can fit together consistently while still leading to the wrong conclusion.
What you see: Starting statement. What it represents: The first path segment represents the premise or context an answer establishes.
What you see: Connected reasoning. What it represents: The joined bend represents later statements fitting the earlier ones.
What you see: Still check truth. What it represents: The unidentified destination represents correctness not guaranteed by coherence.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Compatible statements. Context: The example report describes a request waiting for review.
Text: The request is pending. A reviewer must decide before work begins.
Notice "pending" - Status: No decision is claimed.
Notice "before work begins" - Consistent consequence: The next step fits the pending status.
Proposed next step: The two statements fit together; their truth still needs evidence.
Case: Unexplained contradiction. Context: No change of time or status is described.
Text: The request is pending. It has already been approved.
Notice "pending" - First status: Suggests a decision is outstanding.
Notice "already been approved" - Conflicting status: Contradicts pending under the same status definition and time.
Proposed next step: Ask which status is current or clarify the timeline; do not smooth over the conflict.
Guided practice: Repair “The office is closed all day. Visit the office this afternoon” without inventing facts.
Hint: Identify the conflict before choosing which sentence to keep.
Compare your reasoning: Flag the conflict and verify the opening schedule. Do not invent afternoon opening hours to reconcile it.
Predict: Can adding “therefore” repair a contradiction?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Connecting words cannot fix incompatible claims. Clarify meanings and time frames, then check factual support separately.
Optional math: claims + referents + timeline -> compatibility and understandable progression
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: A coherent explanation must be accurate.
Correction: A false story can be internally consistent. Use separate checks for truth, relevance and source support.
Check your understanding: What does a coherence check establish?
That the answer is factually correct. - Smooth structure can carry false claims. Check truth independently.
Whether the ideas fit together, not whether they are true. - Correct. Coherence is one dimension, not a universal pass mark.
Use it with purpose: Read an answer one sentence at a time. Which sentence changes an earlier claim?
Guidance: Check actors, dates, units and status transitions. Preserve an explicit update when the facts genuinely changed.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-coherence-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-coherence-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-coherence-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-coherence-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Take this distinction into Agent Evaluation and check how it changes a real system decision.
References:
Microsoft - AI evaluation metrics and their distinct purposes: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-metrics-built-in

LESSON: Relevance
URL: https://www.openagentschool.org/ai-ml-foundations/relevance
Also called: answer relevance, query relevance, usefulness
Summary: Ask whether an answer helps with this person’s actual question.
In everyday words: Relevance is how directly information addresses the current question, task and situation. It is not the same as being on the same general topic. A relevant answer can still be inaccurate, and a true fact can still be irrelevant.
Why it matters: Long responses can consume attention without helping someone decide. Start with the answer they need, then offer optional depth.
Scenario - Direct answer: The question is “What time does pickup close today?” The verified time is 17:00. Message: "Pickup closes at 17:00 today."
Analogy - Answering the question asked at a service desk: Someone asking when the desk closes needs the time, not the history of the building.
Where the analogy stops: Sometimes a brief safety caveat or clarification is essential even if the person did not explicitly ask for it.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/relevance.webp
Picture description: A simple clock and short blank response card sit on a service counter, with a thick history book set aside.
Picture caption: An interesting answer can still miss what the person needs to know right now.
What you see: The actual need. What it represents: The clock represents a specific time-related question from the person.
What you see: Directly useful. What it represents: The short response represents information that addresses that question.
What you see: Interesting detour. What it represents: The book represents related material that may not help with the current need.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Direct answer. Context: The question is “What time does pickup close today?” The verified time is 17:00.
Text: Pickup closes at 17:00 today.
Notice "17:00 today" - Question answered: Supplies the requested time and day.
Proposed next step: Directly useful for deciding when to arrive; include the time zone if needed.
Case: Related, but not responsive. Context: The same question asks today’s closing time.
Text: Our pickup service has helped the community for many years.
Notice "pickup service" - Topic overlap: The subject matches, but the requested time is missing.
Proposed next step: Replace the background story with the closing time or explain that it is unknown.
Guided practice: A person asks “Do I need to bring ID?” What should lead the answer?
Hint: Answer the practical requirement first.
Compare your reasoning: State whether ID is required under the applicable policy, which kind if known, and what to do if they do not have it.
Predict: Does sharing the same topic make both replies equally relevant?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Relevance depends on the question and needed decision. Topic similarity is not enough.
Optional math: question + user situation -> necessary information -> answer coverage
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: More information is always more helpful.
Correction: Extra detail can hide the answer and increase cognitive load. Layer optional detail after the immediate need.
Check your understanding: A true paragraph does not answer the user’s question. Which check fails?
No check fails because every fact is true. - Truth alone does not establish usefulness for this question.
Relevance, even if factual accuracy passes. - Yes. Correct facts can still be unhelpful for the task.
Use it with purpose: Shorten an answer to the minimum that enables the next decision.
Guidance: Keep necessary qualifications. Put background and technical depth behind an optional expansion.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-relevance-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-relevance-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-relevance-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-relevance-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Take this distinction into Agent Evaluation and check how it changes a real system decision.
References:
Microsoft - AI evaluation metrics and their distinct purposes: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-metrics-built-in

LESSON: Groundedness: Is the Claim Supported?
URL: https://www.openagentschool.org/ai-ml-foundations/groundedness
Also called: faithfulness, source support, attribution, citation checking
Summary: Trace each important claim to evidence, then check whether that evidence deserves trust.
In everyday words: Groundedness asks whether an answer’s claims are supported by the supplied evidence. Grounding is the process of connecting an answer to evidence; groundedness is a property we check. A citation must support the actual claim, not merely point to a related document.
Why it matters: A source-supported answer can still be wrong if the source is outdated or unreliable. Check support and source quality separately.
Scenario - Supported by the supplied source: Source: “Standard delivery takes 3–5 business days.” No other timing is provided. Message: "Standard delivery takes 3–5 business days."
Analogy - A receipt for each important claim: You should be able to show where a claim came from and what that evidence actually says.
Where the analogy stops: A receipt can be wrong, old or about a different transaction. Traceability is not automatic truth.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/groundedness.webp
Picture description: A small bag of containers and its clipped receipt sit beside a separate statement card without an attached record.
Picture caption: Important claims need support you can inspect; a citation is not a decoration or a truth guarantee.
What you see: Source material. What it represents: The bag and contents represent the evidence relevant to the claim.
What you see: Traceable support. What it represents: The attached receipt represents checking what a cited source actually supports.
What you see: Unsupported claim. What it represents: The unattached statement represents a detail that still lacks evidence.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: Supported by the supplied source. Context: Source: “Standard delivery takes 3–5 business days.” No other timing is provided.
Text: Standard delivery takes 3–5 business days.
Notice "3–5 business days" - Supported claim: Matches the source’s range and business-day qualification.
Proposed next step: Supported by this source. Verify the policy’s date and applicability before relying on it.
Case: An unsupported upgrade. Context: The supplied source still only says “3–5 business days”.
Text: Your delivery is guaranteed tomorrow.
Notice "guaranteed tomorrow" - Unsupported claim: Neither the guarantee nor tomorrow follows from the source.
Proposed next step: Remove the invented promise. A link to the delivery policy would not make this claim grounded.
Guided practice: A source says “up to 20 people”. A summary says “exactly 20 people”. Is it supported?
Hint: Compare the quantity’s qualification.
Compare your reasoning: No. The source gives an upper bound, not an exact count. Preserve “up to”.
Predict: Would attaching a citation make the second answer supported?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: A citation is a pointer, not proof of entailment. Check the statement against the actual passage, including exceptions and uncertainty.
Optional math: claim -> supporting passage -> applicability, freshness and reliability checks
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: Grounded means true in the world.
Correction: It means supported relative to evidence under the chosen rubric. A stale source can support a currently false answer.
Check your understanding: A reply faithfully quotes an obsolete policy. What can be true?
It must be accurate because it has a source. - Evidence can be outdated. Verify its authority and effective date.
It is source-supported but currently inaccurate. - Correct. Source support and current-world correctness are separate checks.
Use it with purpose: Choose three consequential claims in an answer. Can you show support for each?
Guidance: Record source passages, versions and applicability. Mark unsupported claims explicitly; ask or abstain when evidence is insufficient.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-groundedness-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-groundedness-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-groundedness-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-groundedness-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Hallucination & Grounding (concept): https://www.openagentschool.org/concepts/hallucination-grounding - Take this distinction into Hallucination & Grounding and check how it changes a real system decision.
References:
Microsoft - AI evaluation metrics and their distinct purposes: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-metrics-built-in

LESSON: Accuracy: Correct About What?
URL: https://www.openagentschool.org/ai-ml-foundations/accuracy
Also called: factual accuracy, classification accuracy, correctness, verification
Summary: Check correctness against an appropriate reference, not confidence or fluency.
In everyday words: Accuracy in ordinary language means correctness about the thing being claimed. For a classifier, accuracy is the fraction of evaluated predictions that match reference labels. For generated answers, correctness may require claim-by-claim factual, numerical and temporal checks; there is no universal accuracy score for every answer.
Why it matters: A correct-looking explanation can use the wrong unit, date or calculation. Even a high classification accuracy can hide failures on rare but important cases.
Scenario - A checkable calculation: A fictional bill has three items at 4 units each, with no tax or other charge. Message: "The total is 12 units."
Analogy - Checking a bill against quantities and prices: You compare the amount with records and arithmetic, not with how confidently it was presented.
Where the analogy stops: Some tasks lack one uncontested reference answer. Define the rubric and acknowledge uncertainty or disagreement.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/accuracy.webp
Picture description: Three reusable cups, a blank receipt with matching counters and an unmarked calculator arranged for a quantity check.
Picture caption: Check a claim against suitable evidence and arithmetic, not the confidence of its presentation.
What you see: Observed quantity. What it represents: The cups represent facts or reference observations used to assess correctness.
What you see: Claim to compare. What it represents: The receipt represents an asserted quantity that still needs verification.
What you see: Independent check. What it represents: The calculator represents an appropriate checking method, not a guarantee from fluent wording.
Interactive annotated wording: authored teaching cases, not live model predictions or quality scores.
Case: A checkable calculation. Context: A fictional bill has three items at 4 units each, with no tax or other charge.
Text: The total is 12 units.
Notice "12 units" - Correct under stated assumptions: Three multiplied by four is twelve; the unit is preserved.
Proposed next step: The arithmetic matches the defined bill. This does not validate other unstated charges.
Case: A fluent numerical error. Context: The same bill has three items at 4 units each, with no extra charge.
Text: The total is 16 units.
Notice "16 units" - Incorrect total: The total does not match the stated quantities and prices.
Proposed next step: Correct the calculation; do not treat confident wording as evidence.
Guided practice: A classifier says “normal” for all 100 cases; 99 are normal. Is 99% enough?
Hint: What happened to the one important exception?
Compare your reasoning: It has 99% accuracy but misses every exception. Inspect recall, false negatives and the cost of that miss.
Predict: What independent check distinguishes these two fluent answers?
Try: Choose each prepared case. Compare the exact wording, context, annotated observations and proposed response. These are authored examples, not predictions from a running language model.
Explain: Use the task’s actual reference: here, arithmetic and a bill definition. In other tasks, evaluate facts, units, timing and the reliability of reference labels.
Optional math: classification accuracy = correct predictions / evaluated predictions
For open answers: define the claim-level correctness rubric first.
->: A relationship or design decision in this teaching sketch, not proof of meaning or authorization.
This is a conceptual sketch, not a mathematical law or executable parser. Real systems need representative examples, explicit definitions and checks with the people affected.
Common misconception: One high accuracy number proves an agent is safe.
Correction: Break results down by error type and relevant groups. Check task adherence, groundedness, relevance and consequences separately.
Check your understanding: An answer is coherent, relevant and confident. Is its accuracy established?
Yes. Those qualities imply correctness. - A convincing answer can still be false. Add independent verification.
No. Verify its claims against suitable evidence or calculations. - Correct. These dimensions can vary independently.
Use it with purpose: Define a correctness check for one answer your assistant gives.
Guidance: Specify the reference, units, time frame, exceptions and reviewer disagreement. Do not collapse all quality dimensions into one unexplained number.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-accuracy-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-accuracy-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-accuracy-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-accuracy-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Take this distinction into Agent Evaluation and check how it changes a real system decision.
References:
Microsoft - AI evaluation metrics and their distinct purposes: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-metrics-built-in

LESSON: Descriptive Statistics: Center and Spread
URL: https://www.openagentschool.org/ai-ml-foundations/descriptive-statistics
Also called: mean, median, variance, standard deviation, outliers, descriptive statistics
Summary: Describe what is typical and what varies without hiding the difficult cases.
In everyday words: Descriptive statistics summarize observed data. The mean adds values and divides by their count; the median is the middle ordered value. Variance summarizes squared distances from the mean, and standard deviation returns that spread to the original units. A summary describes these observations; it does not automatically generalize to everyone.
Why it matters: An agent can feel slow for some people even when its average looks acceptable. Inspect spread, tails and relevant groups alongside a central value.
Scenario - Before changing the system: Five fictional requests take 2, 3, 3, 4 and 20 seconds. The mean is 6.4 seconds, but the median is 3. The slow request matters even though it is not typical.
Analogy - An average journey is not every journey: A travel-time summary helps planning, but one very long wait may matter more than the average to the person experiencing it.
Where the analogy stops: Different distributions can share the same mean and spread. Keep the underlying observations and context available.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/descriptive-statistics.webp
Picture description: Five reusable bottles stand on a common tabletop, with several similar heights and one distinctly taller bottle.
Picture caption: A typical value and the spread tell different parts of the story; one unusual observation can matter.
What you see: Typical cluster. What it represents: The similar bottles help picture a central value without describing every observation.
What you see: Variation. What it represents: The different heights represent spread that a single average can hide.
What you see: Unusual observation. What it represents: The tall bottle represents a value that can pull the mean while leaving the median less changed.
Guided practice: A dashboard reports only mean response time. What would you add?
Hint: Ask who waits longer and how often.
Compare your reasoning: Show the median, a suitable high percentile, the sample size and group-level distribution, with a clear measurement window.
Predict: If the fifth request slows further, which changes more: mean or median?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: The slow value pulls the mean upward and increases spread; the middle of these ordered values stays at 3 seconds.
Optional math: mean = sum(x) / n; sample variance s² = sum((x - mean)²) / (n - 1); sample SD = sqrt(s²)
n: Number of observations; sample variance requires at least two.
s: Sample standard deviation, in the same units as x.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: The average describes what every user experiences.
Correction: The mean can hide slow tails, unequal groups and multiple clusters. Use the median, distribution and relevant subgroup summaries too.
Check your understanding: What happens to the median when only the largest of these five values increases?
It must increase as much as the mean. - Not yet. The median depends on order and the middle value; the mean uses every magnitude.
It stays at the unchanged middle value. - Yes. The median depends on order and the middle value; the mean uses every magnitude.
Use it with purpose: Which summary could hide a harmful experience in your agent workflow?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-descriptive-statistics-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-descriptive-statistics-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-descriptive-statistics-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-descriptive-statistics-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Sampling Bias: Who Is Missing?
URL: https://www.openagentschool.org/ai-ml-foundations/sampling-bias
Also called: sampling, selection bias, population, representative sample, sampling uncertainty
Summary: Separate an unrepresentative sample from ordinary random variation.
In everyday words: A population is the set of cases you want to understand; a sample is the subset you observe. Sampling bias happens when the selection process systematically misrepresents that target. Sampling variability is different: even well-designed random samples differ by chance.
Why it matters: An evaluation made only from convenient, short or successful interactions can overstate an agent’s usefulness.
Scenario - Before changing the system: The real workload is half easy and half hard. The agent succeeds on 90% of easy tasks and 50% of hard ones. An 80%-easy sample reports 82% success; the target mix gives 70%.
Analogy - Listening only near the entrance: Feedback collected from people who reach one entrance misses those who could not get there or used another route.
Where the analogy stops: Group weighting helps only when relevant groups, target shares and within-group coverage are credible.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/sampling-bias.webp
Picture description: A mixed crate of small and large apples sits beside a sample basket containing only large shiny apples.
Picture caption: A bigger basket of the same selected cases still does not show what was left out.
What you see: Target population. What it represents: The varied source crate represents the full workload the evaluation aims to describe.
What you see: Selected sample. What it represents: The limited basket represents a selection process favoring one kind of case.
What you see: Missing coverage. What it represents: The remaining apples represent cases that more of the same sampling will still exclude.
Guided practice: Your test set contains only completed chats. Which cases should you investigate?
Hint: Completion itself can select for easier experiences.
Compare your reasoning: Inspect abandoned, timed-out, inaccessible and unsupported-language interactions, using appropriate privacy and consent safeguards.
Predict: Can changing the sample mix change the score without changing the agent?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: Yes. The experiment holds the two group rates fixed. Only representation changes, exposing bias rather than random noise.
Optional math: target rate = sum(target group share × group rate)
group share: A group’s fraction of the intended population; the shares sum to one.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: A million examples must be representative.
Correction: More observations can narrow random uncertainty around a biased estimate. Size does not repair who was excluded.
Check your understanding: What changes the biased task mix in this example?
Duplicating every easy example ten times. - Not yet. Representation must address the selection mechanism; duplicated observations do not create missing coverage.
Sampling to cover the intended workload, not just adding more easy tasks. - Yes. Representation must address the selection mechanism; duplicated observations do not create missing coverage.
Use it with purpose: Who or what is absent from the evidence you would use to approve an agent?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-sampling-bias-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-sampling-bias-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-sampling-bias-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-sampling-bias-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Conditional Probability: Read the Base Rate
URL: https://www.openagentschool.org/ai-ml-foundations/conditional-probability
Also called: Bayes theorem, base rates, conditional probability, positive predictive value, prevalence
Summary: Understand why an accurate-looking alert can still need careful verification.
In everyday words: Conditional probability asks how likely one event is within cases where another event is known. P(issue | alert) concerns alerted cases; P(alert | issue) concerns cases with an issue. Bayes’ rule connects these directions using the base rate: how common the issue was before the alert.
Why it matters: Rare-event detectors can generate more false alerts than true ones. Learners need the denominator, not just an impressive detection rate.
Scenario - Before changing the system: Among 10,000 requests, 1% have an issue. A detector catches 90 of those 100 and falsely flags 495 of the other 9,900. Only 90 of 585 alerts are true issues.
Analogy - Two different queues: “How many issue cases were flagged?” and “How many flagged cases have an issue?” count people in different queues.
Where the analogy stops: The rates depend on the population and measurement quality; the toy detector does not represent a real product.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/conditional-probability.webp
Picture description: A large parcel collection, a smaller mixed inspection tray and one opened parcel arranged for a routine check.
Picture caption: “Among the flagged cases” is a different question from “among cases with an issue.”
What you see: Starting mix. What it represents: The full collection represents the base rate before an alert is observed.
What you see: Condition on alerts. What it represents: The inspection tray represents the subset that defines the alert-conditioned denominator.
What you see: Verify the condition. What it represents: The opened parcel represents checking which alerts actually meet the criterion.
Guided practice: An alert dashboard gives sensitivity but not prevalence or false-positive rate. Can you estimate the fraction of real issues among alerts?
Hint: Identify the missing denominator.
Compare your reasoning: No. Obtain a relevant base rate and false-positive rate, then inspect calibration and population shift before interpreting the alert.
Predict: With sensitivity and specificity fixed, what happens to alert reliability when issues become more common?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: The fraction of true issues among alerts increases. The detector’s conditional error rates stayed fixed while the base rate changed.
Optional math: P(issue | alert) = sensitivity × prevalence / [sensitivity × prevalence + (1 - specificity) × (1 - prevalence)]
sensitivity: P(alert | issue).
specificity: P(no alert | no issue).
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: A detector that catches 90% of issues makes 90% of its alerts correct.
Correction: Sensitivity is not the fraction of correct alerts. That fraction also depends on false positives and the base rate.
Check your understanding: Which denominator belongs to P(issue | alert)?
All cases with a real issue. - Not yet. The event after the conditioning bar defines the cases being considered.
All alerted cases. - Yes. The event after the conditioning bar defines the cases being considered.
Use it with purpose: What would a rare-event alert mean before a person verifies it?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-conditional-probability-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-conditional-probability-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-conditional-probability-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-conditional-probability-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Hypothesis Testing: A Claim Meets Noisy Evidence
URL: https://www.openagentschool.org/ai-ml-foundations/hypothesis-testing
Also called: null hypothesis, alternative hypothesis, statistical significance, hypothesis test
Summary: Compare data with a stated baseline without confusing a test decision with proof.
In everyday words: A hypothesis test compares observations with a specified null model, often “no change”. An alternative describes departures of interest. A pre-chosen decision rule may reject the null when the data are sufficiently inconsistent with it. Failing to reject is not the same as proving the null true.
Why it matters: A small observed improvement might be noise; a detectable improvement might still be unhelpful. A test is one piece of evaluation evidence.
Scenario - Before changing the system: A fictional quality score has known population standard deviation 10. With 100 independent observations, an observed mean 2 units above baseline has z = 2 and a two-sided p-value near 0.046.
Analogy - A comparison rule agreed before measuring: Like checking a delivery against a pre-agreed tolerance, decide what counts as unusual before inspecting the result.
Where the analogy stops: Statistical thresholds concern a sampling model. They do not automatically encode business value, safety or causal attribution.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/hypothesis-testing.webp
Picture description: A reference container with a simple gauge, a varied sample of containers and an unsigned decision card.
Picture caption: Set the comparison rule before looking at the sample; the resulting decision is not proof.
What you see: Stated reference. What it represents: The reference object represents the null model and a pre-chosen comparison rule.
What you see: Noisy observations. What it represents: The varied containers represent measurements that can differ through sampling variation.
What you see: Limited conclusion. What it represents: The unsigned card represents a test decision that still needs effect size and assumption checks.
Guided practice: A team chooses its metric after seeing which result looks best. What should change?
Hint: The decision procedure includes the selection process.
Compare your reasoning: Pre-specify a primary metric and comparison plan, or account for the searched family and validate the finding on fresh evidence.
Predict: With the observed difference unchanged, what happens when independent sample size increases?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: Standard error falls, so the same observed difference is larger relative to sampling noise. That does not increase its size in useful units.
Optional math: SE = sigma / sqrt(n); z = (sample mean - baseline) / SE
SE: Standard error: spread of sample means under repeated independent sampling.
sigma: Known population standard deviation in this simplified normal model.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: Not significant means the systems are equivalent.
Correction: A test may simply lack enough evidence. Equivalence needs a justified margin and an appropriate design, not a failed difference test.
Check your understanding: What does failing to reject the no-change null establish?
Proof that there is no meaningful difference. - Not yet. A non-rejection can reflect noise, limited power or a genuinely small effect. It does not prove equivalence.
Insufficient evidence against that null under this test. - Yes. A non-rejection can reflect noise, limited power or a genuinely small effect. It does not prove equivalence.
Use it with purpose: What would you decide before comparing two versions of an agent?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-hypothesis-testing-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-hypothesis-testing-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-hypothesis-testing-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-hypothesis-testing-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Type I Error: A False Alarm
URL: https://www.openagentschool.org/ai-ml-foundations/type-i-error
Also called: type 1 error, type I error, false rejection, alpha, false positive
Summary: See what it means to reject a true null hypothesis and why stricter cutoffs have a cost.
In everyday words: A Type I error happens when a test rejects a null hypothesis that is actually true. If the null is “no improvement”, this is a false claim of improvement. Alpha is the procedure’s chosen false-rejection rate or bound when its null and assumptions hold; it is not the probability that your particular finding is false.
Why it matters: False signals can trigger unnecessary model changes, investigations or deployment costs. Control the testing procedure, not only the displayed threshold.
Scenario - Before changing the system: Suppose an agent update makes no population-level difference. Random variation still produces some apparently positive samples. A one-sided rule with cutoff z = 1.6 falsely rejects about 5.48% of true-null samples in the teaching model.
Analogy - An alarm when nothing changed: An alarm can be useful while sometimes sounding without the condition it is intended to detect.
Where the analogy stops: “False positive” maps to Type I error only after you define the null and what counts as rejecting it. An alarm analogy is not a statistical proof.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/type-i-error.webp
Picture description: An empty parcel mat and a moving curtain sit beside a softly lit arrival-notification device.
Picture caption: A notification can appear even when the condition it claims to detect is absent.
What you see: Null is true. What it represents: The empty mat represents no real change in the condition being tested.
What you see: Variation or noise. What it represents: The curtain represents variation that can make an ordinary situation look unusual.
What you see: False rejection. What it represents: The lit indicator represents rejecting the true null: a Type I error.
Guided practice: A team tests one hundred unrelated improvements at 5% each. Is the overall false-alarm risk still 5%?
Hint: There are now many opportunities for a false signal.
Compare your reasoning: No. Define the comparison family and use a suitable multiplicity procedure or fresh confirmatory evaluation; inspect the Multiple Testing lesson.
Predict: If you raise the rejection cutoff, what happens to false alarms and missed real improvements?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: A stricter cutoff decreases alpha but increases beta at a fixed real effect and sample size. One error cannot be minimized without considering the other and the consequences.
Optional math: alpha = P(reject H0 | H0 true); one-sided normal alpha = 1 - Phi(cutoff)
H0: The null hypothesis, here no population change.
Phi: Standard-normal cumulative probability up to the cutoff.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: Alpha = 0.05 means this significant finding has a 5% chance of being false.
Correction: Alpha describes the long-run rejection procedure under a true null. A finding’s credibility also needs design, prior context, effect size and replication.
Check your understanding: When does a Type I error occur?
A real alternative exists, but the test fails to reject the null. - Not yet. Type I is a false rejection. Missing a specified real alternative is Type II.
The null is true, but the test rejects it. - Yes. Type I is a false rejection. Missing a specified real alternative is Type II.
Use it with purpose: What would an unnecessary false alarm cost in your evaluation workflow?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-type-i-error-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-type-i-error-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-type-i-error-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-type-i-error-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Type II Error: A Missed Signal
URL: https://www.openagentschool.org/ai-ml-foundations/type-ii-error
Also called: type 2 error, type II error, beta, false negative, missed detection
Summary: Understand why no detected change is not proof that nothing changed.
In everyday words: A Type II error happens when a test fails to reject its null even though a specified alternative is true. Beta is the chance of that miss at a particular effect, sample size, variability and decision rule. A tiny effect and a large effect generally have different beta values.
Why it matters: A small or noisy evaluation can miss an important improvement or regression. The null, direction and effect of interest must be explicit.
Scenario - Before changing the system: In the one-sided teaching model with 100 observations and sigma = 10, a real 2-unit increase is missed about 34.46% of the time at cutoff z = 1.6.
Analogy - A quiet alarm does not settle the question: A detector can remain quiet even when the condition exists, especially when the signal is weak relative to noise.
Where the analogy stops: In practice you do not know the true state from the test result alone. Beta is evaluated under a specified alternative model.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/type-ii-error.webp
Picture description: A parcel rests on its mat while a partially obscured sensor and an unlit notification device remain nearby.
Picture caption: No notification does not prove nothing happened; a real signal can be missed.
What you see: Real alternative. What it represents: The parcel represents a specified change that truly exists in the teaching example.
What you see: Limited detection. What it represents: The obstructed view represents why evidence can fail to reveal a real change.
What you see: Missed signal. What it represents: The quiet device represents failing to reject the null under that alternative: Type II error.
Guided practice: An inconclusive evaluation is called “proof of safety”. What evidence is missing?
Hint: Ask what adverse change the evaluation could reliably detect.
Compare your reasoning: Define meaningful harms, representative exposure and detection sensitivity. Examine uncertainty and power for relevant changes; non-rejection alone cannot establish safety.
Predict: At the same cutoff, does making the true increase larger reduce misses?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: Yes. The shifted sampling distribution moves further into the rejection region. Beta depends on which real effect you ask the test to detect.
Optional math: beta(delta) = P(do not reject H0 | true effect = delta); power = 1 - beta
delta: The specified true departure from the null, not merely the observed estimate.
beta: Miss probability for that alternative and testing procedure.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: Alpha and beta add up to one.
Correction: They condition on different states of the world. Power and beta sum to one for the same specified alternative.
Check your understanding: What is the complement of beta for a specified alternative?
Alpha: the probability of rejecting a true null. - Not yet. Power and beta partition outcomes under the same alternative; alpha uses a true-null condition.
Power: the probability of detecting that alternative. - Yes. Power and beta partition outcomes under the same alternative; alpha uses a true-null condition.
Use it with purpose: What important change could your current evaluation be too weak to detect?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-type-ii-error-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-type-ii-error-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-type-ii-error-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-type-ii-error-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: P-Values: Unusual Under Which Assumption?
URL: https://www.openagentschool.org/ai-ml-foundations/p-values
Also called: p value, p-value, significance level, statistical evidence
Summary: Read a p-value as a conditional data comparison, not a probability that a claim is true.
In everyday words: A p-value is the probability, under the null model and its assumptions, of a test statistic at least as extreme as the one observed, using the test’s defined direction. It is not P(null is true), not an effect size, and not the probability that “chance caused the result”.
Why it matters: A threshold can become a misleading yes/no shortcut. Report the estimate, uncertainty and procedure that produced the value.
Scenario - Before changing the system: A standardized observation z = 2 has a two-sided p-value about 0.0455 under a standard-normal null. Both tails count because changes in either direction were part of the planned test.
Analogy - Surprise under a stated story: Ask how unusual the measurement would be if the reference story were correct, not how likely the story is after seeing the measurement.
Where the analogy stops: Several different stories can make data unusual. Rejecting a null does not identify a cause or validate every alternative.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/p-values.webp
Picture description: A mixed laundry bag, a separately selected handful of socks and an open observation notebook with a lens.
Picture caption: Ask how unusual the observation is under a stated model—not how likely the model is to be true.
What you see: Assumed model. What it represents: The bag mix represents a specified null model used for the calculation.
What you see: Observed result. What it represents: The selected handful represents the data statistic compared with that reference.
What you see: Conditional question. What it represents: The notebook represents evaluating extremeness under the model, not assigning it a truth probability.
Guided practice: One team reports p = 0.049 and another p = 0.051. Should you treat their evidence as radically different?
Hint: Threshold labels can exaggerate a tiny numeric difference.
Compare your reasoning: No. Inspect their effects, intervals, designs and assumptions. A reporting cutoff does not create a sharp change in practical evidence.
Predict: Do z = 2 and z = -2 have different two-sided p-values?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: No. Both are equally far from zero. Choosing a one-sided test after observing the sign would change the planned procedure.
Optional math: two-sided p = 2 × [1 - Phi(|z|)]
|z|: Distance of the observed standardized score from zero.
Phi: Cumulative area under the standard-normal null curve.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: p = 0.03 means a 97% probability that the improvement is real.
Correction: The p-value is computed assuming the null; it does not assign posterior probabilities to hypotheses.
Check your understanding: What is assumed when calculating the p-value?
That the alternative is already known to be true. - Not yet. A p-value measures data extremeness within the null model, not truth probabilities for the competing claims.
The specified null model and testing assumptions. - Yes. A p-value measures data extremeness within the null model, not truth probabilities for the competing claims.
Use it with purpose: What would you report alongside a p-value to make a model comparison understandable?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-p-values-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-p-values-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-p-values-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-p-values-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Confidence Intervals: Show the Uncertainty
URL: https://www.openagentschool.org/ai-ml-foundations/confidence-intervals
Also called: confidence interval, margin of error, standard error, frequentist coverage
Summary: Show an estimate with a range from a stated method, and distinguish it from individual variation.
In everyday words: A confidence interval is produced by a procedure with a stated long-run coverage rate under its assumptions. A 95% method covers the fixed population parameter in about 95% of repeated samples. Once computed, an interval either covers that parameter or does not; 95% is not a posterior probability for this particular interval.
Why it matters: A point estimate hides how precisely an evaluation measured the population quantity. Intervals make meaningful uncertainty visible.
Scenario - Before changing the system: With sample mean 50, known sigma 10 and n = 100 independent normal observations, the approximate 95% interval for the population mean is 48.04 to 51.96.
Analogy - An estimate with a measuring band: The band shows the precision of a method, rather than pretending the center is exact.
Where the analogy stops: It is not a fence containing 95% of individual outcomes. Bias or invalid assumptions can also break the stated coverage.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/confidence-intervals.webp
Picture description: Two adjustable stop blocks surround a center marker on an unnumbered measuring rail, with varied containers behind it.
Picture caption: A band around an estimated average is not a promise about every individual observation.
What you see: Lower limit. What it represents: The left stop helps picture one endpoint produced by an interval method.
What you see: Point estimate. What it represents: The center marker represents an estimated population quantity, not an exact known value.
What you see: Upper limit. What it represents: The right stop represents the other endpoint; coverage describes the method over repeated samples.
Guided practice: A mean improvement interval includes both a useful gain and a harmful loss. What does it support?
Hint: The unresolved possibilities matter for the decision.
Compare your reasoning: It supports uncertainty, not equivalence. Consider more representative evidence or a lower-risk trial before a consequential change.
Predict: If you quadruple independent sample size, how does the interval width change?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: In this model it halves, because standard error scales as one divided by the square root of n. The estimate can move between real samples too.
Optional math: approximate 95% normal interval = sample mean ± 1.96 × sigma / sqrt(n)
sigma: Known population SD; when estimated, different methods such as a t interval may be needed.
n: Number of independent observations.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: The 95% confidence interval contains 95% of future response times.
Correction: An interval for the mean concerns a population average. Individual prediction requires a different, typically wider interval.
Check your understanding: What does the 95% refer to?
A guarantee that 95% of future observations lie in this range. - Not yet. Coverage concerns the population parameter across repeated intervals, not the spread of individual observations.
Long-run coverage of the interval-producing method under its assumptions. - Yes. Coverage concerns the population parameter across repeated intervals, not the spread of individual observations.
Use it with purpose: Which decisions would change if you saw the uncertainty around a reported average?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-confidence-intervals-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-confidence-intervals-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-confidence-intervals-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-confidence-intervals-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Effect Size: Is the Change Useful?
URL: https://www.openagentschool.org/ai-ml-foundations/effect-size
Also called: effect size, practical significance, standardized difference, minimum meaningful effect
Summary: Separate how much changed from how detectable the change was.
In everyday words: An effect size describes the magnitude of a difference or relationship. A raw difference stays in useful units; a standardized difference divides by a stated reference spread. Neither significance nor a conventional “small/large” label decides whether the effect matters to the people affected.
Why it matters: A huge test set can detect a trivial gain. A meaningful gain can remain uncertain in a small test. Report magnitude and uncertainty together.
Scenario - Before changing the system: A fictional workflow values saving at least 3 seconds per task. An observed saving of 2 seconds is below that chosen threshold, even if a large study could make it statistically significant.
Analogy - Measurable is not the same as helpful: A ruler can detect a tiny change that makes no useful difference to the job.
Where the analogy stops: Usefulness depends on task frequency, costs, quality and harms; there is no universal seconds-saved threshold.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/effect-size.webp
Picture description: Two similarly packed totes, an unnumbered measuring strip and a storage cubby awaiting a practical fit check.
Picture caption: A difference can be detectable without being large enough to help with the actual task.
What you see: Size of the change. What it represents: The two totes represent the magnitude of a difference in useful real-world units.
What you see: Measured carefully. What it represents: The strip represents detectability, which does not decide practical value.
What you see: Meaningful benefit. What it represents: The cubby represents a task-specific threshold for whether the change is useful.
Guided practice: A model is 0.1 seconds faster but costs twice as much. What comparison is missing?
Hint: Relate the measured difference to the workflow.
Compare your reasoning: Estimate user impact, workload frequency, cost and any quality or safety change, with uncertainty. Do not approve it solely because a test rejects no change.
Predict: If reference spread doubles while seconds saved stay fixed, what happens to the standardized difference?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: It halves. The raw benefit has not changed, which is why the denominator and original units need to stay visible.
Optional math: standardized difference = observed difference / chosen reference SD
reference SD: The explicitly defined standard deviation used to scale the difference; not automatically a pooled estimate.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: A statistically significant effect must be important.
Correction: Detectability and practical importance are different. Define a meaningful change before evaluating, and consider uncertainty and tradeoffs.
Check your understanding: Which report best helps a practical decision?
Only the word “significant”. - Not yet. A threshold label hides magnitude and decision context.
The difference in useful units, its uncertainty and a justified relevance threshold. - Yes. A threshold label hides magnitude and decision context.
Use it with purpose: What improvement would actually matter enough to justify changing your agent?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-effect-size-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-effect-size-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-effect-size-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-effect-size-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Statistical Power: Can the Test Notice?
URL: https://www.openagentschool.org/ai-ml-foundations/statistical-power
Also called: statistical power, power analysis, 1 minus beta, detectable effect
Summary: Plan whether an evaluation can detect a change that matters.
In everyday words: Power is the probability that a test rejects its null when a specified alternative is true: one minus beta. It depends on effect size, variability, independent sample size, test design and alpha. Plan it for a meaningful effect before collecting data.
Why it matters: An underpowered evaluation can make useful improvements or harmful regressions look inconclusive.
Scenario - Before changing the system: For a one-sided normal-mean test with alpha 0.05, sigma 10 and n = 100, a real 2-unit increase has about 64% power. That is not a 64% probability that an observed improvement is true.
Analogy - A measurement plan needs enough resolution: To notice a subtle signal in noise, the observation plan must be sensitive enough for the change you care about.
Where the analogy stops: More readings help only as the design permits; repeated correlated readings are not independent evidence.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/statistical-power.webp
Picture description: A subtle contrasting stitch, a stable inspection lens and several separate fabric samples on a modern worktable.
Picture caption: Plan whether the observation method can notice the size of change that actually matters.
What you see: Effect worth finding. What it represents: The subtle stitch represents a specified real departure the evaluation aims to detect.
What you see: Detection design. What it represents: The lens represents the test’s ability to distinguish signal from variation.
What you see: Independent evidence. What it represents: The separate swatches represent additional observations under an appropriate design.
Guided practice: A team claims high power but will not state the effect used. What should you ask?
Hint: Detecting only a huge effect may be unhelpful.
Compare your reasoning: Ask for the meaningful effect, assumed variability, independent sample units, test direction and alpha, then inspect the design and sample-size calculation.
Predict: At the same sample size, which is easier to detect: a true increase of 1 unit or 4?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: The larger change has greater power. A single power number without its assumed effect does not describe an evaluation adequately.
Optional math: power(delta) = 1 - Phi(1.64485 - delta × sqrt(n) / sigma)
1.64485: Approximate one-sided standard-normal cutoff for alpha = 0.05.
delta: The true increase the test is planned to detect.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: 80% power means an 80% chance a completed finding is true.
Correction: Power is a rejection probability under a specified true effect. It is not a posterior confidence in a finding.
Check your understanding: What happens to power when the meaningful effect gets smaller, all else fixed?
It stays fixed because alpha stays fixed. - Not yet. Alpha controls true-null false rejections; power depends on the specified alternative.
It decreases. - Yes. Alpha controls true-null false rejections; power depends on the specified alternative.
Use it with purpose: What change should your evaluation reliably notice before you trust a non-rejection?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-statistical-power-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-statistical-power-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-statistical-power-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-statistical-power-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Sample Size: Enough Independent Evidence
URL: https://www.openagentschool.org/ai-ml-foundations/sample-size
Also called: sample size, precision planning, independent observations, standard error of mean
Summary: Plan for precision without mistaking more rows for better evidence.
In everyday words: Sample-size planning connects a decision goal to assumptions about variation and study design. For a normal-population mean with known standard deviation, precision improves with the square root of the number of independent observations. Planning a margin of error is different from planning power for a comparison.
Why it matters: A fixed “we tested 100 prompts” rule ignores uncertainty, task mix and repeated observations from the same source.
Scenario - Before changing the system: With known sigma 10, a target 95% margin of 2 units needs ceil((1.96 × 10 / 2)^2) = 97 independent observations. A margin of 1 needs 385.
Analogy - More independent views, not more photocopies: Repeating a record can increase a row count without adding a new view of the problem.
Where the analogy stops: Even independent observations need appropriate sampling. Large datasets can still systematically exclude important groups.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/sample-size.webp
Picture description: Separate physical containers sit beside repeated image cards of one container and a tray of new containers.
Picture caption: New independent observations can add evidence; copies of the same observation do not.
What you see: Observed units. What it represents: The distinct containers represent genuinely separate sampling units.
What you see: Copies are not more. What it represents: The repeated cards represent duplicated rows that do not add independent information.
What you see: New coverage. What it represents: The additional containers represent collecting further suitable observations.
Guided practice: You can test more variants of the same few prompts or collect new representative tasks. What distinction matters?
Hint: Separate repeatability from population coverage.
Compare your reasoning: Repeated variants can measure within-task variation but not replace independent coverage of the intended workload. Plan both at the correct analysis level.
Predict: Does halving the requested margin require twice or about four times the observations?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: About four times. The square-root relationship gives diminishing precision gains, and the final count must be rounded up.
Optional math: n = ceil((1.96 × sigma / target margin)^2)
ceil: Round upward to the next whole observation.
target margin: Desired half-width of the normal-mean 95% interval.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: Ten thousand calls from one session equal ten thousand independent cases.
Correction: Shared users, tasks or sessions can create dependence. Choose the sampling and analysis unit deliberately.
Check your understanding: What does this experiment’s n formula plan?
A universally correct A/B-test sample size. - Not yet. Sample size depends on the goal and design; comparisons, binary outcomes and clustered data need suitable methods.
Precision of a mean under known-variance normal assumptions. - Yes. Sample size depends on the goal and design; comparisons, binary outcomes and clustered data need suitable methods.
Use it with purpose: What is one genuinely independent evaluation unit in your application?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-sample-size-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-sample-size-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-sample-size-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-sample-size-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Multiple Testing: Count Every Chance to Find a Signal
URL: https://www.openagentschool.org/ai-ml-foundations/multiple-testing
Also called: multiple comparisons, multiplicity, Bonferroni, family-wise error, false discovery rate, p-hacking
Summary: Account for the many metrics, variants and subgroups searched before reporting a winner.
In everyday words: Multiple testing creates repeated opportunities for false rejections. Family-wise error is the probability of at least one false rejection in a defined family. False discovery rate is a different target: the expected proportion of false discoveries among the rejected hypotheses. Choose the error target and procedure to fit the decision.
Why it matters: Trying many prompts, metrics or groups and reporting only the best result makes ordinary thresholds misleading.
Scenario - Before changing the system: Twenty independent true-null tests at alpha 0.05 have a 1 - 0.95^20, or about 64%, chance of at least one false rejection. The expected number of false rejections is 1, not a guarantee of exactly one.
Analogy - More attempts, more chances for a coincidental hit: A surprising-looking outcome is less surprising when you know how many opportunities there were to find it.
Where the analogy stops: The exact independence formula is a teaching case; real metrics may be correlated and adaptive searching needs additional care.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/multiple-testing.webp
Picture description: Many simple stamps and a sheet of varied trial impressions sit beside one singled-out attractive impression.
Picture caption: A selected result means something different when you know how many attempts came before it.
What you see: Many opportunities. What it represents: The stamps represent the family of metrics, variants or hypotheses explored.
What you see: Keep the full search. What it represents: The trial sheet represents comparisons that must remain visible in interpretation.
What you see: Selected finding. What it represents: The singled-out card represents a result that needs selection-aware error control or confirmation.
Guided practice: After exploring 50 metrics, one looks promising. How can you investigate honestly?
Hint: Separate discovery from confirmation.
Compare your reasoning: Label the result exploratory, disclose the search, and pre-specify a confirmatory evaluation on new suitable data or use a valid multiplicity procedure.
Predict: What happens to the overall false-alarm chance as you add independent true-null tests?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: It rises. Bonferroni uses the family’s alpha divided by its test count to bound any false rejection, but stricter individual thresholds can reduce power.
Optional math: P(at least one false rejection) = 1 - (1 - alpha)^m; Bonferroni cutoff = family alpha / m
m: Pre-defined number of tests; the first formula assumes independence and true nulls.
alpha: Individual test level in the first formula; family-level budget in the second.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: Only the test I publish counts toward the error risk.
Correction: Selection across attempted comparisons is part of the procedure. Keep a record of the search and use appropriate confirmation.
Check your understanding: Does Bonferroni require independent individual tests to bound family-wise error?
Yes; without independence it has no error bound. - Not yet. The independence assumption belongs to the displayed exact unadjusted formula, not the Bonferroni bound.
No, but the individual tests must be valid and the family defined appropriately. - Yes. The independence assumption belongs to the displayed exact unadjusted formula, not the Bonferroni bound.
Use it with purpose: How many opportunities to find a winning result are hidden in your current evaluation loop?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-multiple-testing-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-multiple-testing-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-multiple-testing-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-multiple-testing-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm

LESSON: Randomized Experiments: Compare Like With Like
URL: https://www.openagentschool.org/ai-ml-foundations/randomized-experiments
Also called: A/B testing, random assignment, causal inference, confounding, controlled experiment
Summary: Distinguish a changed system from a changed mix of people or tasks.
In everyday words: A randomized experiment assigns eligible units to conditions by chance under a planned design. This helps separate treatment effects from pre-existing differences by balancing other factors in expectation. Random assignment is not random sampling, and a trial does not automatically generalize beyond the studied population.
Why it matters: Comparing last week’s difficult tasks with this week’s easy ones can credit the model for a workload change.
Scenario - Before changing the system: Two identical versions succeed on 90% of easy tasks and 50% of hard tasks. A receives 80% easy cases and scores 82%; B receives half easy cases and scores 70%. The apparent 12-point gain is entirely task mix.
Analogy - A fair comparison needs comparable starting conditions: To compare two routes, account for who traveled, when and under which conditions—not just which arrival time was lower.
Where the analogy stops: Randomization balances in expectation; small samples can be imbalanced. Interference, noncompliance, missing outcomes and unsafe treatments still matter.
AI-generated analogy illustration: https://www.openagentschool.org/images/foundations/randomized-experiments.webp
Picture description: A mixed collection of containers, an opaque draw bag with blank tokens and two equal testing trays awaiting assignment.
Picture caption: A fair comparison plans assignment before outcomes, while keeping uncertainty and practical safeguards visible.
What you see: Eligible units. What it represents: The mixed collection represents the cases eligible for the planned comparison.
What you see: Chance assignment. What it represents: The draw bag represents a known randomization process, not choosing easy cases for a favorite.
What you see: Compare conditions. What it represents: The equal trays represent study conditions balanced in expectation, not guaranteed identical samples.
Guided practice: The same user would see both variants and carry information between them. What design issue appears?
Hint: An outcome in one condition can affect the other.
Compare your reasoning: Consider user-level or cluster-level assignment, carryover and interference, and analyze uncertainty at the appropriate level. Do not assume independent calls.
Predict: When A’s task mix reaches 50/50, what remains of its apparent advantage?
Try: Change one input in this fictional model. Compare the calculated values, then check the assumptions before applying the idea elsewhere.
Explain: It disappears in this deterministic example. Real randomized comparisons still have sampling uncertainty and need appropriate analysis.
Optional math: observed difference = treatment contribution + imbalance + sampling variation (conceptual decomposition)
randomization unit: The independently assigned user, task or cluster; choose it to avoid cross-condition interference.
These formulas apply to the stated teaching model. Read its assumptions; a formula does not repair biased data, dependence or an unsuitable design.
Common misconception: A larger before/after score difference proves the update caused it.
Correction: Population shifts, other changes and measurement differences may explain it. Use a defensible design and causal assumptions.
Check your understanding: What does random assignment guarantee?
Exactly identical groups and automatic generalization to all users. - Not yet. Randomization supports causal comparison under assumptions; it does not remove uncertainty or establish representative sampling.
A known assignment process that balances confounders in expectation, not perfect balance in every sample. - Yes. Randomization supports causal comparison under assumptions; it does not remove uncertainty or establish representative sampling.
Use it with purpose: Which safe, ethical comparison could distinguish a model change from a workload change?
Guidance: Name the target population, what was observed, which assumptions are credible, and what evidence is still needed. Keep effect size, uncertainty and consequences visible rather than turning one threshold into a deployment rule.
Quiz practice: https://www.openagentschool.org/quiz?qid=foundation-randomized-experiments-quiz
socratic practice: https://www.openagentschool.org/study-mode?qid=foundation-randomized-experiments-socratic
scenario practice: https://www.openagentschool.org/study-mode?qid=foundation-randomized-experiments-scenario
debug practice: https://www.openagentschool.org/study-mode?qid=foundation-randomized-experiments-debug
Assessment feedback is formative, not proof of mastery. Existing Quiz and Study Mode access and optional AI-provider settings apply.
Where to go next:
Agent Evaluation (concept): https://www.openagentschool.org/concepts/agent-evaluation - Use this statistical distinction to design representative evaluations and interpret evidence before changing an agent.
References:
OpenIntro Statistics - open textbook and learning resources: https://www.openintro.org/book/os/
NIST - statistical assumptions and normal-model sample-size planning: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm