What Is a Loss Function? Definition and Examples

A loss function turns prediction error into the signal training optimizes. It gives a model a concrete way to score its own output against a target, then gives the training process something to reduce.

That sounds dry because the math is dry. The consequences are not. A loss function decides what counts as an error during training. It shapes what the model gets rewarded for learning, what it ignores, and where it may develop strange habits.

PE Collective publishes weekly data from 22,000+ job postings, a useful reminder that ML work increasingly sits inside ordinary product teams, where training decisions show up later as product behavior.

TLDR

A loss function measures how wrong a model’s output is relative to its training target. Training adjusts model parameters to reduce that score. The objective influences behavior, but data, architecture, and training procedure decide how far that influence reaches.

What a Loss Function Measures

A model makes a prediction. The training data provides a target. The loss function compares the two and returns a value representing the penalty for that prediction.

Lower loss means the model performed better according to the rule you chose. Higher loss means it performed worse.

The important phrase is “according to the rule you chose.” A model can be excellent at reducing its loss while still producing answers a person finds unhelpful, misleading, or unsafe. The objective only measures the target it was designed to measure.

For a classifier, the target may be a category label. For a language model, the target is often the next token in a sequence. For a recommendation system, it may be whether someone clicked, watched, purchased, or returned. Each setting needs a loss function that converts a different kind of mismatch into a learnable signal.

The model does not inspect its error, feel embarrassed, and revise its answer. An optimization process calculates how each parameter contributed to the loss, then adjusts those parameters in the direction expected to reduce it on future examples. Repeat that process over a large dataset and the parameters begin to encode patterns that help with the chosen task.

This is why loss sits at the center of training. It is the scoreboard.

It is also why arguments about model behavior often become arguments about objectives. If a system is rewarded for predicting likely next tokens, it learns a different set of habits than one rewarded for matching human preferences. If the score treats a confident wrong answer gently, the model has little reason to avoid that particular failure.

PE Collective’s newsletter identifies 2,700+ subscribers, including readers who may want the related classifier glossary when prediction targets are discrete labels rather than text.

Loss Function Notation and a Simple Example

Loss function notation often looks like this:

\[ L(y, \hat{y}) \]

Here, \(y\) is the correct target and \(\hat{y}\) is the model’s prediction. The function \(L\) returns a penalty.

During training, the model has parameters, commonly written as \(\theta\). The prediction depends on those parameters:

\[ \hat{y} = f_{\theta}(x) \]

The training objective is to find parameter values that reduce loss across the dataset:

\[ \min_{\theta} L(y, f_{\theta}(x)) \]

In practice, training usually evaluates loss across many examples and uses the average. The notation can get busy quickly, but the idea remains plain: generate an output, score it, adjust the model, repeat.

Consider a simple classification task. A model must identify whether an image belongs to one category or another. It sees an image and assigns a high probability to the wrong category. That prediction should receive a larger penalty than a prediction that assigns only a slight edge to the wrong category.

Why? Because probability matters. A model that is confidently wrong has learned something more problematic than a model that is uncertain.

Cross-entropy handles this well. It penalizes the model according to the probability assigned to the correct answer. If the correct category receives a high probability, loss is low. If the correct category receives a tiny probability, loss rises sharply.

That gives the model a useful incentive: put more probability mass on the correct target.

For language models, the mechanism is similar. Given a sequence of text, the model predicts a distribution over possible next tokens. The training data tells it which token came next. Cross-entropy penalizes the model when it assigned too little probability to that observed token.

This does not mean the observed token was the most truthful, ethical, or useful continuation. It means the token appeared in the training example. That distinction hangs over much of modern model behavior.

PE Collective invites 1,300+ prompt engineers to its newsletter, and the inference glossary explains the later stage where a trained model generates outputs from the patterns this objective helped shape.

Common Objectives in Model Training

Large language model training uses more than one objective because the job changes across stages.

Pre-training usually relies on cross-entropy for next-token prediction. The model reads text, predicts what comes next, and receives a penalty based on the probability it gave the observed token. This is a broad learning objective. It teaches language patterns, facts, styles, code structures, and plenty of unwanted material too.

Later training stages try to steer the model toward outputs people prefer.

Objective	Training stage	What it scores	Typical use
Cross-entropy	Pre-training	Probability assigned to observed next tokens	Learning broad language and pattern completion
Reward-model objective	Preference-based training	A learned score for outputs people rate more highly	Training toward human judgments
Preference objective	Direct preference optimization	Which response is preferred within a comparison	Moving a model toward selected responses

A reward model converts human judgments into a score. People compare outputs or rate them. The reward model learns patterns in those judgments, then assigns a score to new outputs. Reinforcement learning from human feedback, often shortened to RLHF, uses that score to encourage responses that rate well.

That process often includes a penalty for moving too far from the earlier model. Without a constraint, the system can chase the reward model too aggressively and lose useful capabilities or develop brittle behavior. The training objective therefore balances preference scores with a cost for drifting from the original distribution.

Direct preference optimization, often called DPO, uses preference comparisons more directly. It adjusts the model toward responses selected over alternatives without requiring the same reward-model-and-reinforcement-learning setup.

These objectives aim at related outcomes, but they are not interchangeable. Cross-entropy asks whether the model predicted the training text. A reward model asks whether an output resembles outputs people preferred. A preference objective asks whether one candidate should become more likely than another.

The difference matters because every objective has blind spots.

Cross-entropy can encourage fluent imitation of text without distinguishing fact from plausible fiction. A reward model can reward answers that sound polished, agreeable, or confidently helpful when raters lack the time or expertise to verify them. Preference data can import the tastes, assumptions, and shortcuts of the people making the comparisons.

The objective can also become a source of failure when it measures a proxy rather than the thing a product team wants. Clicks are not satisfaction. Completion is not comprehension. A response chosen in a short comparison is not always the response that holds up in a complicated real-world task.

That is the familiar problem of optimizing the metric while missing the goal. Models are unusually good at exposing it because they will pursue whatever the training signal consistently rewards.

How Optimization Changes a Model

Loss reduction happens through optimization. The model starts with parameter values that produce poor predictions. Training calculates how changing those values would affect loss, then applies many small updates.

The point is not to memorize every training example. At least, that is not the point when training works well. The model should learn patterns that generalize to new inputs.

A model can still reduce training loss by memorizing too much. When it performs well on material it has seen but poorly on new examples, that is overfitting. The loss may look good inside training while the product behaves badly outside it.

This is why teams evaluate models on held-out data and real tasks. Training loss is valuable, but it is not the only evidence worth looking at. A falling loss curve says the system is getting better at its stated objective. It does not settle whether the objective matches the work users need done.

The page’s current article markup includes Article value 99, while the 2026 date value on that same page keeps the definition tied to a current reference point rather than a static textbook entry.

What Loss Does Not Explain by Itself

Loss functions help explain model behavior. They do not fully determine it.

Training data decides what patterns are available to learn. Architecture affects what the model can represent and how it processes inputs. The optimization procedure affects which solution training reaches. Compute, data filtering, tokenization, regularization, sampling, and evaluation all change the final result.

Two models can use the same loss function and behave differently because they saw different data or followed different training procedures. A small change in preference data can alter what behavior becomes common. A change in decoding settings can alter outputs without changing the trained loss function at all.

The objective is still worth studying because it defines the direction of travel. It tells you what the training process is trying to improve.

It cannot tell you every place the model will end up.

This is especially important when diagnosing failures. If a language model hallucinates, cross-entropy provides part of the explanation: it rewards likely continuations, not a direct test of truth. But the full diagnosis also requires looking at the data, retrieval setup, prompting, evaluation task, and inference settings.

If a model becomes excessively cautious after alignment training, the preference objective may be involved. Yet the specific behavior could also come from the examples used, the reward model’s biases, the strength of the optimization, or product-layer rules applied after training.

Treat loss as one causal layer, not the entire story.

That framing gives you a better way to read claims about model improvements. Ask what objective changed. Ask what data supplied the signal. Ask how the team evaluated the result. Then ask whether the measured improvement resembles the behavior users will encounter.

A lower loss value can be good news. It can also be a model getting more efficient at satisfying a narrow proxy. The gap between those outcomes is where much of ML engineering lives.

Key Takeaways

• A loss function converts prediction error into the score a training process tries to reduce.
• Cross-entropy rewards probability assigned to observed targets, including next tokens in language-model training.
• Reward-model and preference objectives use human judgments to steer models after pre-training.
• Loss influences behavior, while data, architecture, optimization, and inference settings shape the final system.
• A lower loss only proves improvement against the chosen objective.

Sources

• PE Collective weekly job-posting data, 22,000+ job postings.
• PE Collective newsletter audience, 2,700+ subscribers.
• PE Collective prompt-engineer newsletter invitation, 1,300+ prompt engineers.