How LLMs Actually Work

What Is an LLM?

A Large Language Model (LLM) is a neural network trained on trillions of tokens of text. It predicts the next token in a sequence given the preceding context. That is all it does — but this simple mechanism produces remarkably capable language understanding and generation.

For SRE teams, LLMs are useful because they can reason over natural-language runbooks, summarize incidents, generate queries, and assist with troubleshooting. But they are probabilistic — they can be wrong, and they do not know what they do not know.

Tokenization

Models do not read text character by character. They break input into tokens — pieces of words. Common words are one token. Rare words or technical terms may be split into multiple tokens. This directly affects cost and speed.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("kubectl get pods --namespace production")

print(f"Token count: {len(tokens)}")
print(f"Tokens: {[enc.decode([t]) for t in tokens]}")

Rough rule of thumb: 1 token is approximately 4 characters of English text, or about 0.75 words. A typical runbook page is 500 to 1000 tokens.

Context Window

The context window is the maximum number of tokens a model can process in a single request — input tokens plus output tokens combined. GPT-4 supports 8,192 tokens. GPT-4 Turbo supports 128,000. Claude 3 supports 200,000.

Temperature & Randomness

Temperature controls how creative or deterministic the model is. Lower temperature means more predictable, consistent output. Higher temperature means more varied, surprising output.

import openai

# Deterministic: same input, same output
response = openai.ChatCompletion.create(
    model="gpt-4",
    temperature=0.0,
    messages=[
        {"role": "system", "content": "You are an SRE assistant."},
        {"role": "user", "content": "What causes high p99 latency?"}
    ]
)

# Creative: varied output each time
response = openai.ChatCompletion.create(
    model="gpt-4",
    temperature=0.7,
    messages=[
        {"role": "system", "content": "You are an SRE assistant."},
        {"role": "user", "content": "Suggest a runbook title for Redis outages."}
    ]
)

Why Models Hallucinate

Hallucination is when a model confidently states something that is factually wrong. It happens because the model optimizes for plausible-sounding text, not truth. It fills gaps with patterns from training data, even when those patterns are incorrect for your specific infrastructure.

• Training data is outdated — your infrastructure changed since the model was trained.
• The model has never seen your specific Kubernetes cluster or deployment patterns.
• It conflates similar-sounding services, endpoints, or procedures.
• It generates plausible but fabricated commands, metrics, or log entries.

Choosing the Right Model

Not every task needs the most powerful model. A simple classification task (is this alert P1 or P2?) can use a smaller, cheaper, faster model. Complex root-cause analysis benefits from a larger model with a bigger context window.