You used an AI tool today. For example, ChatGPT may have drafted an email, or Claude helped debug a function. Perhaps you looked up something you would not have asked a colleague out loud. These tools sit in almost every workflow now — and yet, most people using them cannot explain what happens after they hit send.
Tokens. Context windows. Hallucinations. Knowledge cutoffs. Agents. The terms show up in standups, headlines, and LinkedIn threads. People repeat them, but few can define them. That gap matters, because the better you understand the machinery, the better you work with these tools — and consequently, the less likely you are to trust them blindly.
Therefore, this article breaks down seven fundamentals that explain how modern AI actually works. No math. No vendor hype. Just the concepts worth knowing if you use ChatGPT, Claude, or any LLM in your day-to-day work.
The Gap Between Using AI and Understanding It
The tools are already here — access isn’t the problem. Instead, what trips people up is that AI feels like it understands you, so it’s easy to assume it reasons, remembers, and verifies the way a person would.
It does not.
Behind the chat interface sits a system trained on text, predicting responses one piece at a time, working inside hard limits on memory and knowledge. When those limits show up — a forgotten budget in a long chat, a fabricated API method, a confident answer about last week’s news — they look like bugs. However, they’re not. That’s just how the thing works.
In short, that’s what this article is about.
How an LLM Is Built — and What It Actually Does
Before you can use an LLM well, you need to know what it is. Specifically, three ideas cover the foundation: where its knowledge comes from, what it optimizes for, and how it reads your input.
Training Data: How an AI Is Born
Every large language model starts with reading — not thinking.
Before a model answers a single prompt, it trains on an enormous volume of text drawn from the public internet: web pages, books, code repositories, Wikipedia, forums, scientific papers. As a result, frontier models ingest trillions of words — more than any human could read across many lifetimes.
Moreover, training at this scale is expensive. A frontier model may train for hundreds of days across thousands of specialized chips, with costs running north of $100 million. In fact, some of the largest runs reportedly approach a billion dollars.
So when someone says an AI “knows” something, this is the source: it read it once during training. That fact alone explains much of what follows — including why models can sound authoritative about things they never truly verified.
The LLM: A Next-Word Prediction Machine
After training, you get a Large Language Model — an LLM. Despite the branding, its core job is simple: predict the next word.
For example, ask “What is the capital of France?” and the model does not query a database or search the web. Instead, it runs a loop: given everything I have read, what word most likely comes next? The answer emerges token by token — The. Capital. Of. France. Is. Paris. — with the model weighing tens of thousands of possible continuations at each step.
If you take away one thing: the model optimizes for likelihood, not truth. Usually those line up — which is why the output feels intelligent. However, when they don’t, the model still sounds sure of itself. As a result, that gap is where hallucinations come from.
Tokens: How AI Reads Your Prompts
The model does not see words or letters the way you do. Rather, it sees tokens — chunks of text roughly four characters long, or about three-quarters of a word.
Short, common words often map to a single token. Meanwhile, longer words get split. “Pear” is one token. “Hamburger” becomes three: ham, bur, ger. Similarly, take the sentence “You miss 100% of the shots you don’t take.” — nine words to you, eleven tokens to the model.
Tokens matter for two practical reasons. First, they are the billing unit — input, output, and context limits are all measured in tokens. Second, they explain why models fail at letter-level tasks. For instance, counting characters, spelling tricks, and questions like “how many R’s are in strawberry?” trip models up because they never see individual letters.
Additionally, if you prompt in a language other than English, watch your token count. In my experience, the same sentence in Hindi often costs two to three times more tokens than the English version — same meaning, bigger bill, context window filling up faster.
The Limits That Shape Every Response
An LLM does not operate with open-ended memory or up-to-the-minute knowledge. Instead, three constraints — context window, knowledge cutoff, and hallucination risk — show up in almost every serious use case.
The Context Window: AI’s Short-Term Memory
The context window is how much text a model can hold in working memory at once, measured in tokens. Furthermore, your messages, the model’s replies, and any pasted documents all share this single window.
When the window fills, the oldest content drops off to make room for new input. For example, imagine planning a trip and mentioning a $3,000 budget early in the conversation. After enough back-and-forth about flights, hotels, and itineraries, that detail may simply be gone. Ask “what was my budget again?” and the model is not being careless — rather, the information left its memory.
Modern models offer large windows — from 128,000 tokens to a million or more. A 200,000-token window holds roughly 150,000 words, about two novels. That sounds generous, but then you paste in a repo, attach three error logs, and spend an hour going back and forth on a single bug. Suddenly you’re out of room.
Still, the window is never infinite. My fix is boring but it works: new topic, new chat. I learned that after one too many sessions where the model confidently “forgot” a constraint I’d stated in message two.
Knowledge Cutoff: Why AI Is Frozen in Time
Every model has a knowledge cutoff — the date its training data ends. Therefore, it knows the world up to that point and nothing beyond it.
You might push back: “But my ChatGPT knew about something from last week.” In that case, it likely did — but through web search, not through training.
Turn web search off and ask about the latest iPhone. A model frozen at its cutoff may confidently answer iPhone 16. That reflects what the model actually knows. Now turn search on, ask again, and it pauses, retrieves current information, and returns iPhone 17. The answer improves — but the model did not know it. Instead, it looked it up. Web search is a patch on a frozen brain.
The blind spot is often a year or wider. Meanwhile, tools paper over it with retrieval and search. Therefore, keep that in mind when an AI states something recent with total confidence — and you do not see it searching.
Hallucinations: When AI Confidently Gets It Wrong
When a model lacks information and cannot retrieve it, it rarely says “I don’t know.” Instead, it often generates a plausible answer and presents it as fact. That is a hallucination.
I run into this weekly. For example, ask “How do I remove duplicate rows from a DataFrame in pandas?” and the model may return clean code using df.remove_duplicates(). It looks legit. However, run it and you get an AttributeError — that method doesn’t exist. The real one is df.drop_duplicates(). The model picked a name that sounded right. I’ve shipped hallucinated APIs before I caught myself. Now I run the code first. Every time.
Furthermore, it’s not just a dev problem. A lawyer in New York used ChatGPT to draft a legal brief and cited six court cases that were entirely made up. He filed them and was sanctioned by a federal judge. In other words, fluent doesn’t mean verified.
My rule: use it to move faster, but check anything that matters.
From Chatbots to Agents — and What Comes Next
So far, everything described produces text. You ask a question; the model answers. That is a chatbot.
By contrast, an agent adds two capabilities on top of the same prediction engine: tools and a loop. Instead of only generating instructions, it can take action — search a codebase, edit a file, run tests, open a pull request.
Consider the same request given to both: “The export button on my app is broken — fix it.”
A chatbot returns a numbered list of steps for you to follow. Meanwhile, an agent searches the repository, locates the relevant file, identifies the bug, applies a fix, runs the test suite, and opens a PR. You still review the result — but the model did the legwork.
That’s the shift everyone keeps talking about — AI that doesn’t just reply, but actually does things. However, agents still run on the same engine underneath. Token limits, knowledge cutoffs, hallucinations — none of that goes away. In fact, a wrong API name in a chat is annoying, whereas a wrong file edit from an agent is a different story.
MCP (Model Context Protocol) is the piece I’d cover next. Specifically, it’s how agents plug into your tools and data — a standard port between AI and the rest of your stack. Important enough that I’m writing a separate post on it, along with RAG, reasoning models, and fine-tuning.
What Changes When You Understand the Machinery
None of this is magic. Rather, it’s a model that read the internet once, predicts text token by token, forgets old messages when the window fills up, can’t know anything past its training date without search, and sometimes makes things up instead of admitting uncertainty. Additionally, agents add action on top of that same stack.
Once you see it that way, the weird behavior stops surprising you. As a result, prompting changes — you verify before you ship, start a fresh chat instead of wondering why the model “forgot” your requirements. The tools are still worth using. Ultimately, you just stop treating them like they think.
Resources and Next Steps
AI flashcard deck. I turned these fundamentals — plus prompt engineering, MCP, and more — into a flashcard deck for fast revision and interview prep. Grab it on Topmate →
Book a 1:1 session. I run technical mock interviews, resume reviews, and career strategy sessions on Topmate — 590+ sessions and counting. Book a slot →
More from me: all my resources · GitHub
Something click — or something you’d explain differently? Drop it in the comments. I read them.
Video Explanation
