Chatbots that write essays, summarize reports and answer questions have moved from novelty to everyday tool in a short time. They are powered by large language models, usually shortened to LLMs. The tools feel conversational, and that is part of the problem: a fluent answer sounds like an informed one, whether or not it is. A basic picture of how these systems work makes their strengths and failures much easier to anticipate.
How a language model works
An LLM is a statistical system trained on very large amounts of text. During training, it learns to predict the next piece of text, typically a word fragment called a token, given everything before it. Repeated across billions of examples, that simple objective teaches the model grammar, facts, styles and patterns of reasoning as they appear in its training data. The knowledge is stored in numerical parameters, not in a database of statements that it looks up.
After this first stage, developers usually refine the model further. They train it on examples of helpful conversations and use human feedback to prefer answers that people rate as better, safer or more useful. That process shapes the assistant’s tone and its willingness to refuse certain requests.
When you type a prompt, the model produces a response one token at a time, each choice drawn from a probability distribution. The same question can yield different wording on different tries.
What they do well
LLMs tend to do well on tasks where the answer is mostly about language and pattern:
- Drafting and editing. First drafts, rewrites for tone or length and grammar fixes.
- Summarizing a document you provide, particularly when the source is in front of the model.
- Translating between languages and reformatting information, such as turning notes into a table.
- Explaining familiar concepts at different levels.
- Writing and explaining code, especially common patterns, where results can be tested.
- Brainstorming, since the goal is a range of ideas, not one correct answer.
The common thread is that a person can quickly review the result, and an occasional error costs little.
Why do they make things up?
The technical term is “hallucination”: output that is fluent and confident but false. It happens because the model is built to produce plausible text, not to verify facts. If it lacks reliable information, it can still generate something that looks right, including invented quotations, statistics, court cases and citations.
A widely reported example involved lawyers who submitted a court filing with case citations produced by a chatbot. The cases did not exist, and a federal judge sanctioned the attorneys in 2023. The episode shows the risk of treating generated text as a source.
Other recurring limits include:
- A knowledge cutoff. The model reflects its training data up to a certain date. Unless it is connected to a search tool or given documents, it may not know about recent events.
- Weakness with exact reasoning. Arithmetic, counting and multi-step logic can fail without tools such as a calculator or code interpreter.
- Sensitivity to wording. Small changes in a prompt can change the answer.
- Bias. Models reflect patterns in their training data, including stereotypes.
- Opacity. It is hard to explain why a model produced a particular answer, and the explanation it offers may not describe what actually happened inside.
Search, retrieval and fact-checking
Some products pair a model with a search tool or a set of documents, a design often called retrieval. This can help, because the model can quote from sources it has just retrieved, and many tools show links. Retrieval reduces some errors but does not remove them: the model can misread a source, cite it for a claim it does not make, or retrieve a poor source. The user still has to open the link.
Using the answers wisely
- Match trust to stakes. For brainstorming a title, a quick answer is fine. For medical, legal, financial or news questions, treat the output as a starting point and confirm it with authoritative sources.
- Ask for sources and then check them. Verify that each citation exists and says what the answer claims.
- Provide the material. Pasting the relevant text and asking for a summary is more reliable than asking from memory.
- Mind privacy. Depending on the service and settings, prompts may be stored or used to improve products. Avoid entering sensitive personal or confidential information unless you understand the policy.
- Keep a human in the loop for anything consequential.
The National Institute of Standards and Technology has published an AI Risk Management Framework that organizations use to think through these issues, including accuracy, bias and security.
What does this mean for what you read online?
Generated text and images are now common in feeds and search results, often unlabeled. A polished surface is no longer evidence of care or truth. The habits in how to check a viral claim before you share it apply directly: look for the source, find independent confirmation and be cautious of material with no clear author. It also connects to the skill of working out what kind of article you are reading, because a text can imitate the style of reporting without any reporting behind it.
Sources: National Institute of Standards and Technology; Stanford Institute for Human-Centered Artificial Intelligence; U.S. District Court for the Southern District of New York; Federal Trade Commission
