# Small language models: *when smaller can be better*#

Large language models have received most of the attention in the recent wave of AI. But many of the things organisations want AI to do are narrower and more predictable, and not every task requires a large model.

Large language models have understandably received most of the attention in the recent wave of AI. Their ability to work across a wide range of tasks—from writing and coding to analysis and reasoning—has made them the default choice for many AI applications.

But not every task requires a large model.

Many of the things we want AI to do inside organisations are narrower and more predictable: classify a customer request, extract information from a document, route a query, generate a response within a defined context, or perform a repetitive domain-specific task.

This is where **Small Language Models, or SLMs**, become interesting.

SLMs are built on many of the same fundamental ideas as larger language models, but with fewer parameters and, consequently, different trade-offs in capability, cost, speed and deployment. They can also be combined with larger models, allowing different parts of a workflow to be handled by models appropriate to the work.

So the interesting question is not whether **small models are better than large models**.

It is:

**When is a smaller model enough—and when does the work genuinely require a larger one?**

This article explores that question: what makes a language model "small," how SLMs are created and adapted, where they can be useful, and how organisations can decide which model belongs where.

## What Exactly Is a Small Language Model?#

A Small Language Model, or **SLM**, is fundamentally the same kind of technology as a Large Language Model. Both can be built using transformer-based architectures, both learn from large collections of text, and both learn by adjusting parameters during training.

The main difference is **scale**.

A language model consists of an architecture and a large collection of learned parameters. These parameters are essentially numbers whose values are adjusted during training. Together, they capture patterns the model has learned from its training data and influence what it predicts when given an input.

The approximate number of parameters is determined when the model architecture is designed. A model might have hundreds of millions, a few billion, tens of billions, or hundreds of billions of parameters. Training changes the **values of those parameters**, rather than simply adding more parameters to make the model larger.

There is no official parameter limit that defines an SLM. As a practical guide, however, **models with roughly 1–7 billion parameters sit comfortably in what is commonly considered SLM territory**, while the label is sometimes extended to models around 10 billion parameters or even beyond. The boundary is not fixed—the term describes a class of relatively compact language models rather than a formal specification.

So, as a simple mental model:

**1–7B parameters → comfortably SLM territory** **Around 7–10B and beyond → the terminology becomes less consistent** **Tens or hundreds of billions → generally considered large models**

That leads to an important point: **SLMs and LLMs are not two fundamentally different technologies. They sit on a spectrum.**

The more useful question, therefore, is not simply whether a model is small or large, but **how much model capacity the task actually needs**.

## How Do We Decide How Large a Model Should Be?#

Once we know that language models can be built at very different sizes, an obvious question follows: **how does someone decide whether a model should have 1 billion, 7 billion or 70 billion parameters?**

It is not simply a matter of choosing a number. Model developers have to balance three things: **the amount of training data available, the size of the model, and the compute available to train it.**

A larger model has more parameters and therefore more capacity to learn. But increasing the number of parameters without providing enough training data does not necessarily make good use of that additional capacity. Similarly, having enormous amounts of data but a model that is too small can limit how much the model can learn from it.

Research on **scaling laws** has tried to understand this relationship.

One influential example is DeepMind's 2022 **Chinchilla** research. The researchers trained hundreds of models of different sizes on different amounts of data and studied how model size and training data should be balanced for a given training-compute budget.

A widely remembered rule of thumb from this work is approximately:

**20 training tokens for every parameter.**

So, as a very rough illustration, **20 billion training tokens would correspond to a model of around 1 billion parameters** under that compute-optimal rule of thumb.

This is not a fixed formula or an industry standard. Modern models may be trained with substantially more data per parameter, particularly when developers are willing to spend more on training to create a smaller model that will later be cheaper or faster to run.

The important idea is the relationship:

**Training data ↔ Model size ↔ Compute**

Scaling research gives model builders a sensible starting point. They can then train and evaluate different configurations around that starting point to understand whether increasing or decreasing model size actually improves the outcome.

So deciding to build a small model is not simply about choosing fewer parameters. **It is about finding an appropriate model size for the available data, training budget and ultimately the job the model is expected to perform.**

## Why Use a Small Model When Large Models Are So Easily Available?#

If large language models are more capable and can be accessed easily through an API, an obvious question is: **why use a smaller model at all?**

For many applications, using a large hosted model is the simplest choice. There is no model infrastructure to manage, the model can handle a wide variety of requests, and an application can often be built quickly without first determining exactly how difficult every request will be.

But greater capability is not always necessary.

Consider an application that needs to classify thousands of customer messages into a fixed set of categories, extract a few fields from documents, route requests to the right team, or generate text within a tightly defined format. If a smaller model can perform that task reliably, using a much larger model for every request may provide little additional value.

This is where SLMs can offer practical advantages. They generally require less compute to run, which can translate into **lower inference cost and faster response times**. Their smaller size can also make it practical to run them on an organisation's own infrastructure or, for sufficiently small models, directly on devices. This can be useful where privacy, connectivity or control over deployment matters.

The economics, however, are not as simple as **"SLMs are cheaper."**

At low volumes, calling a hosted large-model API may be considerably simpler and economically sensible. Running an SLM yourself introduces its own costs: infrastructure, engineering, evaluation, monitoring and maintenance. At high volumes, however, repeatedly sending a narrow and predictable task to a large model can become unnecessary expense if a smaller model can achieve the required quality.

This leads to a useful principle:

**The objective is not to use the smallest model possible. It is to use the smallest model that can reliably meet the requirements of the task.**

And that immediately raises the next question: **what kinds of tasks are actually good candidates for an SLM?**

## Where Are Small Language Models Useful?#

SLMs are particularly interesting when the work is **well-defined, repetitive and does not require broad, open-ended reasoning**.

Consider a customer-support operation receiving thousands of messages every day. Many tasks are relatively structured. The organisation may need to identify whether a message concerns billing, delivery, cancellation or a product issue; extract an order number or product name; determine which team should receive the request; or generate a response within an established format.

These are examples where a smaller model may be sufficient.

The same pattern appears across many business processes. An SLM might be used to **classify documents, extract structured information from text, route requests, detect intent, summarise relatively predictable content, or generate constrained responses**.

SLMs can also be useful when the model needs to run **close to where the work happens**. Their smaller size can make deployment possible on devices with limited computing resources.

Consider a **drone operating in an area with unreliable connectivity**. Rather than continuously sending information to a large model in the cloud, a sufficiently small model could run locally to interpret instructions or help reason about observations. Similarly, smaller models can potentially run on **phones, vehicles, robots, cameras and industrial equipment**, depending on the model and available hardware.

This can matter for more than cost. Local processing can reduce latency, continue working when connectivity is limited, and keep certain information on the device.

So there are broadly two kinds of situations where SLMs become interesting: **well-defined, high-volume business tasks**, where a large model may be unnecessary, and **resource-constrained or edge environments**, where running a large model may simply be impractical.

But the boundary is not simply **routine task = SLM, complex task = LLM**. A seemingly simple task can contain difficult exceptions. A customer-support request might begin as a straightforward delivery query but require reasoning across conflicting information from an order system, courier records and previous conversations.

This is why the decision ultimately needs to be empirical. Take representative examples of the actual work and test whether the smaller model can perform them at the required level of **quality and reliability**. If it can, there may be little reason to use a larger model for that task. If it cannot, a more capable model may be justified.

And this leads to another important practical question: **if an SLM looks suitable for your use case, do you need to train one yourself?**

## Do You Need to Train Your Own SLM?#

Identifying a task that could be handled by a small model does **not** mean that an organisation needs to build and train a language model from scratch.

There are already many pretrained small language models available. In most cases, the practical starting point is to choose an existing model that appears suitable for the task and test it with representative examples from the actual work.

Often, the model may work well with **prompting alone**. If the model needs access to information it was not trained on—company policies, product documentation, customer information or other changing knowledge—it can be given that information at runtime, often through techniques such as **Retrieval-Augmented Generation (RAG)**.

If the requirement is more specialised, **fine-tuning** becomes another option. Instead of creating a new language model, we take an existing pretrained SLM and train it further on examples relevant to the task. For example, a company processing a large volume of specialised documents could fine-tune a small model using examples of the inputs and outputs it expects.

This is very different from training a language model from scratch. The original model developer may have trained the model on billions or trillions of tokens using substantial computing resources. Fine-tuning starts with what that model has already learned and adapts its behaviour to a more specific purpose.

A useful progression is therefore:

**Existing SLM → Prompt it → Add relevant knowledge when needed → Fine-tune if necessary**

Training a new base model from scratch sits much further along that spectrum and is unlikely to be the starting point for most organisations.

The important question is therefore not **"How do we build our own SLM?"** but **"Can an existing small model reliably perform this task, and how much adaptation does it need?"**

And there is another possibility: the SLM does not have to handle the entire task. **A small model and a large model can work together within the same application.**

## SLM or LLM? Sometimes the Answer Is Both#

The choice between a small and a large language model does not always have to be either-or. Within the same application, **different parts of the work may require different levels of model capability**.

Consider a customer-support application for an e-commerce company receiving thousands of requests every day.

A customer writes:

*"I ordered two shirts. One arrived, but the other hasn't. Can you check what happened?"*

Much of the work required to process this request is relatively predictable. A small language model could identify the request as a **delivery issue**, extract the relevant order information, classify the intent and route the request to the appropriate workflow. The application can then retrieve the order and delivery information from the company's systems.

Suppose the records show that the second shirt is still in transit and expected tomorrow. This is a routine case. The SLM could generate an appropriate response using the retrieved information and the company's response guidelines. There may be little reason to involve a much larger model.

But now consider a different case. The courier system says the entire order was delivered. The customer says only one item arrived. The payment system shows two charges, while another system indicates that a refund may already have been initiated. There are also previous conversations with the customer that may be relevant.

This is no longer a simple classification or response-generation problem. The application may need to **reason across conflicting information, understand the history of the case and determine an appropriate course of action**. This is where the request could be escalated to a more capable large language model.

The workflow might therefore look something like:

**Customer request → SLM classifies and extracts → Business systems retrieve information → Routine case stays with SLM → Complex or uncertain case goes to LLM**

The routing itself does not necessarily require another large model. It could be based on deterministic rules, confidence thresholds, a lightweight classifier or another small model.

This kind of **multi-model architecture** allows an organisation to use smaller models for high-volume, predictable work while reserving larger models for situations where their additional reasoning capability is valuable.

The objective is therefore not to decide that an application is an **"SLM application" or an "LLM application."** It is to decide **which model is appropriate for each part of the work.**

## How Should a Business Decide?#

Choosing between an SLM and an LLM should not begin with model size. It should begin with the **requirements of the work**.

A useful way to make the decision is to evaluate candidate models across a few practical dimensions: **accuracy and reliability, speed, cost, privacy, deployment constraints and the complexity of the task.**

**Accuracy and reliability** should come first. A smaller model is useful only if it can perform the task to the required standard. For example, if an SLM classifies customer-support requests correctly 98% of the time and that meets the business requirement, moving to a much larger model for a marginal improvement may not be worthwhile. On the other hand, if the task involves interpreting complex contracts where subtle differences in meaning matter, the additional capability of a larger model may be valuable.

**Speed** can become important in applications where responses need to happen quickly. Imagine an AI capability running inside a factory, vehicle or interactive application. A smaller model running locally may respond faster than sending every request to a large cloud-hosted model and waiting for a response. For a background task that prepares a report overnight, however, a few additional seconds may make little difference.

**Cost** needs to be considered at the scale at which the application will actually operate. If an application processes only a few hundred requests a day, using a hosted LLM may be simpler and economical. If it performs millions of predictable classifications or extractions every day, even a small difference in inference cost can become significant. At that scale, running a smaller model may become attractive—provided the savings outweigh the cost of hosting and maintaining it.

**Privacy and data control** may influence the decision as well. A smaller model can potentially run within an organisation's own environment or directly on a device. For applications involving sensitive information, this may allow certain processing to happen locally rather than sending the information to an externally hosted model. Of course, large models can also be deployed with strong enterprise privacy controls, so this is a deployment decision rather than an inherent SLM advantage.

**Hardware and connectivity** matter particularly for edge applications. A drone, robot, vehicle or mobile device may have limited computing capacity or unreliable connectivity. A large cloud model may be impractical for every interaction, while a sufficiently capable smaller model may be able to perform selected tasks locally.

Finally, consider **task complexity**. Classification, extraction, routing and constrained generation may be good candidates for smaller models. Tasks involving ambiguous instructions, unfamiliar situations, multiple sources of conflicting evidence or substantial reasoning may benefit from a larger model.

The decision can therefore be framed around a simple set of questions:

**Can the smaller model achieve the accuracy we need? Is it fast enough? What will it cost at our expected volume? Where must the model run? What are our privacy requirements? And what happens when the model encounters something it cannot handle reliably?**

The best way to answer these questions is not by assuming that an SLM or LLM will perform better. **Build an evaluation set from real examples of the work, test candidate models, and compare their quality, latency and total operating cost.**

In many cases, the answer may not be one model at all. The right architecture may be to let a smaller model handle the majority of predictable work and **escalate the difficult cases to a larger model**.

The goal is not to use the smallest model.

**It is to use the smallest model that can reliably do the work.**

## How We Think About This at Thoughtful Robots#

At Thoughtful Robots, we think the growing range of language models gives organisations an opportunity to be more deliberate about **where different levels of AI capability are actually needed**.

The most capable model is not automatically the right model for every task. A routine classification may need something very different from a complex decision-support workflow. An application running on a device has different constraints from one running in the cloud. And within the same workflow, some tasks may be handled well by a small model while others genuinely benefit from the reasoning capability of a larger one.

Our starting point, therefore, is not **"Should we use an SLM or an LLM?"** It is the work itself: what needs to be done, how reliably it needs to be done, where it needs to run, and what trade-offs around cost, speed, privacy and capability matter.

From there, the right model—or combination of models—becomes an engineering and business decision rather than a technology preference.

**The goal is not to use the biggest model available. It is to use the right amount of intelligence for the work.**

**Exploring how different models could fit into your organisation or product? We'd be happy to work through it with you.**

Figuring out where AI belongs — in your product or how your organisation works? **Let’s work through it together.**

### Understanding MCP: *connecting AI to the tools we already use*#

What MCP is, how hosts, clients and servers fit together, the difference between resources, tools and prompts, how MCP relates to APIs, and where businesses should start.