What is AI actually good for – and what is it not? An in-depth guide for technical decision-makers
06 | 2025 Jari Huilla, CTO & Partner, Kipinä
Artificial intelligence has long been a component of digital solutions in various forms. While the pace of advancement is currently extremely rapid, have you stopped to consider why many models are still fundamentally capable of processing only about 50 pages of text at a time?
CTO Jari Huilla provides an in-depth look for AI professionals into what happens at the core of language models and why their internal Transformer architecture precisely limits the size of the context a model can process in a single pass.
In this article, we dive deep into the internals of language models through technical descriptions to tackle the root causes of these challenges. Nearly all organizations leverage AI to some extent, but when building business-critical solutions upon AI, a deeper understanding of the technology helps conceptualize the limits and limitations of these models.
If you are skimming the article or simply prefer a more practical focus, you can jump straight to the practical tips at the end on how to account for language model limitations during the solution design phase. Dedicated readers will be rewarded with an engaging, in-depth analysis of how language models truly operate under the hood.
(AI was utilized in brainstorming this article, but it was written and illustrated manually.)
First—yes, artificial intelligence is more than just large language models
Various methods classified as AI have long been used specifically to solve optimization and other constrained problems. The limitations of more traditional mathematical and logical models have been relatively straightforward to understand, as the problem must be thoroughly analyzed before selecting an applicable model, and the models themselves are highly specialized.
For example, when planning production or transit schedules, mathematical modeling and linear optimization inherently require an understanding of the model's constraints. Key considerations for linear models suited to schedule planning include:
what decision variables are involved in the problem? (e.g., how many people each on-demand bus should pick up)
how is the quality of the solution calculated? (e.g., at how many stops each on-demand bus must stop on average)
what constraints must the solution satisfy to be acceptable? (e.g., a bus cannot take a negative number of passengers, and the passenger count must be an integer)
what data is required to calculate the solution? (e.g., passengers registered at various stops, maximum duration of the bus route)
is an optimal solution required, or is any solution that satisfies the constraints sufficient?
Thus, even during the modeling phase, it becomes relatively clear what constraints the solution entails. In contrast, understanding constraints in machine learning—and particularly in deep learning models—becomes significantly more complex.
This is further accentuated in large pre-trained models such as large language models, especially because they develop surprising “emergent” capabilities, and it is easy to ask a natural-language-capable model practically anything. How, then, can one address the limitations of large language models?
A peek under the hood: The fundamental causes of current limitations in large language models
Current large language models are trained on staggering amounts of data, and methods for connecting company-specific data to them are advanced and continuously evolving. Yet, when agentically modifying large codebases, one may hit a brick wall where the agent can no longer proceed, or when reasoning via manual prompting, a model may return an error when the context window is exhausted. What is the root cause of these context-size limitations, and what can be done about them?
A brief content warning: this deep dive contains mathematics, but only precisely as much as is strictly necessary! If you do not require such an in-depth overview right now, you can skip to the more practical tips at the end of the article.
Neural Networks
As an introduction to why the Transformer architecture underpinning modern models was originally needed, let us briefly examine neural networks—or recap them if they are already familiar. A neural network is a graph whose nodes are called neurons and whose edges are called synapses. The neurons are arranged in sequential layers, and if there are additional (hidden) layers between the so-called input layer and output layer, it is a deep neural network. The core idea of a neural network is to mimic the structure of biological brains, where electrical impulses travel through a network of nerve cells.
A small neural network with four layers. The input layer of this network has three input neurons, and the output layer has two output neurons. In reality, even the smallest neural networks typically comprise hundreds of neurons, while the largest contain millions.
In the brain, connections between nerve cells change and develop through experiences, thinking, and external stimuli. This is modeled in the neural network training phase by assigning a weight coefficient to each synapse, which determines how much the activation of each neuron in the previous layer influences each neuron in the subsequent layer.
Additionally, each neuron has its own bias term, which helps regulate the significance of an individual neuron's impact on the activation of the next layer.
All neurons in the preceding layer influence each neuron in the subsequent layer in proportion to the weight coefficient formed during training.
Thus, the activation of each neuron is calculated as a weighted sum as follows:
To enhance the network's learning capability, the weighted sum is passed through a normalization function, though this is not relevant in this context. The sum presented above is more generally formulated by arranging the activations of the previous layer into a single vector and the weights into a matrix. Consequently, the activations of the entire $a^{(1)}$ layer can be represented as a matrix-vector product.
The matrix on the left contains the weights of the synapses leading from the previous layer to this layer. The first vector contains the activations of the neurons from the previous layer, and the latter vector contains the bias terms.
This notation will help us understand a bit later why large language models have context window limitations of their current size. Furthermore—in case you hadn't thought about it—this is the exact reason why GPU manufacturer Nvidia has surged to the top of the world's most valuable companies: matrices and vectors are extremely crucial not only in artificial intelligence, but also in 3D graphics!
Neural networks of this type are highly effective for tasks such as image recognition, but in natural language processing and especially in machine translation, even large-scale neural networks of this kind have not proven practical. This is because this type of network practically "forgets" what was mentioned at the beginning of a sentence by the time it reaches the end (see the vanishing gradient problem). Various approaches were tested to solve this problem, but the true breakthrough occurred in the 2017 paper "Attention Is All You Need," which has since become legendary.
Tokens, vectorization, and embeddings
Before diving into the core of this famous paper, let's briefly review what large language models actually do and how natural language is converted into vector format.
Large language models take text as input (prompt), which is divided into tokens. Tokens are short pieces of text, such as syllables, punctuation marks, and individual letters, but for the sake of simplicity, they are presented here as if they were entire words.
The language model essentially calculates probabilities for the next possible word. Here, the probability is represented by the length of the bar preceding the word.
Based on the input it receives, the language model predicts the next word that, according to its training material, would logically follow the previous text. Afterward, the model predicts the next word, and so on. If you have noticed in ChatGPT or Copilot, for instance, that the response appears on the screen in chunks, this is precisely why.
A small amount of randomness is added to language models to make their generated responses more diverse and natural. When using models via APIs, the level of randomness can typically be adjusted.
To perform calculations on words in the first place, a vector representation is calculated for them during the model's training phase. In the case of GPT-4 models, for example, this representation has 12 to 16 thousand dimensions. This representation aims to capture semantic meaning as well, which is why the vector is referred to as an embedding vector. As a result, the vector representations for words like "cat" and "dog" end up relatively close to each other in vector space.
However, what causes current models to understand a word like "kuusi" differently depending on the context—interpreting it either as the number six or the spruce tree?
The Transformer architecture and attention mechanisms
The true breakthrough in large language models' text comprehension and, above all, machine translation was achieved when a suitable mechanism was invented to allow the entire context to influence the vector representation of words. For each input word, an independent vector representation is initially taken from the so-called embedding matrix formed during model training:
The vector representations taken from the embedding matrix do not yet contain any information about each other.
These individual vector representations of the input words are combined into a matrix, which then passes through two types of layers in the Transformer architecture:
Attention layers, where the words in the context also influence one another
Feed-forward layers, where words do not influence each other but instead pass through essentially identical neural networks independently
An arbitrary number of such layers can be stacked sequentially, but they share the common trait that attention layers and feed-forward layers alternate.
Thus, it is within these attention layers that the language model's actual "understanding" of word context is formed. This is the mechanism that allows modern language models to distinguish between the different meanings of a word like "kuusi," significantly improving their performance in translation tasks. If the context contains both instances of the word "kuusi"—one meaning the tree and the other meaning the number—they will end up in different parts of the vector space within the attention layers, even though their initial content is identical!
In the attention layers, other words in the context influence the interpretation of neighboring words, essentially shifting them to a "more correct" position in the vector space.
However, this attention mechanism is currently the exact factor that imposes the strictest limitations on the model's context size. We will not deeply review the various types of attention here, but attention models that genuinely account for all words in the context share a certain computational heaviness.
Specifically, so-called query and key vectors are calculated from all the words in the context and their positions. Query vectors can be thought of as encoding questions related to the word and its position—such as "are there adjectives preceding this word describing it?"—while key vectors serve as answers to these questions. Typically, these query and key vectors are not as high-dimensional as the model itself; instead, they might be, for example, 128-dimensional.
To determine how well each query vector (Q in the illustration) matches each key vector (K in the illustration) for every word pair, the dot products of all query vectors with all key vectors are calculated. This means that the number of required dot products grows exponentially relative to the size of the context window! Therefore, if the input length quadruples, for instance, the required number of dot products increases 16-fold.
For this reason, even at the time of writing this article (June 2025), some models still limit the context window to, for example, 128,000 tokens, whereas previous generations commonly referred to limits of 4, 8, or 16 thousand tokens. In earlier models, the context limit was also frequently encountered in scenarios like ChatGPT conversations, where the model seemed to forget parts of the dialogue. While these constraints have eased somewhat, the context limitations of current models still become apparent when working with large codebases or employing chains of thought.
For comparison: converted into standard book pages, the original GPT-3.5-turbo model could process only about 1.5 pages of text at a time. In the GPT-4o model, this limit is around 50 pages.
At the end of this article, we will also discuss how some modern models, such as GPT-4.1 and Gemini 2.5, offer support for 1 to 2 million tokens (equivalent to 400–800 pages of text), but before that, let's examine what measures should be taken to condense the context—as API calls with large contexts quickly become expensive even when technically supported by the model.
What is the remedy – context optimization
Fortunately, it is not necessary to feed all data to the language model in a single context. One technique used alongside language models is Retrieval-Augmented Generation (RAG), which was originally developed to give language models access to information beyond their training data. This makes it possible to utilize up-to-date or non-public data with the language model. However, RAG also assists with context size: by performing the actual information retrieval outside the language model—for instance, by leveraging vector databases—the context can be pre-filtered to include only the most relevant information. In practice, this means the language model does not need to (nor can it) be fed an entire company's Slack history or SharePoint documents; instead, only the necessary messages or documents are added to the context.
Other techniques related to context optimization include:
pre-summarization: source material is summarized once or, if necessary, even recursively
contextual chunking: attempting to break down the source material into semantically meaningful segments rather than mechanically sized blocks
external memory and/or external tools: leveraging memory outside the model or enabling the use of external tools when needed
What does the future look like?
As noted earlier, the computational workload required by the attention mechanism grows exponentially relative to the length of the context window, if the influence of every single token in the context on the interpretation of all other tokens is to be accounted for. In other words, while a language model can still consider every single word when interpreting a 25-page document, this becomes unfeasible with a 100-page document today because the required computing power increases 16-fold!
A natural follow-up question is whether every single word truly needs to be factored into the interpretation of all other words. Or even if they are, do all computations need to be recalculated every time, or could they be cached?
Current models therefore combine various techniques so that (using the book example) words on the same page have a stronger interrelation than those located dozens of pages apart. Additionally, caching techniques are employed, where managing memory consumption is crucial to prevent resource usage from spinning out of control.
-
When caching is involved, memory requirements easily spiral out of control because, naively, the cache would need to store the query vector, key vector, and the resulting value vector for every single token pair.
With these challenges in mind, various approaches have already been tested across different models to achieve sufficiently good results even if only a subset of token pairs influence one another. On the other hand, research has explored how to reduce recalculation so that cache memory requirements do not get out of hand. These methods include:
sparse attention, where only a subset of token pairs is computed
local + global attention, where tokens close to each other have a stronger effect, and only a subset of tokens influences the entire remaining context
multi-query attention and grouped-query attention, which optimize the calculation of key and value vectors with caching to keep memory requirements more manageable
rotary positional embeddings (RoPE), where a token's positional information (its index within the context) is stored by "rotating" each token pair in different directions. This has significantly improved the encoding of long sequences
multi-head latent attention (MLA), where smaller dimensional latent vectors are stored in the cache instead of standard multi-head attention (MHA). This approach was explored during the development of DeepSeek V2 and is utilized in DeepSeek V3, among others.
In addition to this, the aforementioned techniques are combined with other methods. For example, Mixture of Experts (MoE) is a technique where, instead of a single broad network, the model internally consists of multiple more specialized networks. In the Transformer architecture, it is often used to select which feed-forward layers should be executed for an input, thereby reducing the computational capacity required for inference, although it does not inherently ease the computation required by the attention mechanism. However, it is well-suited for use alongside MLA, for instance.
Finally: Practical tips for achieving better results
So, how does all this work under the hood affect the practical design of AI solutions? No doubt your organization has already experimented with various AI use cases or is already leveraging large language models, but below are our tips for situations where AI solutions are being scaled and moved to production:
-
As stated initially, established methods already exist for many constrained, mathematically oriented problems—solutions for which LLMs are practically entirely unsuited. Do not attempt to solve problems like routing or shift scheduling using a large language model, as they yield inaccurate results. Instead, select or, if necessary, train a specialized model tailored to the problem.
-
The solution typically becomes more predictable, reliable, observable, and cost-effective when the problem is broken down into smaller components. This approach also allows you to leverage other types of models for areas where they perform better, while keeping the context size of individual LLM calls manageable.
-
Even large language models perform better the more relevant the information provided to the constrained context is. Both data quality and the models' accessibility to data directly impact the final outcome. Do not reinvent the wheel: leverage technologies such as MCP (Model Context Protocol) and pre-built tools designed around it.
-
Reasoning models are exceptional for tasks such as ad-hoc coding, but avoid overusing them when operationalizing formulaic solutions. In such cases, it is advisable to explore the problem using reasoning models initially, but then break down the problem yourself once viable solution patterns begin to emerge. Reasoning consumes its own share of the context window, and you may encounter token limits or unexpected costs if reasoning is used unnecessarily.
-
Rapid advancements are continuously occurring regarding both context size and other capabilities, meaning that a familiar model family is not always the optimal choice. For instance, if you typically rely on OpenAI models and hit limitations, check the features offered by the latest releases from Gemini, Claude, Mistral, or DeepSeek. For example, Gemini 1.5 Pro introduced a one-million-token context window several months before it became available in OpenAI's models via GPT-4.1.
Explore these tips on how to overcome underlying constraints to scale your AI solutions from proof-of-concept to production-grade readiness. By approaching your situation and challenges systematically, we at Kipinä provide strong strategic vision and top-tier technical expertise to help you build the next generation of AI solutions!
If you need a sounding board on how to raise the bar for your AI solutions or insights into comprehensive digital development, get in touch with Jari!
Jari Huilla, CTO & Partner, Kipinä
Jari Huilla is Kipinä's CTO, bringing an exceptionally extensive and versatile background in technology; his first role in the industry was at Nokia Research Center at the age of 15. Over the years, he has served as a developer, leader, and builder of growth companies. At Kipinä, Jari combines deep technical expertise with business-driven thinking. He is particularly passionate about making AI solutions not only technically functional but also genuinely purposeful—and ensuring their limitations are understood rather than simply overlooked.