What Is Really Slowing AI Down? It’s Often Not Computing Power

When discussing the development of AI, attention usually focuses on the capabilities of new models, their speed, or the number of parameters they contain. However, once a company begins operating AI at scale, it quickly becomes clear that the model itself is only one part of the system.

Where the model runs, how it works with memory, and how quickly the infrastructure can move the necessary data are just as important. A well-designed model can be slow and expensive on unsuitable hardware. On the other hand, the right combination of software, hardware, and system architecture can improve performance without constantly adding more servers.

When Data Movement Becomes the Bottleneck

The performance of processors and graphics accelerators is often measured by the number of mathematical operations they can perform per second. With today’s AI models, however, calculating quickly is not enough. The accelerator must also retrieve large amounts of data from memory, process it, and pass it to other parts of the system.

This is where the so-called Memory Wall appears. For some AI workloads, the main limitation is no longer computing power itself, but the speed at which the system can load model weights and move data between memory, accelerators, and individual servers.

‍

A recent study by researchers from the Barcelona Supercomputing Center and IBM Research shows that even under larger workloads, inference can remain limited by memory bandwidth while part of the GPU’s computing capacity stays underutilized. The study Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference explores this problem in greater detail.

Put simply, even a powerful chip cannot reach its full potential if it has to wait for data. This is why the capacity and bandwidth of high-speed memory, the speed of connections between accelerators, network infrastructure, and the software managing the entire process have become so important. The performance of AI infrastructure therefore cannot be assessed solely by looking at the specifications of an individual GPU. The entire system matters.

Why Adding More GPUs Is Not Always Enough

Large models often require more processing power than a single accelerator can provide. The model or its calculations must therefore be distributed across several devices that constantly communicate with one another. Simply adding more GPUs, however, may not deliver a corresponding increase in performance. If the bottleneck lies in memory, networking, or the way calculations are distributed, part of the expensive infrastructure may remain underutilized.

‍

This is why technologies such as NVLink and NVSwitch address not only the performance of individual chips but also the speed of communication between them. In modern AI infrastructure, accelerators, memory, networking, and the software coordinating the entire process must be designed to work together.

General-Purpose GPUs Versus Specialised Accelerators

GPUs are popular primarily because of their versatility. They can be used for model training, inference, image processing, and many other computationally intensive tasks. At a large enough scale, however, this versatility can become expensive. When a technology company processes millions of AI requests, optimising hardware for specific types of calculations can start to make sense.

‍

Google develops its own TPUs, accelerators focused on the tensor and matrix operations used in machine learning. AWS offers Trainium and Inferentia chips, which provide an alternative optimised for training and running AI models.

‍

Apple Silicon takes a different approach. Its CPU and GPU share the same memory space, reducing the need to repeatedly copy certain data between system and graphics memory. This architecture can be highly efficient for local development and running smaller or quantised models. There is no single type of hardware that is best for every situation. The right choice depends on the model, the scale of usage, the required speed and the available budget.

How Software Helps Overcome Hardware Limitations

Higher performance does not always require purchasing additional servers. A significant part of today’s progress comes from using existing infrastructure more efficiently. Quantisation and FlashAttention are good examples.

‍

Quantisation converts the values used by a model into a lower-precision numerical format, for example from 16 bits to 8 or 4 bits. This can reduce the model’s size, memory requirements, and the amount of data that needs to be transferred. As a result, the model may run faster or operate on less expensive hardware.

‍

The impact on quality, however, depends on the specific model, the method used, the hardware, and the way the model is deployed. Quantisation is therefore not merely a form of technical compression. It is a decision that balances performance, cost, and output quality.

‍

FlashAttention optimises how a model works with memory while calculating attention. It changes the order of operations to reduce the amount of data transferred between different levels of memory. The result can be lower memory consumption and faster training and inference. This shows that AI performance is not improved only by adding more transistors. Smarter algorithms and more efficient data handling can be just as important.

When Should Companies Actually Address This?

Not every company needs its own GPU cluster or an in-depth understanding of accelerator architecture. For the initial validation of an AI use case, using an existing API is often the fastest approach. The company does not need to invest in its own infrastructure or accurately predict future usage in advance.

‍

The situation changes when AI becomes an important part of a product or internal process. Architecture should receive greater attention when the number of requests increases significantly, operating costs begin to affect the economics of the product, the company needs more stable response times, or the model works with sensitive data.

At that point, the company must decide between an external API, a managed cloud service, its own deployment, or a combination of several approaches. In some cases, a smaller model may be enough. In others, quantisation, caching, or more efficient request routing may provide the best result. There is no single architecture suitable for every company. The right solution always depends on the actual workload, expected growth and the economics of the entire use case.

The Real Advantage Emerges Between Software and Hardware

Buying access to an AI model or running an open-source solution on a cloud GPU is relatively easy today. Building a system that is fast, stable, secure, scalable and economically sustainable is much more difficult. This is where software development and infrastructure meet. The right architecture may allow a company to solve the same problem with a smaller model, less expensive hardware, or fewer calculations. A poorly designed system, on the other hand, may consume a significant budget without delivering corresponding value.

How We Approach It at Techmates

At Techmates, we do not look at AI only through the capabilities of the model itself. We consider the entire system: the problem it is intended to solve, where it will run, how it works with data, what its performance requirements are, and how much it will actually cost to operate. Not every company needs its own model or GPU infrastructure. Every company should, however, understand whether its chosen architecture reflects its real needs and expected growth.

‍

The goal is not technology for technology’s sake, but a solution that continues to work reliably beyond the initial experimental stage. This is often the difference between an interesting AI prototype and a production-ready solution that delivers a genuine business outcome.

‍

Are you looking for a way to deploy AI that can grow alongside your business?
Explore our case studies or get in touch with us.