Get Ready for a Shakeout in Edge NPUs

When the potential of edge AI first sparked our imagination, semiconductor designers realized that performance (and low power) required accelerators, leading many to build their own. The requirements are not overly complex, commercial alternatives are limited, and few want to add another patent fee to further reduce profit margins. NPUs have become ubiquitous, both internal, startup, and commercial IP portfolios expanding. We remain in this mode, but signs indicate this melee must end, especially for edge AI.

Accelerated Software Complexity

The flood of innovation around neural‑network architectures, AI models and foundation models is inevitable. From CNN to DNN, to RNN, and now transformers, models evolve from vision, audio/voice, radar, lidar to large‑language models such as ChatGPT, Llama and Gemini. The only certainty is that whatever is state‑of‑the‑art today will need to be upgraded next year.

The operator/instruction‑set complexity required to support these models is exploding. A simple convolution model once needed fewer than 10 operators; today the ONNX standard defines 186, and NPUs allow extensions to this core set. Modern models combine matrix/tensor, vector, scalar and mathematical operations (activation, softmax, etc.). Supporting this range requires a software compiler to map low‑level hardware to simplified network models, plus an instruction‑set simulator to verify performance on the target platform.

NPU vendors now must provide pre‑validated, optimized model zoos (e.g., CV, audio) on their platforms to alleviate buyer concerns about adoption and ownership costs.

Accelerated Hardware Complexity

Training platforms are architecturally constrained today, mainly by the choice of GPU or TPU. Inference platforms, however, are not. Initially they were seen as scaled‑down training platforms, mainly converting floating‑point to fixed‑point, more aggressively quantized representations. This view has shifted dramatically. Most hardware innovation now occurs in inference, especially for edge applications under intense performance‑and‑power pressure.

When optimizing trained networks for edge deployment, pruning zeros out parameters that have little impact on accuracy. Given that some models have billions of parameters, pruning can dramatically improve performance and reduce power by skipping those computations.

While sparsity can be effective when hardware processes one computation at a time, modern hardware leverages massive parallelism in pulse‑array accelerators, which cannot skip scattered computations. Software and hardware techniques exist to recover pruning benefits, but they are still evolving and unlikely to be solved soon.

Convolutional networks were the starting point for modern AI and remain a crucial feature‑extraction component, even in vision transformers. They can run on pulse‑array hardware but less efficiently than the dense matrix multiplications common in LLMs. Accelerating convolutions remains a hot research topic.

Beyond these acceleration challenges, vector calculations such as activations and softmax either require math not supported by standard pulse arrays or run inefficiently because most arrays are idle during single‑row or single‑column operations.

A common approach to these challenges is to combine tensor engines (pulse arrays), vector engines (DSPs) and scalar engines (CPUs), possibly across multiple clusters. The tensor engine handles operations it excels at, vector work goes to the DSP, and the remaining (including custom/math) is passed to the CPU.

This solution inevitably raises product cost in chip area and potential patent fees, increases power consumption, and makes software development, debugging and updating more complex. Hence developers prefer a universal NPU engine and programming model to absorb all this complexity.

Supply‑Chain/Ecosystem Growing Complexity

Intermediate manufacturers in the supply chain must build or at least adapt models for target systems, considering variations such as camera lenses. They lack time or flexibility to support every platform, limiting which NPUs they can support.

Software ecosystems also aim to serve high‑capacity edge markets, e.g., audio‑personalization for earbuds and hearing aids. These value‑added software firms will tend to support only a few platforms they are prepared for.

The survival‑of‑the‑fittest dynamics may accelerate faster than the early CPU boom, but some competition between options remains necessary. Nonetheless, the current Cambrian explosion of edge NPUs is expected to end soon.

Source: https://semiwiki.com/artificial-intelligence/349906-get-ready-for-a-shakeout-in-edge-npus/