Decoding AI TOPS: Essential Metrics for AI Chips and TOPS Comparison Chart

(Illustration: A lot of hard work has been going on behind the scenes. Le Bouchon Ogasawara, Shibuya, Tokyo. Image source: Ernest)

tl;dr

• TOPS (Trillions of Operations Per Second) is a key indicator for measuring the computational power of AI chips and NPU chips, reflecting the trillions of operations a processor can execute per second.
• Using the “frying eggs” analogy to intuitively understand TOPS: A regular CPU is like a chef who can only fry one egg at a time, while a high TOPS performance AI chip is like a super chef who can fry an incredibly large number of eggs simultaneously.
• TOPS is an important reference for comparing AI chip performance, but when evaluating AI hardware, factors such as energy efficiency and memory bandwidth should also be considered. Additionally, TOPS values typically reflect theoretical peak performance, and actual performance needs to be judged based on a combination of other metrics suitable for the application scenario.

What is TOPS (Simple Life Version)

TOPS, which stands for Trillions Operations Per Second, is an important indicator for measuring the computational power of Artificial Intelligence (AI) chips or Neural Processing Units (NPUs). TOPS is used to express the maximum number of operations a processor can execute per second, calculated in trillions. In the future, if computational power continues to increase, the initial T may be replaced by other larger units of measurement.

We can use an example from daily life to explain and more intuitively understand TOPS:

Imagine AI computation as the process of frying eggs, where data is the egg being heated.

A regular chef (ordinary processor, CPU) might only be able to fry one egg at a time, while a super chef (AI chip) might be able to fry 1 trillion eggs simultaneously! TOPS is like a measure of this “super chef’s” ability, telling us how many “data eggs” it can “fry” per second.

TOPS is one of the important indicators for understanding and comparing AI chip performance, but it’s not the only one.

When evaluating AI hardware, AI phones, or AI computers, remember to consider other factors such as energy efficiency, memory bandwidth, software ecosystem, etc. Using TOPS can help us compare the computational power of different AI chips, providing a reference point for choosing AI hardware devices suitable for specific applications.

What is TOPS (In-depth Version, for Those Who Insist)

Before delving deeper into understanding TOPS, we need to first understand what an “operation” is:

In digital circuits and computer science, an “operation” typically refers to a basic mathematical or logical computation. For AI chips or NPUs, these operations mainly include:

1. Floating-point operations: such as addition, subtraction, multiplication, and division.
2. Matrix operations: Large-scale matrix multiplication is one of the most common operations in deep learning.
3. Vector operations: including dot product (scalar product), cross product (vector product), etc.
4. Activation functions: such as ReLU, Sigmoid, Tanh, etc.
5. Convolution operations: widely used in Convolutional Neural Networks (CNN).

These operations are typically performed in FP32 (32-bit floating-point) or FP16 (16-bit floating-point) formats. Some AI chips also support lower precision formats like INT8 (8-bit integer) to improve performance and reduce energy consumption, typically used for inference.

The calculation of TOPS can be simplified as:

TOPS = (Number of operations per clock cycle) × (Clock frequency) / 1 trillion

For example, if an AI chip can perform 1000 operations per clock cycle and has a clock frequency of 1GHz, then its theoretical peak performance is 1 TOPS.

1000 operations/cycle × 1GHz = 1000 × 10^9 operations/second = 10^12 operations/second = 1 TOPS

When understanding TOPS, please note the following points:

1. TOPS typically represents theoretical peak performance; actual performance may vary due to factors such as memory bandwidth and chip architecture.
2. TOPS values may differ for different types of operations (such as FP32, FP16, INT8).
3. A high TOPS value doesn’t necessarily mean better performance in all AI tasks, as actual performance also depends on software optimization and the characteristics of specific tasks.

TOPS Comparison Table

(Focus mainly on the “INT8 Ops” column. You can swipe left and right to see more comparison data)

Reference

• A guide to AI TOPS and NPU performance metrics | Qualcomm
• Embedded - TOPS AI vs. Real World Performance
• Neural Network Processors and DSPs for AI | Synopsys
• TOPS, Memory, Throughput And Inference Efficiency
• AI-Benchmark