Central Processing Unit (CPU), Graphics Processing Unit (GPU) and Tensor Processing Unit (TPU) are processors with a specialized purpose and architecture. We have compared these in respect to Memory Subsystem Architecture, Compute Primitive, Performance, Purpose, Usage and Manufacturers.
Note that if the correct compiler is present, all CPU, GPU and TPU can achieve the same task or result but by following a different path and different performance.
Memory Subsystem Architecture
This image captures the memory architecture of CPU, GPU and TPU:
This image summarizes the compute primitive (smallest unit) in CPU, GPU and TPU:
The dimension of data are:
- CPU: 1 X 1 data unit
- GPU: 1 X N data unit
- TPU: N X N data unit
As a comparision, consider this:
CPU can handle tens of operation per cycle
GPU can handle tens of thousands of operation per cycle
TPU can handle upto 128000 operations per cycle
Central Processing Unit (CPU): A processor designed to solve every computational problem in a general fashion. The cache and memory design is designed to be optimal for any general programming problem.
Graphics Processing Unit (GPU): A processor designed to accelerate the rendering of graphics.
Tensor Processing Unit (TPU): A co-processor designed to accelerate deep learning tasks develop using TensorFlow (a programming framework); Compilers have not been developed for TPU which could be used for general purpose programming; hence, it requires significant effort to do general programming on TPU
Central Processing Unit (CPU): General purpose programming problem
Graphics Processing Unit (GPU): Graphics rendering, Machine Learning model training and inference, efficient for programming problem with parallelization scope, General purpose programming problem
Tensor Processing Unit (TPU): Machine Learning model (only in TensorFlow model) training and inference
Central Processing Unit (CPU): Intel, AMD, Qualcomm, NVIDIA, IBM, Samsung, Hewlett-Packard, VIA, Atmel and many others
Graphics Processing Unit (GPU): NVIDIA, AMD, Broadcom Limited, Imagination Technologies (PowerVR)
Tensor Processing Unit (TPU): Google