Different Layers / Operations in Machine Learning models

There are over 50 different types of layers / operations in Machine Learning models. A model is simply a combination of these layers. Each layer has a distinct function and takes an input of a particular dimension and the output is in another dimension.

Following are the layers/ operations in a Machine Learning model:

Convolution

Convolution is the most fundamental operation in Machine Learning models. It help to extract features from data. There are several variants of Convolution which are used in different models for different applications.

Convolution is the most compute intensive operation in Machine Learning models consuming over 70% of the execution time. Following are the complete list of different types of Convolution layers:

  • Convolution operations:
    • Convolution 1D
    • Convolution 2D
    • Convolution 3D
    • Depthwise Convolution (1D, 2D, 3D)
    • Seperable Convolution (1D, 2D, 3D)
    • Convolution 2D_in_plane
    • Convolution 2D_transpose
    • Convolution 3D_transpose
    • Convolution 3D_in_plane
    • Quantized version (INT8, INT16) of above Convolutions
    • Lower precision Convolutions like BF16, FP16
    • Convolutions fused with other post ops like Bias, ReLU

Pooling

Pooling is a fundamental operation in Machine Learning models to highlight features detected by Convolution ops. This is the second most computation intensive operation. Following are the different types of Pooling operations:

  • Pooling operations:
    • Average Pool (1D, 2D, 3D)
    • MaxPool (1D, 2D, 3D)
    • Minimum Pool (1D, 2D, 3D)
    • Median Pool (1D, 2D, 3D)

Activation functions

Activation functions are the core of Machine Learning models. Extensive research has been done in this though ReLU op is the most widely used. The complete list of different types of Activation functions are:

  • Activation
    • Binary Step Function
    • Linear Function
    • Sigmoid
    • Tanh
    • eLU
    • ReLU
    • ReLU6
    • Legacy ReLU
    • Parameterized ReLU
    • Exponential Linear Unit

Ops for Quantization

Quantization is a hot research area and is an effective technique to accelerate Inference and reduce the size of Machine Learning models. All ops like Convolution, Pooling and others have separate versions for Quantization. Apart from that, these are the exclusive operations that are available only in Quantization:

  • Min op
  • Max op
  • Quantize op
  • Dequantize op

Other core Ops

  • Reorder: Ops work on different data formats. Reorder op is used to change data from one data format to another.
  • Reshape: Reshape op is to change the dimensions of the input. The total size of data remains the same.
  • Padding: Pad is a fundamental op which increases the size of the input but adding default data in different dimensions.
  • Batch Normalization
  • Bias Add op (fused)
  • dense_to_sparse
  • Dropout op: This op is used to randomly remove elements from an input. This is a standard optimization technique.
  • flatten
  • fully_connected
  • GDN
  • images_to_sequence
  • layer_norm
  • linear
  • one_hot_encoding
  • repeat
  • scale_gradient
  • sequence_to_images
  • softmax
  • spatial_softmax
  • stack
  • unit_norm
  • legacy_fully_connected
  • legacy_linear
  • maxout

With this, you must have the complete idea of the different operations that exist in Machine Learning models. All mainstream ML frameworks like TensorFlow support these operations.