Abstract

Inference is the process of applying trained models to new data for predictions. It plays a vital role in real-world applications, enabling insights, automation, and real-time responses. Throughput measures task efficiency, while latency indicates prediction delay. Top-K accuracy assesses model performance, and precision encompasses numerical representation. This article also highlights the significance of inference in harnessing AI's potential across diverse domains.

Table of Content

	Topics
1	What is Inference?
2	Why do we need inference?
3	Throughput
	- Factors Affecting Throughput
4	Latency
5	Top-K Accuracy
6	Output Type
7	Precision
	- Full Precision (32-bit or 64-bit)
	- Half Precision (16-bit)
	- Low Precision (8-bit or lower)
	- Mixed Precision
8	Pre-trained Model
9	Different formats
10	Conclusion

What is Inference?

Inference refers to the process of drawing conclusions or making deductions based on available information, evidence, or reasoning. In the context of machine learning, inference refers to the process of applying a trained model to new, unseen data to make predictions or draw conclusions. During the inference phase, the model takes input data, processes it, and produces an output, which could be a classification label, a regression value, or some other kind of prediction, depending on the nature of the ML problem.

Why do we need inference?

Inference is a critical component of machine learning (ML) because it's the phase where the trained models are put to practical use by making predictions on new, unseen data. Here are some reasons why inference is essential:

Real-World Applications: The primary goal of training ML models is to make accurate predictions or decisions on new data that the model has never seen before. Inference allows models to be applied to real-world scenarios, enabling them to provide valuable insights, automate tasks, and assist in decision-making processes.

Scale: Inference allows ML models to handle a large volume of data efficiently. This scalability is crucial for analyzing social media feeds, processing sensor data from IoT devices, or detecting anomalies in financial transactions.

Real-Time Responses: Many applications require real-time responses. Inference makes it possible for ML models to provide instant feedback and responses to user inputs, which is essential for applications like chatbots, voice assistants and autonomous vehicles.

Throughput

Throughput refers to the rate at which a system or process can complete a certain number of tasks or transactions over a specified period. Throughput specifically measures the number of inference tasks a system can handle within a given time frame. For example, if we have a machine learning model deployed for image classification and it can process 1000 images in one second, the throughput of the system is 1000 inferences per second (IPS).

This metric is particularly important in applications that require real-time processing, such as video streaming, autonomous vehicles, and industrial automation, where a high throughput is essential to ensure timely responses.

Throughput is affected by various factors, including:

• Performance of the hardware
• Efficiency of the software stack
• Complexity of the model being used

Latency

Latency refers to the time delay between providing input data to a machine learning model and receiving the corresponding prediction or output. It's the time it takes for the model to process the input and generate a response. It measures how long it takes to predict after it's given some data to work with. In online gaming, latency is often referred to as 'lag' and it's the delay between when a player performs an action and when that action is reflected in the game.

Low latency is crucial in applications where real-time or near-real-time responses are required like:

• Autonomous Vehicles
• Video Streaming
• Online Gaming
• Voice Assistants
• Financial Transactions

Top-K Accuracy

Top-K accuracy is a metric used to evaluate the performance of classification models, particularly in situations where the correct answer might not always be the top prediction. It measures the percentage of times the correct label is present among the top 'K' predicted labels generated by the model.

Here's how top-K accuracy works:

Prediction and evaluation: When you use a machine learning model to classify an input (for example, an image of a dog), the model produces a list of predictions with associated probabilities for each possible class (dog, cat, bird, etc.).

Selecting K: The 'K' in top-K accuracy represents the number of predictions to consider. If we're interested in whether the correct label is among the top 3 predicted labels, then K would be 3.

Checking correctness: The top-K accuracy metric checks whether the correct label (the true class of the input) is present in the top K predicted labels. If it is, then the model's prediction is considered correct and if not, it is considered incorrect.

Calculating accuracy: The top-K accuracy is calculated by counting the number of times the correct label is among the top K predictions and dividing that by the total number of predictions. This gives us the percentage of correct predictions within the top K predictions.

Output Type

The output type in the inference process refers to the kind of result or prediction that a machine learning model generates when it processes input data. Here are a few examples of different output types in the context of machine learning inference:

Classification label: In classification tasks, the model assigns an input data point to a specific category or class. For example, an image classification model might output a label like 'cat', 'dog', or 'bird' to indicate what it thinks the image contains.

Probability distribution: Along with the classification label, some models provide a probability distribution indicating how confident the model is in each possible class. For example, a model might output that an image is 80% likely to be a dog, 15% likely to be a cat, and 5% likely to be a bird.

Regression value: In regression tasks, the model predicts a numerical value based on the input data. For example, a model trained to predict housing prices might output a predicted price in dollars.

Sequence of tokens: In natural language processing tasks like language generation or translation, the model can generate a sequence of words or tokens as the output. For example, a language generation model might produce an entire sentence or paragraph of text.

Precision

Precision in the context of inference refers to the level of numerical accuracy or representation used during computations. Different types of precision are used in deep learning models for inference, including low precision. Here are some types of precision commonly used in inference:

Full Precision (32-bit or 64-bit):

32-bit Single Precision: Standard floating-point representation using 32 bits for each floating-point number.
64-bit Double Precision: It is similar to single precision, but uses 64 bits for each floating-point number, providing higher numerical accuracy.

Half Precision (16-bit):

16-bit Floating-Point (FP16): Uses 16 bits to represent floating-point numbers. It provides a balance between numerical accuracy and memory efficiency, often used for low-power devices and faster computations.

Low Precision (8-bit or lower):

8-bit Integer (INT8): Represents numbers as 8-bit integers, without any fractional component. It requires quantization and often used for specific tasks with minimal accuracy loss, such as image classification.
4-bit Integer (INT4): Extremely low precision quantized representations used in specialized hardware or cases where memory and computation constraints are extremely tight.

Mixed Precision:

This approach involves using different precision formats for different parts of a neural network during inference. Less critical layers can use lower precision, while critical layers may use higher precision.

Pre-trained Models

Pre-trained models are like smart helpers that already know a lot. They've learned from big amounts of information, so we don't have to teach them everything from scratch. This saves time and energy. They're super useful for tasks like understanding pictures, speech recognition etc. These models are great at spotting things, understanding words, and doing other tasks.

To use them, firstly, we tell the computer which model to use. Then, we get the data ready in a certain way. Next, we let the model work on the data and it gives us useful information. We might need to edit and organise this information afterward. Finally, we can use this helpful information to make our apps smarter.

Different Formats

Machine learning models can be deployed and used for inference in various formats, each catering to specific platforms and use cases.

TensorFlow Protocol Buffers (PB) Format:
TensorFlow is a popular open-source machine learning framework developed by Google. TensorFlow models can be saved in a format called Protocol Buffers (PB), which allows efficient storage and transmission of data structures. The PB format includes the model architecture as well as the trained weights and biases.

Open Neural Network Exchange (ONNX) Format:
ONNX is an open standard format for representing machine learning models. It allows interoperability between different frameworks such as PyTorch, TensorFlow, and more. ONNX enables models trained in one framework to be converted and used in another framework without significant modifications.

PyTorch JIT Format:
PyTorch allows models to be exported in its Just-In-Time (JIT) format. This format includes both the model architecture and associated operations in a way that enables efficient runtime execution.

MXNet Model Format:
Apache MXNet is another deep learning framework that has its own model serialization format. MXNet models can be saved in a format that includes both the model architecture and parameters.

Example

import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2, imagenet_utils
from PIL import Image

These line tells the computer that we want to use a library called TensorFlow to work with machine learning models. We're specifically importing two things from TensorFlow's applications module. One is a pre-built model called MobileNetV2 which is good at understanding images, and the other is a utility called imagenet_utils that helps with image-related tasks. We're also importing a part of a library called PIL (Python Imaging Library) that helps us work with images in Python.

model_path = 'path_to_saved_model.pb'
model = tf.saved_model.load(model_path)

We're defining a variable called model_path and assigning it a string value. We are using the tf.saved_model.load function to load the saved TensorFlow model from the specified model_path.

image_path = 'example_image.jpg'

This sets the path to the image file we want to analyze.

image = Image.open(image_path)

We're using the PIL library to open the image from the file we specified earlier.

image = image.resize((224, 224))

We're making the picture a specific size (224 x 224 pixels) because the MobileNetV2 model needs images to be this size.

image_array = tf.keras.preprocessing.image.img_to_array(image)

We're turning the image into a special kind of list that the computer can understand and work with.

processed_image = imagenet_utils.preprocess_input(image_array)

We make some changes to the image list so that it's in a form the MobileNetV2 model can understand better.

processed_image = tf.expand_dims(processed_image, axis=0)

We're adding an extra bit of information to the image list so that the MobileNetV2 model knows we're only giving it one picture to look at.

predictions = model.predict(processed_image)

We're giving the picture to the MobileNetV2 model and asking it to figure out what's in the picture.

decoded_predictions = imagenet_utils.decode_predictions(predictions)

We're taking the information the model gave us and turning it into words that we can understand.

for label in decoded_predictions[0]:

We go through the list of words that describe what's in the picture, one by one.

print(label[1], label[2])

Now, finally, we are printing out the words that describe what's in the picture.

    import tensorflow as tf
    from tensorflow.keras.applications import MobileNetV2, imagenet_utils
    from PIL import Image

    model_path = 'path_to_saved_model.pb'
    model = tf.saved_model.load(model_path)

    image_path = 'example_image.jpg'

    image = Image.open(image_path)
    image = image.resize((224, 224))
    image_array = tf.keras.preprocessing.image.img_to_array(image)
    processed_image = imagenet_utils.preprocess_input(image_array)
    processed_image = tf.expand_dims(processed_image, axis=0)

    predictions = model.predict(processed_image)

    decoded_predictions = imagenet_utils.decode_predictions(predictions)
    for label in decoded_predictions[0]:
        print(label[1], label[2])

Conclusion

The process of inference serves as the bridge between trained machine learning models and their practical application in various real-world scenarios. Inference helps us to harness the potential of artificial intelligence to make informed decisions, automate tasks, and enhance user experiences.

As research delves into mixed-precision approaches, adaptive precision scaling, and probabilistic programming, we're seeing models perform with unprecedented speed and resource optimization without compromising accuracy.

The realm of inference will also intersect with other expanding fields. Federated learning, for instance, is enabling models to be trained collaboratively across distributed devices while preserving privacy. This paves the way for highly personalized applications without compromising on sensitive user data.