Bilinear Upsampling

In this OpenGenus article, let us discuss Bilinear Upsampling which pops up quite bit in regards to image manipulation and processing in a greater detail.

Contents

• Introduction : Upsampling
• Upsampling Techniques
• Bilinear Upsampling
• Pseudocode Implementation
• Precautions
• Applications
• Key-takeaways

Introduction : Upsampling

In image processing, upsampling refers to the process of increasing the resolution or dimensions of an image by adding new pixels between the existing ones. The goal is to make the image larger or more detailed without introducing noticeable distortions or artifacts.

Though the terms 'Interpolation' and 'Upscaling' may sound very similar to 'Upsampling', they are not to be confused with one another.

• Interpolation is a technique used during upsampling to estimate the values of the new pixels based on existing ones.

• Upscaling is the broader process of enlarging an image to a higher resolution while aiming to maintain or even improve visual quality. Unlike basic upsampling, upscaling may use more advanced techniques like machine learning, AI, or deep learning to intelligently predict and add details that were not in the original image to enhance the perceived quality.

Upsampling Techniques :

There are several upsampling techniques used in image processing, each with different approaches to generating new pixels and estimating their values. These techniques can be categorized based on their complexity, ranging from simple mathematical methods to advanced AI-based approaches.
The most common upsampling techniques are:

• Nearest-Neighbor Interpolation : This is the simplest upsampling technique. When increasing image size, the nearest pixel's value is copied into the new pixel location.

• Bilinear Interpolation : This method calculates the new pixel values based on the weighted average of the four nearest surrounding pixels. It considers both horizontal and vertical neighbors.

• Bicubic Interpolation : This takes into account the 16 nearest surrounding pixels to compute the value of new pixels. It uses cubic functions to calculate smoother gradients and transitions between pixels.

and there various other techniques with increasing complexity such as Lanczos Resampling, Spline Interpolation, Fourier-based Upsampling etc.

Now, let us delve into the details of Bilinear Upsampling.

Bilinear Upsampling

Bilinear Upsampling is the process of creating a higher resolution image by generating new pixels through bilinear filtering from a lower resolution image.

• Bilinear upsampling is relatively simple and computationally efficient compared to more advanced techniques like bicubic interpolation or AI-based upscaling. Its complexity is low because it relies on basic mathematical operations—specifically, linear interpolation in two dimensions.
• It also does not require much extra memory since it only operates on the pixel values in a small neighborhood (a 2x2 grid of pixels).

Let us take an example to understand the process more intutively.
Say, we require to upsample images by a factor of 2.
For that, we can apply our bilinear filters primarily in these two ways.

Firstly, in a way which is similar to box filters, so much so, that these two terms can be used interchangeably.

Here, we intend to keep our edges lined up while shifting our pixel centres by half a pixel.

And secondly,

In the second method, we keep our pixel centres aligned and shift the edges by a certain amount.

Once we have our two filters , we can attempt to do Bilinear Upsampling.

This is the simplest approach where we place copies of the original pixels in the new pixels whose pixel centres are aligned with it. In the remaining spaces, we compute and place the average of the adjacent pixels as shown.

Doing this results in a very unusual property of the new image, where some pixels are sharp(the ones which were copied directly), whereas others are a bit more hazy.

This method is not fool-proof and creates certain problems, one of which is, when we perform downsampling of an image and then perform upsampling on top of it.
We see that the entire image is shifted by half a pixel.

This method does not deal with the error of shifting during the upsampling process.

However, we can see that the problem could be fixed by changing our upsampling filter.

In this method, the values of all the pixels are interpolated from the original values with no direct copy being made. A pixel being upsampled has a weight matrix of [0.25 0.75 0.75 0.25] for its four target pixels. Each pixel in the resultant upsampled image is a combination of two parent pixels with weights of [0.75 0.25] or [0.25 0.75] dpending on its position.

We can further spread out the interpolation by making our filters be [0.125 0.375 0.375 0.125].

Pseudocode Implementation

Bilinear upsampling is, as reiterated previously, computing linear interpolation in two dimensions.

1. We must begin by creating an empty 2D array of the desired resized shape containing all zeros(0).
2. Then we need to specify or calculate our scaling factor as decided upon by the user.

#Calculating width and height scaling factors
 w_scale_factor = (new_width) / (old_width)
 h_scale_factor = (new_height) / (old_height)

3. We aim to fill the pixel values by using two loops for iteration along the two dimensions.
4. To compute the values present in the pixels of the new array, for every pixel in the new array, we need to map their neighbouring pixels in the original image.

for i in range(new_height):
 for j in range(new_width):
 
  #Mapping the coordinates of the new pixel to its neighbours in the original image
  x = i * h_scale_factor
  y = j * w_scale_factor
  
  #Calculating the coordinate values for the 4 surrounding pixels.
  x_floor = math.floor(x)
  x_ceil = min( old_height - 1, math.ceil(x))
  y_floor = math.floor(y)
  y_ceil = min(old_width - 1, math.ceil(y))

5. Once we can calculate the coordinates of the neighbouring pixels in the original image, we can use the pixel values for our upsampling operation.

#Getting  the neighbouring pixel values
    v1 = original_img[x_floor, y_floor, :]
    v2 = original_img[x_ceil, y_floor, :]
    v3 = original_img[x_floor, y_ceil, :]
    v4 = original_img[x_ceil, y_ceil, :]

6. Finally, we use the neighbouring pixels values of the original image along with our weights to calculate the pixel values in the new array.

Input Size : The height and width parameters of the original image

Output Size : It depends on the scaling factor.
It follows the formula : Input size * scaling_factor

Time Complexity:
The execution time depends upon the output height and width desired (the two nested loops) as the rest are simple arithmetic computations.
Therefore, the time complexity is O(n^2).

Precautions

• The upsampling operations need to be aware of the downsampling methods used, including how they define the pixel grid offset, and vice versa. Both processes should take each other into account for better results.
• We need to take into account the size of our filters. If we have an odd number of samples in our filter, then there is a defined centre of the filter; but if we have an even number of samples in our filter, then there is no well defined centre, and thus the resultant image is shifted by half a pixel.

Applications

• Image and Video Resizing:
  Bilinear upsampling is commonly used to scale images or videos to larger resolutions. This is often seen in gaming, and image editing software, where smoothness and speed are essential.

• Computer Graphics and 3D Rendering:
  In 3D graphics, textures applied to objects can be resized using bilinear upsampling to fit different resolutions. This ensures smoother transitions when textures are applied to surfaces at varying distances.

• Convolutional Neural Networks (CNNs):
  In deep learning, particularly in CNNs, bilinear upsampling is used in deconvolutional layers to increase the size of feature maps during tasks like image segmentation or image generation.

• Real-Time Applications:
  Bilinear upsampling is often chosen in real-time applications, like video games or mobile apps, due to its computational efficiency, allowing for smooth image scaling without a significant performance hit.

• Web Design:
  For responsive design, where images need to dynamically resize based on the screen size, bilinear upsampling helps deliver images that look smoother and less pixelated on larger displays.

Key-Takeaways

• Bilinear upsampling is a method for increasing the resolution of an image by calculating new pixel values based on the weighted average of the four nearest pixels.
• Bilinear is faster and simpler than more advanced techniques like bicubic interpolation, but it sacrifices detail and sharpness compared to higher-end methods.
• Commonly used in image processing, computer graphics, and video scaling, especially when performance is a consideration but smoother results are desired over nearest-neighbor methods. It is also often used where computational resources are limited and fast scaling is required.