In this article, we have explored the idea of Tensors in TensorFlow, different types of tensor and how to initialize and use them.

Table of contents:

1. Introduction
2. Tensors in tensorflow
3. Properties of Tensors
4. Different types of Tensors in TensorFlow
   • Variable
   • Constant
   • Placeholder
   • Sparseholder
5. Conclusion

Pre-requisite:

• Key ideas in TensorFlow

Introduction

Earlier on, Artificial Intelligence(A.I) algorithms and models were built from scratch, with hundreds of thousands of lines of code which took forever to compile only to have errors that weren't the easiest to detect. The computational costs of running that much code, the erros from running that much code, which included syntax errors or errors in the mathematics of the algorithms inspired research into libraries and frameworks for these purposes. One of these libraries is tensorflow.

Tensorflow is a free, open-source library for Machine Learning(M.L) and Artificial Intelligence. It is most popularly used with the python language but offers support for other languages like: Java, JavaScript and C++. Although it can technically be used on a wide array of tasks in M.L and A.I, it is especially useful for training, inference and deep neural networks.

In this article, we will talk about the way tensorflow works, spcefically how to initialize tensors.

Tensors in tensorflow

Tensorflow programs work by building graphs of objects(tensors) that show the relation between tensors and the arithmetic(operations) linking them. A tensor is a generalization of vectors and matrices to potentially higher dimensions by mathematical manipulation (i.e: algorithms or models). Tensorflow represents tensors as N-dimensional matrices.

For readers that may be familier with MATLAB, it essentially works like MATLAB; by manipulating matrices. Infact, scaler variables are technically represented as 1x1 matrices in tensorflow which can be changed to potentially higher dimensions depending on how you initialize them. In this article, we will attempt to explain the different ways to initialize tensors.

Properties of Tensors

The elements in tensors could be of different datatypes as defined and allowed by the language of use. However, tensors themselves can be of different datatyes, i.e.: float32, float64, int32, int16, string, etc. They also have shapes, which represents the dimensions of the vector or matrix. See below some examples illustrating this, for now ignore the way the tensor is initialized and focus on the attributes.

import tensorflow as tf

string_tensor = tf.Variable("We are OpenGenius", tf.string)
integer_tensor = tf.Variable(44, tf.int32)
float_tensor = tf.Variable(55.5, tf.float64)

We have initialzed 3 tensors, the string_tensor is a 1x1 tensor containing the element "We are OpenGenius" as a string of alphanumeric characters and spaces. The integer_tensor is also a 1x1 tensor containing the element 44 as a 32bit integer, likewise the float_tensor contains a single element of 55.5 as a 64bit floating point.

The rank of a tensor is the number of linearly independent columns in the tensor. In tensorflow, we can just simply refer to the rank as the total number of different dimensions of the tensor minus 1. To determine the rank of a tensor we call the tf.rank(tensor_name). Note that the output of the tensor has a datatype(dtype) of the default int32. This is the datatype of the tensor as a whole, but the individual elements are what we define with tf.datatype.

rank_1 = tf.Variable([5, 4, 3, 2, 1], tf.int16)
rank_2 = tf.Variable([[2, 4, 2], [7, 3, 8]], tf.int16)
also_rank_2 = tf.Variable([[5,8,0,7],[6,4,8,0],[5,2,8,6],[8,5,4,1]], tf.float32)
rank_3 = tf.Variable([[[3, 6], [5,4]], [[4,1], [3,2]]], tf.float64)

tensor_list = [string_tensor, integer_tensor, float_tensor, rank_1, rank_2, also_rank_2, rank_3]

for tensor in tensor_list:
    print(tf.rank(tensor))
    
# Output
# tf.Tensor(0, shape=(), dtype=int32)
# tf.Tensor(0, shape=(), dtype=int32)
# tf.Tensor(0, shape=(), dtype=int32)
# tf.Tensor(1, shape=(), dtype=int32)
# tf.Tensor(2, shape=(), dtype=int32)
# tf.Tensor(2, shape=(), dtype=int32)
# tf.Tensor(3, shape=(), dtype=int32)

In the code above, I defined 3 tensors of different ranks and added them to a list of tensors including the string, integer and float tensors we defined before. The code, then loops through this list and prints out the rank, shape and datatypes of each tensor. We can hence infer that, intuitively,the rank of a matrix is simply the deepst level of a nested list. For rank_2, there is a list of numbers withing the overal list, i.e. [[a,b],[c,d], [e,f], ....]. For rank_3, there are lists that form individual lists within the overall list, i.e, say a, b, c, d, e, f, ... were also lists(a = [3,6], b=[4,5], etc). If you keep thinking about how deep a matrix can go, we conclude that the rank of a matrix is less than or equal to the minimum number of rows/columns of the matrix. So a 2x3 matrix can have a rank of 1 or 2, a 4x4 matrix can have a rank of 1, 2, 3, or 4.

For further intuition, see the exerpt from wolframalpha. Say a tensor represents any physical quantity with magnitude and other properties(let's simply consider that its a object with mass moving in diffferent directions). The number of different quantities(directions of motion) is the rank of the tensor. Hence, a tensor with just one number, i.e. just the magnitude(mass) is a scalar with 0 other proterties(rank-0). Simply put, given a plane with N-dimesions, a rank-n tensor is represented by N^n numbers. A rank-0 tensor has N^0 = 1 numbers, rank-1 has N^1 = N numbers, and a rank-2 tensor has N^2 = NxN numbers(hence any square matrix, NxN is a rank 2 tensor).

The shape of a tensor tells us the number of items we have in each dimension of the tensor. For a 3 degree matrix(i.e., tensor of rank 3), the shape will have 3 numbers(x, y, z), each refering to the number of elements in that dimension. So rank_3 has a shape of (2, 2, 2), 2 seperate sets of, 2 lists of matrices with 2 elements. See below for other lists of tensors and their shapes using tf.shape(tensor_name).

two_three_six = tf.Variable([[[7, 2, 6, 3, 7, 9], [9, 0, 2, 7, 4, 6], [4, 1, 3, 9, 6, 4]], [[8, 3, 3, 7, 5, 10], [4, 2, 8, 7, 5, 1], [5, 1, 10, 5, 8, 7]]], tf.float64)
three_two_one = tf.Variable([[[3], [5]], [[4], [3]], [[6], [10]]], tf.float64)

tensor_list = [string_tensor, integer_tensor, float_tensor, rank_1, rank_2, also_rank_2, rank_3,two_three_six, three_two_one]

for tensor in tensor_list:
    print(tf.shape(tensor))


# Output
# tf.Tensor([], shape=(0,), dtype=int32)
# tf.Tensor([], shape=(0,), dtype=int32)
# tf.Tensor([], shape=(0,), dtype=int32)
# tf.Tensor([5], shape=(1,), dtype=int32)
# tf.Tensor([2 3], shape=(2,), dtype=int32)
# tf.Tensor([4 4], shape=(2,), dtype=int32)
# tf.Tensor([2 2 2], shape=(3,), dtype=int32)
# tf.Tensor([2 3 6], shape=(3,), dtype=int32)
# tf.Tensor([3 2 1], shape=(3,), dtype=int32)

For any tensor, the tensor can be reshaped into different dimensions that have a prduct equal to the product of the original tensors dimensions. So a 3,2,1 tensor (product= 6) ,can be reshaped into a 2, 3, 1 tensor or a 6, 1 tensor or a 1, 6 tensor.

ones = tf.ones([1,2,3])
reshaped_ones = tf.reshape(ones, [6, 1])
reshaped_auto = tf.reshape(ones, [3, -1]) # -1 tells tensorflow to predict the second dimension to make a valid and compatible reshape, whoch in this case is 2(3x2 = 6)
print(ones)
print(reshaped_ones)

# Ouput
# tf.Tensor(
# [[[1. 1. 1.]
#   [1. 1. 1.]]], shape=(1, 2, 3), dtype=float32)
# tf.Tensor(
#  [[1.]
#  [1.]
#  [1.]
#  [1.]
#  [1.]
#  [1.]], shape=(6, 1), dtype=float32)

Play around with these attributes, reshape them, change their datatypes etc. until you fully understand how they work and what they mean. Now, we will talk about the types of tensors you can initialize.

Different types of Tensors in TensorFlow

There are essentially 4 useful types of tensors that can be initialized:

• Variable
• Constant
• Placeholder
• Sparseholder

Variable Tensor in TF

A variable tensor is one that can be changed during the course of the execution of the code. They are the variables in your project that are subject to change, you may later add to it, subtract from it, increase its scale, or delete it and change it to something else all together. Regardless, once a variable is subject to manipulation, tf.Variable is the way to inialize it.

Constant Tensor in TF

A constant tensor is different from a variable tensor in that it is immutable (cannot be changed during the course of the execution of the code). In fact, all of these tensors are immutable except the variable tensor. Initialized by tf.constant(), these tensors cannot be changed at all during code execution. They can be called, and used to change other variables, for arithmetic manipualtion (or otherwise), but they themselves remain the same from start to finish. It is good to note that constant tensors can have their attributes changed.

In practice, tf.Variable, and tf.Constant are the most popularly used. However, it won't help to know about tf.placeholder() and tf.sparseholder(), in case they become useful to us in the future.

Placeholder Tensor in TF

A placeholder is simply a tensor that we will assign data to
later on. It allows us to create our operations and build
our computation graph, even when we don't have the data. It does this but using dummy data - we feed data into the graph through these
placeholders. In practice, a variable tensor can also be used with dummy data like an identity matrix, to build the initial tensorboard graph and then later updated. It's just good to know that the placeholder tensor was designed for this functionality.

Note: This tensor will produce an error if evaluated. Its value must be fed using the feed_dict optional argument to Session.run(),Tensor.eval(), or Operation.run().

Sparse Tensor in TF

A sparse tensor is one with majority of its elements as 0. A sparseholder is hence a place holder for a sparse tensor.

const = tf.constant([[1,2,3], [4,5,6]], name='const') # Immutable in the course of execution
var = tf.Variable([[7,8,9], [10,11,12]], name ='var') # Can be changed in the course of execution
print(const)
print(var)
const = tf.multiply(const, var)
print(const)

# Output
# tf.Tensor(
# [[1 2 3]
#  [4 5 6]], shape=(2, 3), dtype=int32)
# <tf.Variable 'var:0' shape=(2, 3) dtype=int32, numpy=
# array([[ 7,  8,  9],
#        [10, 11, 12]], dtype=int32)>
# tf.Tensor(
# [[ 7 16 27]
#  [40 55 72]], shape=(2, 3), dtype=int32)

A constant tensor named "const" and a variable tensor named "var" are initialized. Later they are multiplied (elementwise multiplication), and stored in const. "But I thought const cannot be changed during the course of the execution of the code?", yes but now const is pointing to a different tensor, the one as a result of multiplying the teo initialized tensors. The immutability of a tensor can be observed if linked to a graph. The tensor cannot be changed in TensorFlow steps like Graph optimizations while a normal tensor can be changed in-place.

Bonus tip:

You can generate a tensor of just ones of zeros by running thre command tf.ones(shape, dtype=tf.dtypes.float32, name=None), and tf.zeros(shape, dtype=tf.dtypes.float32, name=None).

Conclusion

The tensorflow library is stores vectors and matrices in the form of tensors which are objects to be manipulated during code execution. Tensorflow does this by generating a graph of the flowchart of the tensors are how they are linked to each other during the course of the execution of the code.

There are different ways to initialize different types of tensors based on their use and nature. You can also set and change certain attributes if the tensors like the rank, shape, datatype, name etc. See the actual tensorflow website for more detail for initializing tensors, updates on the library and specific questions to help with your ML projects(https://www.tensorflow.org/).