In this article, we have explained Dense Layer in Tensorflow with code examples and the use of Dense Layer in Neural Networks.

Table of contents:

1. Introduction to Neural Network
2. What is a Layer?
3. Dense Layer
4. Dense Layer Examples
5. Advantages and Disadvantages of Dense Layer

Let us get started with Dense Layer in Tensorflow.

Introduction to Neural Network

Artifical Neural Network, or usually simply called Neural Networks, is a computing system inspired by how animal brains works. Neural Network "learn" by considering examples without being programmed with any specific rules. Neural Network refer to system of neurons.

Deep Learning is a class of machine learning algorithms that uses multiple layers to progressively extract higher-level features from the raw input, or easy to say, is a "stacked" neural networks, networks that composed of a several layers.

What is a Layer?

Layers are made of nodes, and node is a place where computation happens. A node combines input from the data with set of coefficients called weights, that either amplify or dampen the input.

Keras Layers

Keras is a deep learning API written in Python, running on top of machine learning platform Tensorflow. Keras provides a plenty of pre-built layers for different Neural Network architectures and purposes via Keras Layers API. In this article, we're going to cover one of the most used layers in Keras, and that's Dense Layer.

Dense Layer

Dense Layer is a Neural Network that has deep connection, meaning that each neuron in dense layer recieves input from all neurons of its previous layer.

Dense Layer performs a matrix-vector multiplication, and the values used in the matrix are parameters that can be trained and updated with the help of backpropagation. But we're not going to cover about backpropagation in this article.

The output generated by dense layer is an 'n' dimensional vector. Dense Layer is used for changing dimensions, rotation, scaling, and translation of the vector.

Implementation in Python

import tensorflow as tf

tf.keras.layers.Dense(
    units,
    activation = None,
    use_bias = True,
    kernel_initializer = "glorot_uniform",
    bias_initializer = "zeros",
    kernel_regularizer = None,
    activity_regularizer = None,
    kernel_constraint = None,
    bias_constraint = None,
    **kwargs
    )

Keras Dense Layer Parameters

1. units
   It takes a positive integer as its value. It'll represent the dimensionality, or the output size of the layer.

2. activation
   A function to activate a node. By default, it will use linear activation function (a(x) = x). Keras provides many options for this parameters, such as ReLu.

3. use_bias
   It takes Boolean as its value. It will decide whether the layer use bias or not. By default, use_bias value is set to True.

4. initializers
   The initializer parameter used to decide how values in the layer will be initialized. In Dense Layer, the weight matrix and bias vector has to be initialized. The weight initializer is defined as kernel_initializer and the bias is bias_initializer. Read More about Keras Initializers

5. regularizers
   Regularizers allow you to apply penalties on layer parameters or layer activity during optimization. Dense Layer has 3 regularizers, kernel_regularizer for the weight matrix, bias_regularizer for the bias vector, and activity_regularizer for the output of the layer. Read More about Keras Regularizers

6. constraint
   Constraint determines the constraint on the weight matrix, kernel_constraint, and the bias vector, bias_constraint. Constraint allow setting constraints (eg. non-negativity) on model parameters during training. Read More about Keras Constraints

Dense Layer Operation

Activation is used for performing element-wise activation, and the kernel is the weight matrix, and bias is the bias vector created by the layer.

output = activation(dot(input, kernel) + bias)

Dense Layer Examples

In this section, I will show you examples how to implement Keras using Python by building neural network with dense layer.

1. Shallow Neural Network

Let's build a simplest neural network with single dense layer using Keras model Sequential.

from tensorflow import keras

model = keras.models.Sequential([
    keras.Input(shape = (16, )),
    keras.layers.Dense(32, activation='relu')
])

The units parameter value is 32, so the output shape is expected to be 32, and we use 'relu' or Rectified Linear Unit as its activation function. If we use the summary() method, we will get the how many layers do we have and it's output.

model.summary()

The output :

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 32)                544       
=================================================================
Total params: 544
Trainable params: 544
Non-trainable params: 0
_________________________________________________________________

As we can see above, we only have one Dense Layer with the output shape of 32.

2. Deep Neural Network with Dense Layer

Previously we already see how to make a shallow neural network with only one layer using Dense Layer and Sequential as its model. Now we're going to build a Deep Neural Network with more than one layer using Dense Layer and also Sequential model from Keras.

from tensorflow import keras

model = keras.models.Sequential([
    keras.Input(shape = (16, )),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(16, activation = 'relu'),
    keras.layers.Dense(8, activation = 'relu')
])

If we want to add more layers, we could use the add() method to add more layers.

model.add(keras.layers.Dense(4, activation = 'relu'))

And if we use the same summary() method, we will get the same information as the example above.

model.summary()

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 32)                544       
_________________________________________________________________
dense_2 (Dense)              (None, 16)                528       
_________________________________________________________________
dense_3 (Dense)              (None, 8)                 136       
_________________________________________________________________
dense_4 (Dense)              (None, 4)                 36        
=================================================================
Total params: 1,244
Trainable params: 1,244
Non-trainable params: 0
_________________________________________________________________

In those example above, we use the simplest method to build shallow neural network and deep neural network with simple Dense Layer with no activation, regularization, and constraints.

Advantages and Disadvantages of Dense Layer

Using a fully connected layers serves advantages and disadvantages.

The advantages of Dense Layer is that Dense Layer offers learns features from all combinational features of the previous layer.

But it comes with disadvantages, and that it is incredibly computationally expensive. Because of its expensive computational resource, sometimes it only used to combine the upper layer features.

Conclusion

We already saw what is Dense Layer and how to implement it using Python. There's many use of Dense Layer, but also consider its advantages and disadvantages. Lastly, thanks for reading, and I hope this article could elevate your Machine Learning skills.