In this article, we have explained Dense Layer in Tensorflow with code examples and the use of Dense Layer in Neural Networks.
Table of contents:
- Introduction to Neural Network
- What is a Layer?
- Dense Layer
- Dense Layer Examples
- Advantages and Disadvantages of Dense Layer
Let us get started with Dense Layer in Tensorflow.
Introduction to Neural Network
Artifical Neural Network, or usually simply called Neural Networks, is a computing system inspired by how animal brains works. Neural Network "learn" by considering examples without being programmed with any specific rules. Neural Network refer to system of neurons.
Deep Learning is a class of machine learning algorithms that uses multiple layers to progressively extract higher-level features from the raw input, or easy to say, is a "stacked" neural networks, networks that composed of a several layers.
What is a Layer?
Layers are made of nodes, and node is a place where computation happens. A node combines input from the data with set of coefficients called weights, that either amplify or dampen the input.
Keras is a deep learning API written in Python, running on top of machine learning platform Tensorflow. Keras provides a plenty of pre-built layers for different Neural Network architectures and purposes via Keras Layers API. In this article, we're going to cover one of the most used layers in Keras, and that's Dense Layer.
Dense Layer is a Neural Network that has deep connection, meaning that each neuron in dense layer recieves input from all neurons of its previous layer.
Dense Layer performs a matrix-vector multiplication, and the values used in the matrix are parameters that can be trained and updated with the help of backpropagation. But we're not going to cover about backpropagation in this article.
The output generated by dense layer is an 'n' dimensional vector. Dense Layer is used for changing dimensions, rotation, scaling, and translation of the vector.
Implementation in Python
import tensorflow as tf tf.keras.layers.Dense( units, activation = None, use_bias = True, kernel_initializer = "glorot_uniform", bias_initializer = "zeros", kernel_regularizer = None, activity_regularizer = None, kernel_constraint = None, bias_constraint = None, **kwargs )
Keras Dense Layer Parameters
It takes a positive integer as its value. It'll represent the dimensionality, or the output size of the layer.
A function to activate a node. By default, it will use linear activation function (a(x) = x). Keras provides many options for this parameters, such as ReLu.
It takes Boolean as its value. It will decide whether the layer use bias or not. By default, use_bias value is set to True.
The initializer parameter used to decide how values in the layer will be initialized. In Dense Layer, the weight matrix and bias vector has to be initialized. The weight initializer is defined as kernel_initializer and the bias is bias_initializer. Read More about Keras Initializers
Regularizers allow you to apply penalties on layer parameters or layer activity during optimization. Dense Layer has 3 regularizers, kernel_regularizer for the weight matrix, bias_regularizer for the bias vector, and activity_regularizer for the output of the layer. Read More about Keras Regularizers
Constraint determines the constraint on the weight matrix, kernel_constraint, and the bias vector, bias_constraint. Constraint allow setting constraints (eg. non-negativity) on model parameters during training. Read More about Keras Constraints
Dense Layer Operation
Activation is used for performing element-wise activation, and the kernel is the weight matrix, and bias is the bias vector created by the layer.
output = activation(dot(input, kernel) + bias)
Dense Layer Examples
In this section, I will show you examples how to implement Keras using Python by building neural network with dense layer.
1. Shallow Neural Network
Let's build a simplest neural network with single dense layer using Keras model Sequential.
from tensorflow import keras model = keras.models.Sequential([ keras.Input(shape = (16, )), keras.layers.Dense(32, activation='relu') ])
The units parameter value is 32, so the output shape is expected to be 32, and we use 'relu' or Rectified Linear Unit as its activation function. If we use the summary() method, we will get the how many layers do we have and it's output.
The output :
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense (Dense) (None, 32) 544 ================================================================= Total params: 544 Trainable params: 544 Non-trainable params: 0 _________________________________________________________________
As we can see above, we only have one Dense Layer with the output shape of 32.
2. Deep Neural Network with Dense Layer
Previously we already see how to make a shallow neural network with only one layer using Dense Layer and Sequential as its model. Now we're going to build a Deep Neural Network with more than one layer using Dense Layer and also Sequential model from Keras.
from tensorflow import keras model = keras.models.Sequential([ keras.Input(shape = (16, )), keras.layers.Dense(32, activation='relu'), keras.layers.Dense(16, activation = 'relu'), keras.layers.Dense(8, activation = 'relu') ])
If we want to add more layers, we could use the add() method to add more layers.
model.add(keras.layers.Dense(4, activation = 'relu'))
And if we use the same summary() method, we will get the same information as the example above.
Model: "sequential_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_1 (Dense) (None, 32) 544 _________________________________________________________________ dense_2 (Dense) (None, 16) 528 _________________________________________________________________ dense_3 (Dense) (None, 8) 136 _________________________________________________________________ dense_4 (Dense) (None, 4) 36 ================================================================= Total params: 1,244 Trainable params: 1,244 Non-trainable params: 0 _________________________________________________________________
In those example above, we use the simplest method to build shallow neural network and deep neural network with simple Dense Layer with no activation, regularization, and constraints.
Advantages and Disadvantages of Dense Layer
Using a fully connected layers serves advantages and disadvantages.
The advantages of Dense Layer is that Dense Layer offers learns features from all combinational features of the previous layer.
But it comes with disadvantages, and that it is incredibly computationally expensive. Because of its expensive computational resource, sometimes it only used to combine the upper layer features.
We already saw what is Dense Layer and how to implement it using Python. There's many use of Dense Layer, but also consider its advantages and disadvantages. Lastly, thanks for reading, and I hope this article could elevate your Machine Learning skills.