Binary Step Function

Free Linux Book

Get FREE domain for 1st year and build your brand new site

Binary step function is one of the most common activation function in neural networks out there. But before we get into it let's take a look at what activation functions and neural networks are.

Table of contents:

  1. Brief overview of neural networks and activation functions
  2. What is binary step function?
  3. Conclusion

Brief overview of neural networks and activation functions

Neural networks are a powerful machine learning mechanism that mimic how the human brain learns. Perceptrons are the basic building blocks of a neural network. A perceptron can be defined as anything that takes multiple inputs and produces one output.

Activation functions are mathematical equations that determine the output of a neural network. They basically decide to deactivate neurons or activate them to get the desired output thus the name, activation functions.
In a neural network, input data points are fed into the neuron. Each neuron has a respective weight which are multiplied by the inputs and added to a staic bias value(unique to each neuron).

x = (weight * input) + bias

This is then passed to an apropriate activation function.

Y = Activation function(∑(weight*input) + bias)

The output achieved will be again fed into the neurons in the next layer and the same process is repeated.

Activation functions can be categorized into three main categories:

  • Binary Step Function
  • Linear Activation Function
  • Non-Linear Activation functions
    • Sigmoid function
    • tanh function
    • ReLU function

to name a few.

What is binary step function?

Binary step function is one of the simplest activation functions. The function produces binary output and thus the name binary step funtion. The function produces 1 (or true) when input passes a threshold limit whereas it produces 0 (or false) when input does not pass threshold.

step-func-1-2

f(x) = 1, x>=0
     = 0, x<0

step-func-2

This is the simplest activation function, which can be implemented with a single if-else condition in python

def binary_step(x):
if x<0:
    return 0
else:
    return 1
    
binary_step(5), binary_step(-1)

Output :

(5,0)

Step function is commonly used in primitive neural networks without hidden layer.

The binary step function can be used as an activation function while creating a binary classifier. But the function is not really helpful when there are multiple classes to deal with.
Moreover, the gradient of the step function is zero which causes a hindrance in the back propagation process. That is if you calculate the derivative of f(x) with respect to x, it comes out to be 0.

f'(x) = 0, for all x

Conclusion

In this article at OpenGenus, we have discussed about neural networks and activation functions in brief and also about binary step function, its uses and its disadvantages.