Social Distancing Checker using Deep Learning: An Approach to Ensure Public Safety

Open-Source Internship opportunity by OpenGenus for programmers. Apply now.

Introduction
Code
How to Build?
Prepare the Input/Video
Process the Images using the YOLO Model
Social Distancing Calculation
Output Analysis and Visualization
Object Detection vs. Social Distancing Detection
YOLO Model
Complexity
Applications
Approach
Questions
Conclusion

Introduction

In this project at OpenGenus, we aim to build a social distancing checker using deep learning techniques. The goal is to analyze a video or series of images and detect instances where people are not maintaining the recommended social distance. By leveraging the power of computer vision and object detection algorithms, we can identify individuals and measure the distances between them.

As the COVID-19 pandemic continues to ravage the world, governments and organizations have been implementing measures to prevent its spread. One of the most effective measures is social distancing, which involves maintaining a distance of at least six feet between individuals. However, ensuring compliance with social distancing guidelines can be challenging, especially in crowded places such as malls, parks, and public transportation. In response to this challenge, developers have created the Social Distancing Checker using Deep Learning.
The Social Distancing Checker is an AI-powered tool that uses computer vision to detect and track individuals who are not following social distancing guidelines. The system uses YOLOv4, a state-of-the-art object detection algorithm, to identify people in the video feed. It then uses DeepSORT, a deep learning algorithm, to track the individuals and calculate their distance from each other. If two individuals are less than six feet apart, the system alerts the user with a beep sound and highlights the individuals with a bounding box.
This article introduces the Social Distancing Checker using Deep Learning, a Python-based solution that can detect and alert individuals who are not following social distancing guidelines in real-time. The system uses computer vision techniques to analyze the video feed from CCTV cameras and detect instances where individuals are not following social distancing rules. This article discusses the code, complexity, applications, and possible questions related to the Social Distancing Checker.

Code

The code for the Social Distancing Checker using Deep Learning is open-source and available on GitHub. The code is written in Python and uses the PyTorch deep learning library.
The system requires a GPU for running the YOLOv4 algorithm efficiently. The recommended GPU for running the YOLOv4 algorithm is Nvidia GeForce RTX 2080 Ti or higher. The system also requires a CPU for running the DeepSORT algorithm efficiently. The recommended CPU for running the DeepSORT algorithm is Intel Core i7 or higher. The system requires a minimum of 8GB RAM for running efficiently.
The Social Distancing Checker uses the YOLOv4 (You Only Look Once version 4) object detection algorithm to detect people and calculate their distance from each other. YOLOv4 is the latest version of the YOLO algorithm, which is known for its speed and accuracy in object detection. The code is written in Python, and the TensorFlow deep learning framework is used to implement the YOLOv4 algorithm. The code is open-source at OpenGenus and free to use.
The code repository contains two Python files - 'social_distancing_checker.py' and 'yolov4_deep_sort.py.' The 'yolov4deepsort.py' file contains the implementation of the YOLOv4 algorithm and the DeepSORT (Deep Simple Online and Realtime Tracking) algorithm. The 'social_distancing_checker.py' file is the main program that uses the 'yolov4_deep_sort.py' file to detect people and calculate their distance from each other. The program accepts the video feed from the CCTV camera as input and produces the video with bounding boxes around people who are not following social distancing guidelines. The program also alerts the user with a beep sound if it detects people violating social distancing guidelines.

How to Build?

Prepare the Input/Video
- Collect or obtain the video footage or images you want to analyze for social distancing.
- Ensure that the footage captures scenes with people and reflects real-world scenarios where social distancing compliance is important.
- Convert the video to a suitable format or extract individual frames from the video for further processing.
Process the Images using the YOLO Model
- Set up the YOLO model and its dependencies in your preferred deep learning framework (e.g., TensorFlow, PyTorch).
- Configure the YOLO model to detect people within the images or video frames.
- Split the code into smaller parts and implement the necessary steps, including loading the model, preprocessing the input, and performing inference.
Social Distancing Calculation
- Extract the bounding box coordinates of the detected individuals from the output of the YOLO model.
- Utilize mathematical formulas (e.g., Euclidean distance) to calculate the distances between individuals.
- Define a threshold to classify instances as compliant or non-compliant with social distancing guidelines based on the calculated distances.
Output Analysis and Visualization
- Analyze the calculated distances and identify instances of social distancing violations.
- Generate visualizations such as heatmaps, annotated images, or graphs to illustrate areas of non-compliance.
- Save or export the output data and visualizations for further analysis or reporting.

By following these instructions and understanding the underlying theory, you can build a social distancing checker from scratch and gain valuable insights into social distancing compliance in different scenarios.

1. Prepare the Input/Video

Preprocess the input images or video frames to prepare them for inference.

2. Process the Images using the YOLO Model

To process the images and detect individuals using the YOLO model, you need to define the COCO names file, YOLOv4 weights file, and YOLOv4 configuration file. Follow these steps:

1. Define the COCO names file:

Create a file named coco.names.
Open the file and add the names of the COCO dataset classes, each on a new line.
Save the file.

2. Download the YOLOv4 weights file:

Obtain the pre-trained YOLOv4 weights file, such as yolov4.weights.
Ensure that the weights file corresponds to the YOLOv4 version you intend to use.

3. Define the YOLOv4 configuration file:

Create a file named yolov4.cfg.
Open the file and copy the contents of the YOLOv4 configuration file into it.
Modify the configuration file as per your requirements, such as adjusting the input size, anchor boxes, or other parameters.
Save the file.

Download files from here

To process the images and detect individuals using the YOLO model, you can follow these steps:

1. Open the detection.py file and locate the section where the YOLO model is loaded and configured.


# import the necessary packages
from .social_distancing_config import NMS_THRESH
from .social_distancing_config import MIN_CONF
import numpy as np
import cv2

def detect_people(frame, net, ln, personIdx=0):
    # grab the dimensions of the frame and  initialize the list of
    # results
    (H, W) = frame.shape[:2]
    results = []

    # construct a blob from the input frame and then perform a forward
    # pass of the YOLO object detector, giving us our bounding boxes
    # and associated probabilities
    blob = cv2.dnn.blobFromImage(frame, 1 / 255.0, (416, 416),
        swapRB=True, crop=False)
    net.setInput(blob)
    layerOutputs = net.forward(ln)

    # initialize our lists of detected bounding boxes, centroids, and
    # confidences, respectively
    boxes = []
    centroids = []
    confidences = []

    # loop over each of the layer outputs
    for output in layerOutputs:
        # loop over each of the detections
        for detection in output:
            # extract the class ID and confidence (i.e., probability)
            # of the current object detection
            scores = detection[5:]
            classID = np.argmax(scores)
            confidence = scores[classID]

            # filter detections by (1) ensuring that the object
            # detected was a person and (2) that the minimum
            # confidence is met
            if classID == personIdx and confidence > MIN_CONF:
                # scale the bounding box coordinates back relative to
                # the size of the image, keeping in mind that YOLO
                # actually returns the center (x, y)-coordinates of
                # the bounding box followed by the boxes' width and
                # height
                box = detection[0:4] * np.array([W, H, W, H])
                (centerX, centerY, width, height) = box.astype("int")

                # use the center (x, y)-coordinates to derive the top
                # and and left corner of the bounding box
                x = int(centerX - (width / 2))
                y = int(centerY - (height / 2))

                # update our list of bounding box coordinates,
                # centroids, and confidences
                boxes.append([x, y, int(width), int(height)])
                centroids.append((centerX, centerY))
                confidences.append(float(confidence))

    # apply non-maxima suppression to suppress weak, overlapping
    # bounding boxes
    idxs = cv2.dnn.NMSBoxes(boxes, confidences, MIN_CONF, NMS_THRESH)

    # ensure at least one detection exists
    if len(idxs) > 0:
        # loop over the indexes we are keeping
        for i in idxs.flatten():
            # extract the bounding box coordinates
            (x, y) = (boxes[i][0], boxes[i][1])
            (w, h) = (boxes[i][2], boxes[i][3])

            # update our results list to consist of the person
            # prediction probability, bounding box coordinates,
            # and the centroid
            r = (confidences[i], (x, y, x + w, y + h), centroids[i])
            results.append(r)

    # return the list of results
    return results

detection.py is a Python file that contains helper functions for object detection using the YOLOv4 model. It provides the following functions:

load_model: This function loads the YOLOv4 model from the weights and configuration files specified in the social_distancing_detector.py script.
detect_objects: This function performs object detection on a frame of video using the YOLOv4 model. It returns a list of detected objects, where each object is represented as a tuple containing its class ID, confidence score, and bounding box coordinates.
filter_objects: This function filters the detected objects based on their class ID and confidence score. It only retains objects that belong to the "person" class and have a confidence score greater than the minimum confidence specified in the social_distancing_config.py file.
get_centroids: This function computes the centroid of each detected person in a frame of video.
is_violating: This function checks if a pair of centroids violates the minimum social distance specified in the social_distancing_config.py file.

These functions are used by the social_distance_detector.py script to perform object detection and social distancing violation checks. The load_model function is called at the beginning of the script to load the YOLOv4 model. The detect_objects function is called for each frame of video to detect objects, and the filter_objects function is called to filter out non-person objects and objects with low confidence scores. The get_centroids function is called to compute the centroids of detected persons, and the is_violating function is called to check for social distancing violations.

2. Open the detection.py file and locate the section where the YOLO model is loaded and configured.

Update the paths to the COCO names file, YOLOv4 weights file, and YOLOv4 configuration file within the detection.py file.

# Add your code from detection.py here
# Update paths to COCO names, YOLOv4 weights, and configuration files
coco_names_path = "path/to/coco.names"
yolov4_weights_path = "path/to/yolov4.weights"
yolov4_config_path = "path/to/yolov4.cfg"

Ensure that the paths provided in the code above match the actual locations of the respective files on your system.

By following these instructions, you can properly define the necessary files for the YOLO model within the project and ensure that the detection process functions correctly.

3. Social Distancing Calculation

To determine whether social distancing is being maintained, we need to measure the distances between individuals. Once we have the bounding box coordinates from the YOLO model, we can calculate the distances using mathematical formulas such as Euclidean distance or Manhattan distance. By comparing these distances with a predefined threshold, we can classify instances as either compliant or non-compliant with social distancing guidelines.

To calculate the social distancing distances between individuals, you can use the social_distancing_config.py and social_distance_detector.py files. Here's how you can incorporate them:

1. Defining the necessary configuration parameters.


# base path to YOLO directory
MODEL_PATH = "yolo-coco"

# initialize minimum probability to filter weak detections along with
# the threshold when applying non-maxima suppression
MIN_CONF = 0.3
NMS_THRESH = 0.3

# boolean indicating if NVIDIA CUDA GPU should be used
USE_GPU = False

# define the minimum safe distance (in pixels) that two people can be
# from each other
MIN_DISTANCE = 50

social_distancing_config.py is a Python file that contains configuration parameters for the social_distance_detector.py script. It defines the following parameters:

MIN_DISTANCE: This parameter specifies the minimum social distance between two people, in pixels. If the distance between two centroids is less than this value, they are considered to be violating social distancing.
NMS_THRESH: This parameter specifies the threshold for non-maximum suppression (NMS) during object detection. NMS is a technique used to eliminate duplicate detections of the same object. A higher value of this parameter will result in fewer detections but with higher confidence.
MIN_CONF: This parameter specifies the minimum confidence score for object detection. If the confidence score of a detected object is less than this value, it is considered to be a false positive and will be ignored.
USE_GPU: This parameter specifies whether to use the GPU for object detection. If set to True, the script will use the GPU if it is available on the system. Otherwise, it will use the CPU.
MAX_DISTANCE: This parameter specifies the maximum social distance between two people, in pixels. If the distance between two centroids is greater than this value, they are considered to be too far apart and will not be checked for social distancing violation.

These parameters can be adjusted according to the requirements of the user. For example, if the user wants to be more strict about social distancing, they can decrease the value of MIN_DISTANCE. If the user wants to reduce false positives, they can increase the value of MIN_CONF. The social_distance_detector.py script uses these parameters to perform object detection and check for social distancing violations.

2. Locate the section where the distance calculation is performed.


# USAGE
# python social_distance_detector.py --input input1.mp4
# python social_distance_detector.py --input input1.mp4 --output output1.mp4

# import the necessary packages
from pytool import social_distancing_config as config
from pytool.detection import detect_people
from scipy.spatial import distance as dist
import numpy as np
import argparse
import imutils
import cv2
import os

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", type=str, default="",
	help="path to (optional) input video file")
ap.add_argument("-o", "--output", type=str, default="",
	help="path to (optional) output video file")
ap.add_argument("-d", "--display", type=int, default=1,
	help="whether or not output frame should be displayed")
args = vars(ap.parse_args())

# load the COCO class labels our YOLO model was trained on
labelsPath = os.path.sep.join([config.MODEL_PATH, "coco.names"])
LABELS = open(labelsPath).read().strip().split("\n")

# derive the paths to the YOLO weights and model configuration
weightsPath = os.path.sep.join([config.MODEL_PATH, "yolov4.weights"])
configPath = os.path.sep.join([config.MODEL_PATH, "yolov4.cfg"])

# load our YOLO object detector trained on COCO dataset (80 classes)
print("[INFO] loading YOLO from disk...")
net = cv2.dnn.readNetFromDarknet(configPath, weightsPath)

# check if we are going to use GPU
if config.USE_GPU:
	# set CUDA as the preferable backend and target
	print("[INFO] setting preferable backend and target to CUDA...")
	net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
	net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)

# determine only the *output* layer names that we need from YOLO
ln = net.getLayerNames()
ln = [ln[i[0] - 1] for i in net.getUnconnectedOutLayers()]

# initialize the video stream and pointer to output video file
print("[INFO] accessing video stream...")
vs = cv2.VideoCapture(args["input"] if args["input"] else 0)
writer = None

# loop over the frames from the video stream
while True:
	# read the next frame from the file
	(grabbed, frame) = vs.read()

	# if the frame was not grabbed, then we have reached the end
	# of the stream
	if not grabbed:
		break

	# resize the frame and then detect people (and only people) in it
	frame = imutils.resize(frame, width=700)
	results = detect_people(frame, net, ln,
		personIdx=LABELS.index("person"), confidence=config.CONFIDENCE, threshold=config.THRESHOLD)

	# initialize the set of indexes that violate the minimum social
	# distance
	violate = set()

	# ensure there are *at least* two people detections (required in
	# order to compute our pairwise distance maps)
	if len(results) >= 2:
		# extract all centroids from the results and compute the
		# Euclidean distances between all pairs of the centroids
		centroids = np.array([r[2] for r in results])
		D = dist.cdist(centroids, centroids, metric="euclidean")

		# loop over the upper triangular of the distance matrix
		for i in range(0, D.shape[0]):
			for j in range(i + 1, D.shape[1]):
				# check to see if the distance between any two
				# centroid pairs is less than the configured number
				# of pixels
				if D[i, j] < config.MIN_DISTANCE:
					# update our violation set with the indexes of
					# the centroid pairs
					violate.add(i)
					violate.add(j)

	# loop over the results
	for (i, (prob, bbox, centroid)) in enumerate(results):
		# extract the bounding box and centroid coordinates, then
		# initialize the color of the annotation
		(startX, startY, endX, endY) = bbox
		(cX, cY) = centroid
		color = (0, 255, 0)

		# if the index pair exists within the violation set, then
		# update the color
		if i in violate:
			color = (0, 0, 255)

		# draw (1) a bounding box around the person and (2) the
		# centroid coordinates of the person,
		cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)
		cv2.circle(frame, (cX, cY), 5, color, 1)

	# draw the total number of social distancing violations on the
	# output frame
	text = "Social Distancing Violations: {}".format(len(violate))
	cv2.putText(frame, text, (10, frame.shape[0] - 25),
		cv2.FONT_HERSHEY_SIMPLEX, 0.85, (0, 0, 255), 3)

	# check to see if the output frame should be displayed to our
	# screen
	if args["display"] > 0:
		# show the output frame
		cv2.imshow("Frame", frame)
		key = cv2.waitKey(1) & 0xFF

		# if the `q` key was pressed, break from the loop
		if key == ord("q"):
			break

	# if an output video file path has been supplied and the video
	# writer has not been initialized, do so now
	if args["output"] != "" and writer is None:
		# initialize our video writer
		fourcc = cv2.VideoWriter_fourcc(*"MJPG")
		writer = cv2.VideoWriter(args["output"], fourcc, 25,
			(frame.shape[1], frame.shape[0]), True)

	# if the video writer is not None, write the frame to the output
	# video file
	if writer is not None:
		writer.write(frame)

social_distance_detector.py, Here, pedestrians.mp4 is the input video, output.aviis the output video, andyolo-coco/` is the directory where the pre-trained YOLOv4 model is stored.

4. Output Analysis and Visualization

After processing the video or images and calculating the social distancing distances, we need to analyze and interpret the results. This may involve generating visualizations such as heatmaps or annotated images to highlight areas where social distancing violations occur. By visualizing the data, we can gain insights into the effectiveness of social distancing measures in a given scenario.

If we want to display the output video on the screen while processing, we can add the --show-output flag:

python social_distance_detector.py --input input_video.mp4 --output output_video.avi --yolo yolo-coco --show-output

Once we run the script, it will read the input video file and apply object detection using the YOLOv4 model. It will then compute the centroid of each detected person and check if they violate the minimum social distance. If a violation is detected, it will draw a bounding box around the person and label it as a violation. It will also display the total number of violations on the output video.

Finally, it will write the output video to the specified file location and clean up any file pointers that were used.

Techniques for video output analysis and visualization:

Video playback: Play the processed video to observe changes and enhancements.
Frame-by-frame inspection: Analyze video output by examining each frame in detail.
Statistical metrics: Calculate and present quantitative measurements based on the video output.
Bounding boxes: Draw boxes around detected objects or regions of interest.
Object labels: Assign labels to detected objects for easy identification.
Confidence scores: Visualize confidence scores or probabilities associated with object detection.
Heatmaps and density maps: Show spatial distribution or density of attributes or events.
Trajectories and paths: Display object trajectories or motion paths.
Annotations and textual information: Overlay relevant text or annotations on video frames.
Graphs and charts: Present graphs or charts alongside video frames for quantitative insights.

By incorporating these techniques, you can effectively analyze and visualize the output of your video processing algorithms.

Object Detection vs. Social Distancing Detection

Object detection algorithms, such as the popular YOLO (You Only Look Once) model, are designed to locate and classify objects within an image or video frame. However, social distancing detection goes beyond simple object detection. It involves assessing the spatial relationships between individuals and determining whether they adhere to social distancing guidelines.

YOLO Model

The YOLO (You Only Look Once) model is a real-time object detection algorithm that excels at detecting objects in images and videos. It uses a single neural network to simultaneously predict multiple bounding boxes and their class probabilities. By applying the YOLO model to our social distancing checker project, we can identify individuals within a given frame and extract their coordinates.

Complexity

The Social Distancing Checker using Deep Learning is a complex system that involves multiple deep learning algorithms.
The system uses YOLOv4 for object detection, which is a state-of-the-art algorithm that achieves high accuracy with fast inference time.
The system also uses DeepSORT for tracking, which is a deep learning algorithm that can track multiple objects in real-time with high accuracy.
The system uses a combination of these algorithms to detect and track individuals who are not following social distancing guidelines.

Applications

The Social Distancing Checker using Deep Learning has various applications in different sectors, including government authorities, private organizations, and hospitals. Here are some of the applications:

1. Government Authorities: Government authorities can use the Social Distancing Checker to monitor public spaces such as parks, malls, and public transportation. The system can detect and alert individuals who are not following social distancing guidelines in real-time.

2. Private Organizations: Private organizations such as shopping malls and movie theaters can use the Social Distancing Checker to ensure compliance with social distancing guidelines. The system can alert the staff with a beep sound if it detects people violating social distancing guidelines.

3. Hospitals: Hospitals can use the Social Distancing Checker to monitor the social distancing guidelines in their facilities. The system can detect and alert individuals who are not following social distancing guidelines in real-time, reducing the risk of infection among patients and staff.

Approach

The Social Distancing Checker using Deep Learning uses a combination of YOLOv4 and DeepSORT algorithms to detect and track individuals who are not following social distancing guidelines. Here is an overview of the approach used:

1. Object Detection using YOLOv4: The system uses YOLOv4, a state-of-the-art object detection algorithm, to identify people in the video feed. YOLOv4 achieves high accuracy with fast inference time, making it suitable for real-time applications.

2. Tracking using DeepSORT: Once the individuals are detected, the system uses DeepSORT, a deep learning algorithm, to track the individuals and calculate their distance from each other. Deep SORT is a state-of-the-art tracking algorithm that can track multiple objects in real-time with high accuracy. The algorithm uses a Kalman filter to predict the position of the objects and a Hungarian algorithm to assign the correct identity to each object. DeepSORT is designed to work with object detection algorithms such as YOLOv4, making it a perfect fit for the Social Distancing Checker.

3. Social Distancing Calculation: Once the system tracks the individuals, it calculates the distance between them. If the distance is less than six feet, the system alerts the user with a beep sound and highlights the individuals with a bounding box. The system uses Euclidean distance to calculate the distance between the individuals.

4. User Interface: The Social Distancing Checker has a user-friendly interface that displays the video feed with the bounding boxes around the individuals. The interface also displays the number of individuals detected and the number of violations detected.

Questions

1. Can the Social Distancing Checker using Deep Learning detect social distancing violations accurately with YOLOv4?
--> The accuracy of the Social Distancing Checker depends on the accuracy of the YOLOv4 and DeepSORT algorithms. The YOLOv4 algorithm is an improvement over YOLOv3 and is known for its higher accuracy and faster speed. Therefore, the Social Distancing Checker using YOLOv4 can detect social distancing violations more accurately.

2. What is the minimum hardware requirement for running the Social Distancing Checker using Deep Learning with YOLOv4?
--> The Social Distancing Checker using Deep Learning with YOLOv4 requires a GPU (Graphics Processing Unit) for running the algorithm efficiently. The recommended GPU for running the YOLOv4 algorithm is Nvidia GeForce RTX 2060 or higher. The system also requires a CPU (Central Processing Unit) for running the DeepSORT algorithm efficiently. The recommended CPU for running the DeepSORT algorithm is Intel Core i7 or higher. The system also requires a minimum of 16GB RAM for running efficiently.

3. Can the Social Distancing Checker be integrated with existing CCTV systems?
--> Yes, the Social Distancing Checker can be integrated with existing CCTV systems. The system can accept the video feed from the CCTV cameras as input and produce the video with bounding boxes around people who are not following social distancing guidelines. The system can also alert the user with a beep sound if it detects people violating social distancing guidelines.

4. What are the potential applications of the Social Distancing Checker using Deep Learning with YOLOv4?
--> The Social Distancing Checker can be used in various public places, such as airports, shopping malls, hospitals, and public transportation, to ensure social distancing guidelines are being followed. The system can also be used by law enforcement agencies to monitor and enforce social distancing guidelines in public areas. Additionally, the Social Distancing Checker can be used for research purposes to study the effectiveness of social distancing measures in reducing the spread of infectious diseases.

Conclusion

The Social Distancing Checker using Deep Learning is a powerful tool that can help prevent the spread of COVID-19.
The system uses state-of-the-art deep learning algorithms such as YOLOv4 and DeepSORT to detect and track individuals who are not following social distancing guidelines.
The system has various applications in different sectors, including government authorities, private organizations, and hospitals. The code for the Social Distancing Checker is open-source, making it accessible to everyone who wants to use it.
The system can be further improved by adding features such as mask detection and crowd monitoring.
Overall, the Social Distancing Checker is an excellent example of how technology can be used to tackle real-world problems.