Region Proposal Object Detection with OpenCV, Keras, and TensorFlow

Last Updated : 29 Mar, 2023

In this article, we'll learn how to implement Region proposal object detection with OpenCV, Keras and TensorFlow.

Install all the dependencies

Use the pip command for installing all the dependencies

pip install tensorflow keras imutils
pip install opencv-contrib-python

Note: Make sure about installing the above OpenCV package otherwise you might face import errors.

Step 1: Read the image and Apply the selective search method of OpenCV

In this step, we'll read the image and apply the selective search method from OpenCV to it. This method will return as a list of rectangles which are basically the region of interest. OpenCV provides us with two different methods for this selective search, one is the "FASTER" method and the other is the "Accurate" method, you have to decide which to use depending on your use case.

Now that we have the rectangles before we go further let's try to visualize what regions of interest it returns.

Python3

import numpy as np
import cv2

# this is the model we'll be using for
# object detection
from tensorflow.keras.applications import Xception

# for preprocessing the input
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications import imagenet_utils
from tensorflow.keras.preprocessing.image import img_to_array
from imutils.object_detection import non_max_suppression

# read the input image
img = cv2.imread('Assets/img2.jpg')

# instantiate the selective search
# segmentation algorithm of opencv
search = cv2.ximgproc.segmentation.createSelectiveSearchSegmentation()

# set the base image as the input image
search.setBaseImage(img)

# since we'll use the fast method we set it as such
search.switchToSelectiveSearchFast()

# you can also use this for more accuracy:
# search.switchToSelectiveSearchQuality()
rects = search.process()  # process the image

roi = img.copy()
for (x, y, w, h) in rects:

    # Check if the width and height of
    # the ROI is atleast 10 percent
    # of the image dimensions and only then
    # show it
    if (w / float(W) < 0.1 or h / float(H) < 0.1):
        continue

    # Let's visualize all these ROIs
    cv2.rectangle(roi, (x, y), (x + w, y + h),
                  (0, 200, 0), 2)

roi = cv2.resize(roi, (640, 640))
final = cv2.hconcat([cv2.resize(img, (640, 640)), roi])
cv2.imshow('ROI', final)
cv2.waitKey(0)

Output:

These are all the Region of Interest that our function receives after filtering out the ROIs which are not sufficiently large, that is to say if the ROI has less than 10% of width or height than that of the image than we won't consider it.

Step 2: Create a List of Final Input Arrays and Bounding boxes using ROI

We'll create two separate lists which contain the image in RGB format and another list will have the bounding box coordinates. These lists will be used for prediction and creating bounding boxes respectively. We'll also make sure that we only make predictions on sufficiently large ROI, say which have at least 20% width or height of our image.

Python3

rois = []
boxes = []
(H, W) = img.shape[:2]
rois = []
boxes = []
(H, W) = img.shape[:2]

for (x, y, w, h) in rects:
  
    # check if the ROI has atleast 
    # 20% the size of our image
    if w / float(W) < 0.2 or h / float(H) < 0.2:
        continue

    # Extract the Roi from image 
    roi = img[y:y + h, x:x + w]
    
    # Convert it to RGB format
    roi = cv2.cvtColor(roi, cv2.COLOR_BGR2RGB)
    
    # Resize it to fit the input requirements of the model
    roi = cv2.resize(roi, (299, 299))

    # Further preprocessing 
    roi = img_to_array(roi)
    roi = preprocess_input(roi)

    # Append it to our rois list
    rois.append(roi)

    # now let's store the box co-ordinates
    x1, y1, x2, y2 = x, y, x + w, y + h
    boxes.append((x1, y1, x2, y2))

Now that we have our region of interests which we have filtered and also preprocessed, let's use them to create predictions using our model.

Step 3: Generate Predictions using Model

We're using the ResNet50 model from the Keras pre-trained models, mainly because it's not heavy on the machine and also has a high accuracy. So, first, we'll create our model instance and then pass in our input -> List of ROIs and generate predictions.

In code it would look like this:

# ------------ Model--------------- #

model = Xception(weights='imagenet')

# Convert ROIS list to arrays for predictions

input_array = np.array(rois)

print("Input array shape is ;" ,input_array.shape)

#---------- Make Predictions -------#

preds = model.predict(input_array)

preds = imagenet_utils.decode_predictions(preds, top=1)

Now, that we have predictions let's s show the result on the image.

Step 4: Create Objects Dictionary

In this step, we'll create a new dictionary that basically contains the label as the key as the bounding box, and probability as the values. This we'll easily let us access the predictions for each label and apply non_max_suppression to them. We can do this by looping through the predictions and filtering out the predictions with more than 90% confidence( you can change it to your needs). Let's see the code:

Python3

# Initiate the dictionary
objects = {}
for (i, pred) in enumerate(preds):
  
    # extract the prediction tuple
    # and store it's values
     iD = pred[0][0]
    label = pred[0][1]
     prob = pred[0][2]
    
  if prob >= 0.9:
    
        # grab the bounding box associated
        # with the prediction and
        # convert the coordinates
        box = boxes[i]

        # create a tuple using box and probability
        value = objects.get(label, [])
        
        # append the value to the list for the label
        value.append((box, prob))
        
         # Add this tuple to the objects 
        # dictionary that we initiated
        objects[label] = value

Output:

{'img': [((126, 295, 530, 800), 0.5174897), ((166, 306, 497, 613), 0.510667), ((176, 484, 520, 656), 0.56631094), ((161, 304, 499, 613), 0.55209666), ((161, 306, 504, 613), 0.6020483), ((161, 306, 499, 613), 0.54256636), ((140, 305, 499, 800), 0.5012991), ((144, 305, 516, 800), 0.50028765), ((162, 305, 499, 642), 0.84315413), ((141, 306, 517, 800), 0.5257749), ((173, 433, 433, 610), 0.56347036)], 'matchstick': [((169, 633, 316, 800), 0.56465816), ((172, 633, 313, 800), 0.7206488), ((333, 639, 467, 800), 0.60068905), ((169, 633, 314, 800), 0.693922), ((172, 633, 314, 800), 0.70851576), ((167, 632, 314, 800), 0.6374499), ((172, 633, 316, 800), 0.5995729), ((169, 640, 307, 800), 0.67480534)], 'guillotine': [((149, 591, 341, 800), 0.59910816), ((149, 591, 338, 800), 0.7370558), ((332, 633, 469, 800), 0.5568006), ((142, 591, 341, 800), 0.6165994), ((332, 634, 468, 800), 0.63907826), ((332, 633, 468, 800), 0.57237893), ((142, 590, 321, 800), 0.6664309), ((331, 635, 467, 800), 0.5186203), ((332, 634, 467, 800), 0.58919555)], 'water_tower': [((144, 596, 488, 800), 0.50619787)], 'barber_chair': [((165, 465, 461, 576), 0.5565266)]}

As you can see it's a dictionary where the label, 'rocking chair', is the key and we have a list of tuple which have bounding boxes and probabilities stored for this label as values.

Step 5: Show the Detected object on the Image

Look at the objects dictionary again if you don't already know, we have multiple bounding boxes for a single label, so won't there be a cluster in the image if directly show it on the image?

Therefore, we need to use the non_max_suppression method which we'll solve this problem for us. But to use this function we need an array of bounding boxes and an array of probabilities, and it returns us an array of bounding boxes.

Python3

# Loop through the labels
# for each label apply the non_max_suppression
for label in objects.keys():
  
    # clone the original image 
    # so that we can draw on it
    img_copy = img.copy()
    boxes = np.array([pred[0] for pred in objects[label]])
    proba = np.array([pred[1] for pred in objects[label]])
    boxes = non_max_suppression(boxes, proba)

    # Now unpack the co-ordinates of the bounding box
    (startX, startY, endX, endY) = boxes[0]

    # Draw the bounding box
    cv2.rectangle(img_copy, (startX, startY), 
                  (endX, endY), (0, 255, 0), 2)
    y = startY - 10 if startY - 10 > 10 else startY + 10

    # Put the label on the image
    cv2.putText(img_copy, label, (startX, y),
                cv2.FONT_HERSHEY_SIMPLEX, 0.45, (255, 0), 2)

    # Show the image
    cv2.imshow("Regional proposal object detection", img_copy)
    cv2.waitKey(0)

Below is the complete implementation:

Python3

# import the packages
import numpy as np
import cv2

# this is the model we'll be using for 
# object detection
from tensorflow.keras.applications import Xception

# for preprocessing the input
from tensorflow.keras.applications.xception import preprocess_input
from tensorflow.keras.applications import imagenet_utils
from tensorflow.keras.preprocessing.image import img_to_array
from imutils.object_detection import non_max_suppression

# read the input image
img = cv2.imread('/content/img4.jpg')

# instantiate the selective search
# segmentation algorithm of opencv
search = cv2.ximgproc.segmentation.createSelectiveSearchSegmentation()

# set the base image as the input image
search.setBaseImage(img)
search.switchToSelectiveSearchFast()

# you can also use this for more accuracy ->
# search.switchToSelectiveSearchQuality()
rects = search.process()  # process the image

rois = []
boxes = []
(H, W) = img.shape[:2]
for (x, y, w, h) in rects:
  
    # check if the ROI has atleast
    # 20% the size of our image
    if w / float(W) < 0.1 or h / float(H) < 0.1:
        continue

    # Extract the Roi from image
    roi = img[y:y + h, x:x + w]
    
    # Convert it to RGB format
    roi = cv2.cvtColor(roi, cv2.COLOR_BGR2RGB)
    
    # Resize it to fit the input requirements of the model
    roi = cv2.resize(roi, (299, 299))

    # Further preprocessing
    roi = img_to_array(roi)
    roi = preprocess_input(roi)

    # Append it to our rois list
    rois.append(roi)

    # now let's store the box co-ordinates
    x1, y1, x2, y2 = x, y, x + w, y + h
    boxes.append((x1, y1, x2, y2))

# ------------ Model--------------- #
model = Xception(weights='imagenet')


# Convert ROIS list to arrays for predictions
input_array = np.array(rois)
print("Input array shape is ;", input_array.shape)

#---------- Make Predictions -------#
preds = model.predict(input_array)
preds = imagenet_utils.decode_predictions(preds, top=1)


# Initiate the dictionary
objects = {}
for (i, pred) in enumerate(preds):
  
    # extract the prediction tuple
    # and store it's values
    iD = pred[0][0]
    label = pred[0][1]
    prob = pred[0][2]

    if prob >= 0.9:

        # grab the bounding box associated
        # with the prediction and
        # convert the coordinates
        box = boxes[i]

        # create a tuble using box and probability
        value = objects.get(label, [])
        
        # append the value to the list for the label
        value.append((box, prob))
        
        # Add this tuple to the objects dictionary 
        # that we initiated
        objects[label] = value


# Loop through the labels
# for each label apply the non_max_suppression
for label in objects.keys():
  
    # clone the original image so that we can 
    # draw on it
    img_copy = img.copy()
    boxes = np.array([pred[0] for pred in objects[label]])
    proba = np.array([pred[1] for pred in objects[label]])
    boxes = non_max_suppression(boxes, proba)

    # Now unpack the co-ordinates of the bounding box
    (startX, startY, endX, endY) = boxes[0]

    # Draw the bounding box
    cv2.rectangle(img_copy, (startX, startY),
                  (endX, endY), (0, 255, 0), 2)
    y = startY - 10 if startY - 10 > 10 else startY + 10

    # Put the label on the image
    cv2.putText(img_copy, label, (startX, y),
                cv2.FONT_HERSHEY_SIMPLEX, 0.45, (255, 0), 2)

    # Show the image
    cv2.imshow("Regional proposal object detection", img_copy)
    cv2.waitKey(0)

Output:

Object Detection with YOLO and OpenCV

bobde_yagyesh

Improve

Article Tags :

Region Proposal Object Detection with OpenCV, Keras, and TensorFlow

Install all the dependencies

Step 1: Read the image and Apply the selective search method of OpenCV

Step 2: Create a List of Final Input Arrays and Bounding boxes using ROI

Step 3: Generate Predictions using Model

Step 4: Create Objects Dictionary

Step 5: Show the Detected object on the Image

Similar Reads

Thank You!

What kind of Experience do you want to share?