OpenFrameworks Lections

5. OpenFrameworks and OpenCV

Denis Perevalov perevalovds@gmail.com

See in-depth details in my book Mastering openFrameworks Books examples are free, see masteringof.wordpress.com

Analysis and change of image

In openFrameworks you can edit ofImage in a such way: 1. Get image bytes unsigned char * data = image.getPixels (); int w = image.width; int h = image.height; // These are not copied, and delete do not have to! 2. Edit build a new image data1 unsigned char * data1 = new unsigned char * [w * h * 3]; .... Fill data1, using data 3. Set image bytes image.setFromPixels (data1, w, h, OF_IMAGE_COLOR);

// OF_IMAGE_COLOR - 3-byte color. // OF_IMAGE_GRAYSCALE - 1-byte gray. // OF_IMAGE_COLOR_ALPHA - 4-byte, color + transparency.

Analysis and change of image

Byte processing - it is usually inconvenient. Better use of special tools. One of the most convenient for today - the library OpenCV. In openFrameworks have addon - ofxOpenCv. To date (April 2011) he works withOpenCV 1.x. This version of OpenCV is inconvenient fact that has only a C-interface. Therefore, we study a more convenient OpenCV 2.x and we use it, without the addon ofxOpenCv.

OpenFrameworks and OpenCV

In OpenCV image given by the class Mat. How to convert ofImage in Mat and back to the example of 3-channel images:

From openFrameworks in OpenCV

ofImage image; //-let it is, 3-channel ... Mat imageCV (cv:: Size (image.width, image.height), CV_8UC3, image.getPixels ()); From OpenCV in openFrameworks Mat imageCV; //-the way it is, 3-channel ... image.setFromPixels ((unsigned char *) IplImage (imageCV). imageData, imageCV.size (). width, imageCV.size (). height, OF_IMAGE_COLOR);

Introduction to OpenCV
- What is OpenCV - The first project to OpenCV - Class Mat - Image processing functions

What is OpenCV

"Open Computer Vision Library" Open library with a set of functions for processing, analysis and image recognition, C / C + +.

What is OpenCV
2000 - First alpha version, support for Intel, C-interface 2006 - Version 1.0 2008 - Support Willow Garage (lab. Robotics) 2009 - version 2.0, classes in C + + 2010 - version 2.2, realized work with the GPU

The first project to OpenCV 1. Creating a Project

We assume that Microsoft Visual C + + 2008 Express Edition and OpenCV 2.1 is already installed. 1. Run VS2008

2. Create a console project File - New - Project - Win32 Console Application, in the Name enter Project1, click OK.
3. Set up the path Alt + F7 - opens the project properties Configuration Properties - C / C + + - General - Additional Include Directories, where we put the value "C: \ Program Files \ OpenCV2.1 \ include \ opencv"; Linker - General - Additional Library Directories, where we put the value of C: \ Program Files \ OpenCV2.1 \ lib \

Linker - Input - Additional Dependencies cv210.lib cvaux210.lib cxcore210.lib cxts210.lib highgui210.lib for Release, cv210d.lib cvaux210d.lib cxcore210d.lib cxts210.lib highgui210d.lib for Debug

The first project to OpenCV 2. Reading the image and display it on screen
1. Preparing the input data: file http://www.fitseniors.org/wp-content/uploads/2008/04/green_apple.jpg write in C: \ green_apple.jpg 2. Writing in Project1.cpp: # Include "stdafx.h" # Include "cv.h" # Include "highgui.h" using namespace cv; int main (int argc, const char ** argv) { Mat image = imread ("C:\\green_apple.jpg");// Load image from disk imshow ("image", image); // Show image waitKey (0); // Wait for keystroke return 0; } 3. Press F7 - compilation, F5 - run. The program will show the image in the window and by pressing any key will complete its work.

The first project to OpenCV 3. Linear operations on images

Replace the text in the main from the previous for example: int main (int argc, const char ** argv) { Mat image = imread ("C:\\green_apple.jpg"); // Image1 pixel by pixel is equal to 0.3 * image Mat image1 = 0.3 * image; imshow ("image", image); imshow ("image1", image1); waitKey (0); return 0; }

The first project to OpenCV 4. Work with rectangular subdomains image

Replace the text in the main from the previous example to: int main (int argc, const char ** argv) { Mat image = imread ("C:\\green_apple.jpg"); // Cut of the picture Rect rect = Rect (100, 100, 200, 200); // Rectangle cut Mat image3; image (rect). copyTo (image3); // Copy of the image imshow ("image3", image3); // Change the part of the picture inside the picture image (rect) *= 2; imshow ("image changed", image); waitKey (0); return 0; }

Mat Class
Mat - Base class for storing images OpenCV.

Mat Class Single-and multi-channel images

The image is a matrix of pixels. Each pixel can store some data. If the pixel stores the vector data, the dimension vector is number of image channels. 1-channel image - also called the half-tone 3-channel images - typically consist of three components (Red, Green, Blue).

Also, OpenCV can be used 2 - and 4-channel image.

Mat Class Creating images

1) Let the picture without some type of
Mat imageEmpty; 2) Image w x h pixels, the values 0 .. 255 (8Umeans "unsigned 8 bit",C1means "a channel"): int w = 150; int h = 100; Mat imageGray (cv:: Size ( w, h ) CV_8UC1 );

Mat Class Creating images

3) 1-channel with the floating-point values (32F means "float 32 bit"):

Mat imageFloat (cv:: Size (w, h), CV_32FC1 );

4) 3-channel image with values 0 .. 255 for each channel: Mat imageRGB (cv:: Size (w, h), CV_8UC3 );

Mat Class Memory management

1. Memory for the image stands out and is automatically cleared That is, OpenCV itself creates the image of the desired size and type, if this image is an output parameter of a function: Mat imageFloat; imageGray.convertTo (imageFloat, CV_32FC1, 1.0 / 255.0); - Here OpenCV itself allocate imageFloat. It is important that if your image is already the right size, there are no operations on memory allocation is performed. 2. Assignment operator shall not copy the data (as does the std:: vector), and not by copying pointers, and using mechanism of the reference count.

Mat Class Memory management

The mechanism of the reference count (In STL is a shared_ptr, in Java it is all signposts) works like this: { Mat A (cv:: Size (100, 100), CV_8UC1); // Allocate memory for the image, and the memories, // This memory is a single image. { Mat B = A; // Here the memory for the image does not stand out, but simply // Data in B point to the same area in memory. // Therefore, if we change B, then changed, and A. // Reference count increased by an image, was equal to 2. } // Here B came out of scope, the reference count is decreased, // And became equal to 1. } // Here A came out of scope, the reference count becomes equal to 0, // And the memory allocated to it are automatically cleared.

Mat Class Memory management

Since the operation Mat B = A; does not copy the image A to B, then in order to create a copy of the image for subsequent independent use, you must use explicit commands copyTo andclone: image1.copyTo (image2); image3 = image1.clone ();

Mat Class Memory management

Outcome: 1) an assignment Mat B = A; is very fast, and does not copy the data and adjusts the pointers in a special way to them. This allows you to transfer Mat in the function directly, without pointers and references. This will not cause unwanted copying Mat the stack (as it would stalal std:: vector). Although, of course, const Mat & will be transmitted still faster. 2) to copy the images to use explicit commands copyToand clone.

Mat Class Per-pixel access to images

In OpenCV has several ways of per-pixel access to images. They vary in the degree of security (typing and go beyond the border), the speed and convenience. Wherever possible, you should try to avoid direct references to the pixels, but instead use the functions of OpenCV, since they usually work faster and the code more understandable.

Mat Class Per-pixel access to images

One way to access the pixels for images that have known the type - the use of theat. For single-channel images 0 ... 255 it is: // Get values int value = imageGray.at <uchar> (y, x); // Set the values imageGray.at <uchar> (y, x) = 100; Note that x and y in the call are swapped.

Mat Class Conversion types

Note In the derivation of the on-screen images with floating-point OpenCV means we must bear in mind that they are displayed on the assumption that their values lie in [0,1]. Therefore, when converting 8-bit images in an image float to do the transformation - the multiplication by 1.0 / 255.0. To convert images of different types of bit mode (float and unsigned char) used a class member convertTo. In its second argument - the type of the image.

imageGray.convertTo (imageFloat, CV_32FC1, 1.0 / 255.0);

The number of channels input and output must match!

Mat Class Conversion types

For converting different color spaces using the function cvtColor. If necessary, it can change the number of channels. For example, the conversion of 3-channel RGB-image to grayscale: cvtColor (inputRGB, outputGray, CV_BGR2GRAY); on the contrary: cvtColor (inputGray, outputRGB, CV_GRAY2BGR);

Mat Class Partition of the channels

Function split divides the multi-channel image into channels. Functionmergestitches together a single-image multi-channel. voidsplit(Const Mat &mtx// Original color image vector <Mat> &mv// Result set is 1-channel // Images ) voidmerge(Const vector <Mat> &mv// Initial set of 1-channel // Images Mat &dst// The resulting color // Image ) Most often they are used to separately to each color image processing, as well as for various manipulations of the channels.

Mat Class Partition of the channels

int main (int argc, const char ** argv) { Mat image = imread ("C:\\green_apple.jpg"); // Split the original image into three channels // - Channels [0], channels [1], channels [2]

vector <Mat> channels; split (image, channels);

// Show the channels in separate windows // Note that the red channel - 2, not 0.

imshow ("Red", channels [2]); imshow ("Green", channels [1]); imshow ("Blue", channels [0]); waitKey (0); return 0; }

Image processing functions Smoothing

Original image

The image, smoothed box 11 x 11

FunctionGaussianBlurperforms image smoothing Gaussian filter.

Most often, the smoothing is applied to remove small noise on the image for subsequent image analysis. Is done by using a filter of small size.

http://www.innocentenglish.com/funny-pics/best-pics/stairs-sidewalk-art.jpg

Image processing functions Threshold

Functionthresholdperforms threshold processing of the image.

Most often it is used to highlight objects of interest pixels in the image.

http://www.svi.nl/wikiimg/SeedAndThreshold_02.png

Image processing functions Fill areas

FunctionfloodFillprovides a fill area, starting from a pixel (x, y), with specified boundaries shutdown using a 4 - or 8 - adjacency pixels. It is important: It spoils the original image - as it fills. Most often it is used to highlight areas identified by the threshold processing, for subsequent analysis.
http://upload.wikimedia.org/wikipedia/commons/thumb/5/5e/Square_4_connectivity.svg/300px-Square_4_connectivity.svg.png http://tunginobi.spheredev.org/images/flood_fill_ss_01.png

Image processing functions Isolation circuits

The contour of the object - this is the line representing the edge of the object's shape. Underline the contour points -Sobel, Leased-line circuit -Canny. Application 1. Recognition. Along the contour can often determine the type of object that we observe. 2. Dimension. With the circuit can accurately estimate the size of the object of their rotation, and location.
http://howto.nicubunu.ro/gears/gears_16.png http://cvpr.uni-muenster.de/research/rack/index.html

Sample project on OpenCV: Search for a billiard ball

Problem
The image of billiard field to find the coordinates of the centers of billiard balls. Algorithm:

1. Threshold find bright pixels. 2. Analysis of areas. The method of casting we find connected regions, Among them we find such dimensions that allow it balls.

1. Threshold
Problem - the image of billiard field highlight the pixels that are not field (shooting conditions such that the field - dark) Mat image = imread ("C:\\billiard.png"); // load the input image imshow ("Input image", image); vector <Mat> planes; split (image, planes); Mat gray = 0.299 * planes [2] + 0.587 * planes [1] + 0.114 * planes [0]; double thresh = 50.0; // The threshold is chosen empirically threshold(Gray, gray, 50.0, 255.0, CV_THRESH_BINARY); imshow ("Threshold", gray);

threshold - an example of application

Please note: We have identified just pixels "not" field. To find the coordinates of the centers of balls and cue position - requires further processing.

2. Analysis areas
floodFill- Allocation of connected regions

morphological operations dilate - dilation(*) erode - Erosion(*)

floodFill - description
Function floodFillprovides a fill area, starting from a pixel (x, y), with specified boundaries shutdown using a 4 - or 8 - adjacency pixels.

Important: it spoils the original image - as it fills.

1. Most often it is used to highlight areas identified by the threshold processing, for subsequent analysis. 2. It can also be used to remove small noise on the binary image (in contrast to the "erosion + dilation" - do not spoil the boundaries of larger areas). 3.If enhance overall box found in the area of 1 pixel on all sides and make the fill, the way you can eliminate the internal hole in the area.

floodFill - a list of options

Announcement and description of the parameter list: int floodFill(Mat & image, Point seed, Scalar newVal, Rect * rect= 0 Scalar loDiff= Scalar (), Scalar upDiff= Scalar (), int flags= 4)
image - The input image, 1 - or 3-channel, 8 or 32-bit. seed - Pixel, from which to start pouring rect - Bounding box found by the field loDiff, UpDiff - allowable difference with its neighbors (Or - with embryonic pixel, if flags | = FLOODFILL_FIXED_RANGE) that is, a new pixel must satisfy valueNew value - loDiff <= valueNew <= value + upDiff. flags = 4 or 8 - connectivity. The resulting value - the number of pixels in the flooded area.

floodFill - a list of options

Note about the types of OpenCV: Point - Integer point to the fields int x, y; Rect - A rectangle with integer fields int x, y, width, height; Scalar - The representation of color, For example, Scalar (255) - 1-channel color Scalar (255, 255, 255) - 3-channel color

2. Analysis areas
Problem - the image of billiard glades find billiard balls - ie. compute their centers and sizes. The idea - using the example of the result threshold, through all connected regions with floodFill, and the found areas to consider those balls whose sizes lie in the predefined boundaries. const int minRectDim = 25; // Max and min size of the balls const int maxRectDim = 35; // Iterate over the image pixels for (int y = 0; y <gray.rows; y + +) { for (int x = 0; x <gray.cols; x + +) { int value = gray.at <uchar> (y, x); if (value == 255) {// If the value of - 255, fill it 200 Rect rect;// Here is written Bounding Box int count = floodFill(Gray, Point (x, y), Scalar (200), & rect);

Analysis areas
// Check size if (rect.width> = minRectDim && rect.width <= maxRectDim && Rect.height> = minRectDim && rect.height <= maxRectDim) { // Center int x = rect.x + rect.width / 2; int y = rect.y + rect.height / 2; // Radius int rad = (rect.width + rect.height) / 4; // Draw a circle the thickness of 2 pixels circle (image, Point (x, y), rad, Scalar (255, 0, 255), 2); } } } } imshow ("out", image);

floodFill - example of application

Comments
In this example, we considered the simplest method for finding the ball in the picture - by analyzing the sizes of bounding boxes. Such an analysis works on the assumption that the image no other sites with similar bounding boxes. For a real application, a more detailed analysis of areas. This is primarily due to the fact that if the balls are near each other, then they can "stick together" in one connected region. Possible approaches to solving this problem: 1. To fill the interior area, select the path obtained by field and assess its areas of convexity and concavity for the selection of balls. 2. Use template "round", which is applied to the obtained area and look for the best of its location.

Debugging in OpenCV
To debug the project, which is being processed by OpenCV, very useful to display intermediate images by usingimshow imshow ("image", image); - It displays the image in the image window with the heading "image". Warning: 1. need # Include "highgui.h" 2. if you display images in the window with the same name, then only the last image will be visible.

Homework
Do a project on openFrameworks, which 1) receive a picture from the camera 2) then this picture is transmitted in OpenCV, where she - Smoothed - Is the threshold processing 3) The picture with the camera and the resulting image is displayed on the screen using openFrameworks.