Friday, July 30, 2021
Home » How To's » Make Your Robot SEE

Make Your Robot SEE

Not very long ago, any attempt at robot vision would have been an expensive and very technically challenging task. Digital cameras were once very expensive, the processing requirements would tax even the fastest CPUs, and you would most likely have to write your own imaging processing software from scratch. Today, you can purchase a decent USB webcam for less than $20, and even lowcost PCs and netbooks have significant processing power; and now, software is available that puts PhD-level image processing and robot control capability into the hands of novices.If you have access to a PC and a webcam, I encourage you to download and try RoboRealm ( This powerful vision and robot control software is remarkably easy to use. Ill walk through a simple example that shows an imageprocessing pipeline that can identify and track a colored object. Note: while RoboRealm is written for Windows-based PCs, I have successfully used it on Windows laptops, netbooks and on Macs running Windows in a virtual machine. Most USB and built-in webcams are supported, including Apple iSight cameras with the appropriate drivers.

Photo 1

As shown in photo 1, the RoboRealm user interface is laid out in three main areas with some surrounding controls. On the left is the module catalog. The large area on the top right shows the current image. Below it is the processing pipeline. In this example, I have selected several modules from the catalog and inserted them into an image-processing and control pipeline that tracks a blue object and moves a servo in response.

Photo 2

Selecting modules is as simple as browsing for them in the catalog, using the module index, or searching for them by name or feature. Double-clicking on a module name in the catalog or clicking on the Insert button inserts the module into the processing pipeline.


RoboRealm executes each module in the processing pipeline from top to bottom and then loops back to the top. By default, an image is captured from a built-in webcam or USB camera that can be selected using the Options control. The image is then passed to the first module in the processing pipeline, which may modify it and pass it on to the next module in the pipeline. This process continues until the last module in the pipeline is executed and RoboRealm starts over by grabbing a new image from the camera. Instead of using a camera, you can also use modules that load images from files, read frames from movies, or even grab images from a webcam on your network or across the Internet.

Photo 3

In photo 1, the Camera_Properties module is highlighted, and the image shown is the output of the camera in this case, the built-in iSight camera on my MacBook Pro. The Camera_Properties module allows you to make typical camera adjustments such as exposure, color balance, etc. Double-clicking on a module in the pipeline usually brings up a dialog box with controls for that module. For the Camera_Properties module, I just used the default values so have not shown its control dialog.

Photo 2 shows the Horizontal Flip module highlighted and its control dialog box. The horizontally flipped image is displayed as the output of the selected module. I added this horizontal flip just to make it easier for me to position things in the image. With the horizontal flip, the displayed image behaves like a mirror: if I move my hand to the left, the image of my hand moves to the left as well.

Photo 4

Now we get to more interesting image processing. Photo 3 shows the result of the RGB_Filter with Blue selected. It does a pretty good job of finding the two ends of the blue pen Im holding. If we want to track just the larger end, we need to do a little more.

In photo 4, Ive configured the Blob Size module to ignore color blobs below 120 pixels in area and to only pass on the largest blob. This effectively eliminates everything but the largest blue object in the image.

Photo 5

Photo 5 shows the result of the Center of Gravity module. This module finds the center of the blue blob and draws a box and crosshairs. It also sets the values of RoboRealm variables including COG_X and COG_Y. In this case, COG_X will be set to 204 and COG_Y will be set to 354.

The next module does not do any image processing but shows a nice feature of RoboRealm. The Set Variable module shown selected in photo 6 can evaluate an expression and assign the result to another variable. In this case, were setting panx to the value COG_X/640*3000. This normalizes the X position of the blue blob in camera space and then converts it to a value in the range of 0 to 3000.

Photo 6


Finally, the converted value stored in the panx variable is used to set the position of a servo using the Pololu_Maestro module as shown in photo 7. Here is where RoboRealm really shines. In addition to hundreds of image-processing modules, RoboRealm has dozens of modules for robot control. The Pololu Maestro is a six-channel USB servo controller and just one of many devices that is easily controlled using RoboRealm. The result of this example is a system that controls a servo in response to moving a colored object to the left and right. Although its somewhat simplistic, imagine how much time and effort it might take to do this if you had to write your own software. With RoboRealm, you can literally construct a system like this in minutes.

Photo 7


Ive only scratched the surface of RoboRealms capabilities. Among its more powerful and noteworthy features are: modules that allow the execution of Visual BASIC or Python scripts; robot control modules that support Roomba, Create, Rovio, VEX, Parallax, NXT and RCX robots; and input modules that support devices that include keyboards, joysticks, buttons and SICK laser rangefinders. Many new modules and capabilities are under development. You may also create your own modules using Windows development tools. Extensive help, documentation and tutorials are available within RoboRealm or at RoboRealm is available as a free, 30-day trial from The usual purchase price is $89, but for a limited time, we’ve arranged for Robot readers to save $50 by visiting



Words By George Mitsuoka