What is optical flow and why is it useful? In this article, I am going to answer these two questions and discuss related topics such as the requirements for its application and the state of the art in research.

An optical flow algorithm estimates the displacement of each pixel in an image with respect to a reference image. Before having a closer look at this definition, let me describe some of its uses. Although you will most probably wonder about its industrial applications, I cannot give many examples. The main reason is that optical flow is just at the verge of being applied more widely. Thus, my examples concentrate on other commercial fields in the hope to inspire you to find your own, new ways of applying this method.

Applications
So what can we do with optical flow fields? In machine vision, Bill Silver of Cognex believes that optical flow is a vital ingredient for sub-pixel accuracy: low-resolution high-speed cameras could be applied to virtually increase the resolution of the image while saving money on the system setup.

In large companies such as Robert Bosch and Daimler, driver assistance systems based on cameras either rely on stereo vision, on optical flow, or both. Distances to obstacles can be measured, moving objects can be detected, collisions can be predicted and much more. Another huge application area is medical imaging. Companies such as Siemens and General Electric do a great deal of corporate research in this field. Here, optical flow is often called "registration". It is for example applied to time series for the analysis of tumor growth or for denoising x-ray image sequences.

Another important field occupied for example by Sony, Microsoft and Canon is entertainment: innovative game controllers use motion trajectories of humans as input device, video and photo cameras compensate motion and TV sets virtually create 100Hz videos from frame rates as low as 25Hz.
Companies such as Industrial Light and Magic employ similar optical methods for movie and special effects production. For example, the famous "Trinity Scene" in the movie Matrix (1999) was first shot with a camera array.

The view between each camera pair was interpolated by creating intermediate frames based on an optical flow field. During stereo postproduction, flow fields are used to create depth maps which can be used to vary the baseline of the stereo rig synthetically. This is for example done in the software "Nuke" (The Foundry Visionmongers Ltd.).

As the flow usually changes very much at object boundaries, it can also be used to separate objects from each other. This information is regularly used in video compression standards such as MPEG. In case the recorded scene is not moving or the camera stands still and only a single object moves, we can create 3D reconstructions of the world and therefore turn our camera into a 3D scanner. The result is often not highly accurate, but this can be very useful in scenarios were a large number of devices is needed as might be the case in B2C products. Of course you know Google Street View which uses motion and stereo vision to reconstruct entire cities in 3D.

Very often, optical flow is used as a scientific measurement device. For example, blood pumps are analyzed by seeding small, reflecting particles into a fluid which is inserted into the device. The flow of the fluid is recorded with a high-speed camera. The characteristics of the motion are extracted from flow fields. Another fluid is air. Here, flow fields are used both scientifically and commercially for weather forecasting and the optimization of car and airplane shapes.

Also a promising but not yet fully commercialized application is to measure motion characteristics in surveillance scenarios. Here, suspicious behavior can be detected based for example on the gait of a person which is contained in the flow field this person generates.

Challenges in Optical Flow Estimation
This list of applications is not complete. There are a host of applications both in science and business. On first sight this technology has enormous potential. Yet, there are a number of challenges. In order to see if and how you can apply optical flow in your environment you should know its limits. So what are the most important questions for determining how to implement optical flow?

First there is the so-called brightness model. To measure the optical flow, one usually takes two subsequent images of a sequence. A typical assumption is that the intensity remains constant along each pixel's motion trajectory. When objects typically change their distance to the light source, it might be useful to assume a quadratic change of intensity from one frame to the other. If the illumination changes wildly, you will most likely opt not to use optical flow.