A real-time image processing framework with an aerial overhead camera for sports

Recently, large horizontal interactive surfaces have begun to be developed. In these systems, an overhead camera is often used to detect the position of objects on the surface even if they are not in contact with the surface. However the issues that these systems face are that they are expensive and a camera cannot be easily attached on top of the surface in some situations. This paper proposes a framework that uses a camera on a drone (UAV) as an overhead camera unit to convert arbitrary horizontal rectangular regions into interactive surfaces. Although commercially available drones that are equipped with cameras have high latencies and are difficult to use in real-time interactive systems, we solved this latency issue using a small PC that performs primitive image processing tasks onboard. First, we describe a drone unit that has an infrared camera and a small PC for real-time image processing, such as surface detection and object detection. Second, we describe novel infrared markers for the robust detection of the four corners of a rectangular region and the objects within that region. Finally, we describe an interactive sports coaching application in which a drone unit is used as an overhead camera both for a large playing field and small tabletop.