2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 Large-Scale Immersive Displays in Entertainment and Education Ed Lantz Spitz, Inc. Abstract. Large-scale immersive displays have an established history in planetaria and large-format film theaters. Video-based immersive theaters are now emerging, and promise to revolutionize group entertainment and education as the computational power and software applications become available to fully exploit these environments. Requirements for an effective visual display are developed. Limitations of commercial projection and image generation technologies are discussed and improvements are suggested. Trade-offs between flat, cylindrical, and spherical projection screens are discussed. Recent work is presented in group telepresence and interactive VR Cinema. Ongoing issues include group interactivity paradigms, show production tools, and the need for research establishments to disseminate compelling source material to public venues. Research topics are suggested in Human Factors, Virtual Reality, Computer Graphics and Display Engineering. Entertainment and Education Entertainment and informal education applications result in display requirements which differ from visualization systems research or collaborative design. Such applications include traditional planetaria, science centers, aquariums, zoos, corporate theaters, visitor centers, location-based entertainment and community special-venue theaters. Emphasis is placed on the overall quality of visitor experience. The entire theater is intergral to visitor experience, of which the information display is but one element. General theater design considerations include the following. Large-scale immersive displays have been in use since the first Zeiss planetarium in 1926. The popular IMAX® Dome format, first demonstrated in 1973, utilizes 70mm film with roughly 5000x4000 line resolution on a dome screen. Simulator rides utilize 35mm or 70mm film projected onto partial dome screens. Over 2500 dome theaters are now in place worldwide. Immersive displays utilizing wrap-around screens are well suited for public presentations as they require no special viewing skills, unlike other VR technologies such as head-mounted displays [1]. Emerging video-based immersive theaters are capable of providing real-time experiences for large groups including guided tours through popular virtual environments, virtual sports, group telepresence and interactive simulations[2]. • Transparency - The theater is designed to “disappear” during the presentation by using dark and non-obtrusive finishings. Projection systems and computer equipment are hidden from view and acoustically isolated. Visual cues associated with the projection surface are minimized (i.e. seamless projection screen). • Comfort - Seating is arranged to assure good sight lines and comfortable viewing for extended periods. • Geometry - Since an entire group cannot simultaneously occupy the ideal eye-point of an immersive display, a projection geometry must be adopted which has a graceful degredation in orthoscopy as the viewer moves off-axis. Geometry should be “acceptable” from all paid seats. Immersive video “digital dome” theater 1 2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 • Interactive Ergonomics - Whatever method is used for group interaction with the immersive experience must be easy to learn, simple to use and accessible to a wide range of skill levels. screen and also affords a degree of acoustic transparency. Ambient sound can therefore penetrate the screen and be absorbed rather than reflected (and focused) back into the theater space. Also, loudspeakers can be freely placed behind a perforated screen. • Audio - Theatrical surround audio is common to all modern theaters and is key to a compelling experience. Screen diameter is often driven by the required seating capacity. More seats can be placed in a given dome if a greater compromise in visitor experience (i.e.greater geometric distortion) is tolerated. In general, smaller domes (<10 meters) provide undesirable visual cues that one is viewing a screen surface. This is probably due to a combination of visual accommodation (focusing) and binocular disparity (stereopsis) for varying screen distances as occurs with eye motion. Any imperfections in the dome surface, such as visable seams or perforations, also contribute to a loss of realism. Small simulator domes avoid this by placing the viewer(s) within a small viewing volume near dome center and using a solid screen surface with specially finished seams. • Realiability and Maintainability - Theaters must operate cost effectively. This requires minimizing down-time and equipment maintenance costs. • Throughput - Special venue theaters require ample seating and virtually no visitor training for an enjoyable experience. Emphasis is on providing sufficient visitor throughput to recover capital and operating costs. Immersive displays such as the CAVE™ do not provide an attractive economic model due to their low throughput, high cost and specialized staff requirements. Immersive Projection Screens Consider the field-of-view (FOV) produced by a flat projection screen. The FOV is proportional to twice the arctangent of the screen height or width. As the screen is made larger (or the viewing distance reduced), the FOV apporaches a limit of 180 degrees. The next progression is to a cylindrical screen with a vertical rotational axis. An image projected onto a cylindrical screen provides a full 360 degree horizontal FOV. Vertical FOV, however, remains limited to 180 degrees even for an infinitely tall screen. Surface reflectivity is a critical issue with curvedscreen theater design. Image contrast can be degraded due to the “integrating sphere” effect caused by light scattered from a projected image back onto another portion of the curved screen. A bright image can therefore “wash out” the entire screen if the screen reflectivity is too high. Two solutions exist for this effect. Some dome simulator systems employ screen “gain,” an increased specular reflectivity with respect to a Lambertian surface which scatters light equally in all directions [3]. This narrows the range of angles reflected from an image to cover the primary “viewing volume,” the volume containing the viewer’s head, and not the screen itself. Drawbacks to this approach include difficulty edge-blending multiple projectors and a reduced seating area. Screen gain has successfully been employed in training simulators and 3D simulator rides such as Imax’s Race for Atlantis in Las Vegas. The polarization-preserving properties of a high-gain screen surface make it crutial for polarized 3D projections [4]. Another approach for attaining a wide field-of-view is to utilize a polygonal screen configuration such as a cube or dodecaherdon. However, polygonal screens such as the CAVE, which employs a cubic configuration, do not provide a graceful degredation in orthoscopy as the viewer moves off-axis. The discontinuities inherent in polygonal screens have therefore limited their use to a small number of special venue theaters. Dome screens offer a wide FOV with the possibility of nearly full visual immersion. Highquality dome screens are readily available ranging from 3 meters to over 27 meters in diameter. Dome screen projection surfaces are typically formed by compound-curved, perforated, powdercoated aluminum panels. The perforated projection surface allows airflow from HVAC to penetrate the Another method for increasing contrast is to reduce the screen reflectivity using a neutral-density powder-coat. Since scattered light requires at least two screen reflections to reach the viewer’s eye, the scattered component is proportional to the square of the reflectivity R, which is always less than one. 2 2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 The image brightness itself is subject to a single reflection and is attenuated by R. Contrast is therefore improved at the expense of overall image brightness. It is useful to note that the exact degredation of contrast due to cross-scattering depends on the nature of the image being projected. The percieved contrast of a projected starfield, for instance, is not appreciably improved by lowering reflectivity much below 0.5 (50%). However, domes made for film projection often employ dome reflectivities of 0.3 or less. the projection source ages. Outboard electronics with a dedicated user interface are typically employed in such applications [8]. Projector deviations from ideal gamma performance can limit achievable edge-blend quality. The blue phosphor in CRT projectors, for instance, has a gamma response which deviates significantly from other phospors and requires linearization circuitry. If a color saturates, for instance, the edge-blend for a bright image will require different settings than for a dim image. Precice gamma correction is not a high priority for projector manufacturers. Neither is matching the color balance and gamma response between two projectors. Creating an acceptable edge-blend is currently more of an art than a science due to a lack of control over these factors. The commercialization of arrayed projection systems will ultimately require auto-calibration features for edge-blending and projector geometry. Projection Techniques The projection of extreme wide field-of-view (FOV) images is problematic. It requires, for instance, over 400 million pixels to cover a hemisphere with eye-limited (1 arcmin) resolution [5]. In contrast, the 1080i high-definition video format provides only 2 million pixels. A more practical approach would be to require the equivalent resolution of a 72 dpi monitor viewed at 0.6 meters (2 feet), which is 4 arcminutes under ideal conditions. The 4 arcminute requirement reduces the hemispheric coverage to 30 million pixels, still out of reach for any single projector. Even the popular IMAX® Dome format, which is not a full hemisphere, provides perhaps 12-20 million pixels for a stationary image [6]. A modern 9-inch CRT video projector can provide up to 2500x2000 addressable pixels, resulting in a 5 million pixel image. Higher resolutions will be possible with the new 12-inch projectors. However, physical limits will soon be reached on signal bandwidth and optical resolution. Arrayed projection avoids these limits by using multiple arrayed (mosaicked) video projectors which are edgeblended (stitched) to seamlessly reconstruct the original high-resolution source [7]. Extreme highresolution displays are possible using this technique. The success of arrayed projection depends in part on the technique used to edge-blend adjacent images. Soft-edge masking can be performed quite easily using an alpha-channel. However the monochrome alpha-channel affords no custom control of individual color layers. In practice, the gamma response of each projected color is slightly different requiring different edge-blend settings on each color component for optimal performance. Separate red, blue and green edge-blend masks must be created interactively for each installation and maintained as Six rendered views are mapped and blended onto dome 3 2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 Geometric mapping is required to transform a twodimensional view-plane raster image into a spherical or cylindrical section. Accurate image mapping is a critical issue for arrayed dome projection, as an effective edge-blend requires pixelaccurate overlap of adjacent frames over a compound curved projection surface. Any mismatch between frames creates a degredation in resolution performance within the edge-blends. required to produce effective edge-blends under darkframe conditions and to project over stars in planetaria. Unfortunately, either brightness or contrast are presently limited by available display technologies. It will be some years before video can match the quality of large-format film [9]. Recent Work Several companies have recently announced largescale immersive video-based theaters for real-time 3D presentations. The first public installation was the Spitz ElectricHorizon™ VR Theater, a temporary experimental theater tested last year in Pittsburg’s Carnegie Science Center [10]. This theater seats 32 persons on an inclined seating deck and includes 3-button responder units for audience interactivity. The screen is a 200° horizontal by 60° vertical FOV partial dome with an 8.5 meters diameter. The image is produced by three edgeblended Electrohome Marquee™ 9500 projectors. Image generation is provided by a single-pipe Onyx® Infinite Reality feeding three SVGA video channels. Two common techniques for mapping are image warping in the image generator and raser warping in the video projector. CRT projectors have geometry correction circuits which are incredibly flexible. Fixed-panel projectors such as the DLP or LCD are not capable of image mapping and therefore place extra demand on the image generator. Console SGI Onyx Exit Right Front Speaker Right Rear Speaker Audience Responders Curtain and Soundproofing Typical image mapping An interesting trade-off exists in system design. The edge-blend width typically ranges from 1-25% of the frame width. A wider edge-blend region increases the eye’s tolerance to adjacent projector color mismatch due to the gradual color change. At the same time, however, the wider blend area is more demanding on projector alignment over a larger area and requires more redundant pixels to be rendered per channel. Left Rear Speaker Left Front Speaker Enter a) Top view 200° x ±30° Perf. Aluminum Dome Screen Display brightness and contrast performance are also critical factors for an effective presentation. The eye’s sensitivity to color degrades with decreasing image brightness. CRT projectors, while they offer the greatest flexibility in geometric mapping and scan rates, are the lowest brightness of all projector types. A typical 9-inch CRT projector provides a brightness of 240 lumens while a high-end light-valve projector currently tops out at around 6000 lumens - 25 times brighter at only five times the cost. High contrast is Video Projectors Seating Deck b) Side view The opening show, ROBOTIX Mars Mission, was developed by Carnegie Mellon’s SIMLAB and funded by Learning Curve Toys of Chicago, IL. The show depicts a mission to the planet Mars based on Learning Curve’s ROBOTIX toys and 4 2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 NASA Mars data sets. This project revealed numerous challenges to be overcome in the commercialization of IPT for entertainment and education. While compelling, the quality of a video-based display projecting real-time 3D images simply cannot match the image quality and photorealism that people have come to expect from special effects in film. The success of these theaters will hinge on the creative use of audience interactivity to differentiate them from similar filmbased theaters. A second project last July converted the ElectricHorizon theater into a telepresence command station for Nomad, a robotic explorer. Nomad was designed by Carnegie Mellon’s Robotics Institute and tested in Chile’s Atacama Desert in conjunction with NASA Ames. A live ElectricSky video panorama tournaments, and laser shows. While this system does not employ high-end real-time 3D image generation, it is easily upgraded for this capability. Numerous such theaters will be installed in coming years, both with and without real-time capabilities, and will provide a new venue for anyone producing visually compelling virtual environments and immersive content. Content Development Much of the content for virtual environments is produced by the commercial simulator and academic communities. The cost of developing a large volume of immersive experiences is too much for any one corporate entity to bear. If these new theaters are to thrive it will be a collaborative effort. There needs to be a pathway for the dissemination of interesting and informative models and simulations into these public venues. Realtime panoramic image from Nomad robot at Carnegie Science Center feed from Nomad’s “panospheric” camera was displayed in real time for audiences at the science center. The camera provided a 360° FOV image which was remapped onto the immersive display by the Onyx. Although the frame rate was slow, since the remapping occurred in real time the audience was able to smoothly rotate the camera view using their responder buttons. A satisfying theatrical experience hinges on the presentation of a compelling show or interactive experience. Regardless of how novel and informative graphic content may seem, if it is not woven into an interesting story it will probably fall flat. For this reason it is important to involve media professionals in any serious attempt to develop entertaining shows for IPT systems. Early projects will help define successful show production models and storytelling conventions for immersive programming. Passive video playback systems avoid some of the pitfalls and expense of experimental real-time 3Dbased theaters. Last year Spitz installed their first ElectricSky™ theater in the town of Watson Lake within Canada’s Yukon Territory. The Northern Lights Centre is a unique multi-use, multi-format special venue theater which employs a partial dome panorama similar to the ElectricHorizon format. Feature shows are pre-rendered from 3D computer graphics, film and video, heavily composited in post production, and played back from three lineinterpolated NTSC sources. Other activities at the Centre include licensed DVD films, video game Our experience has taught us that the rules for IPT presentations go well beyond those of conventional media. The possibility of inducing motion sickness, for instance, requires close attention [11]. While simulator rides exploit this effect, their duration is seldom more than several minutes. If 5 2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 the desire is to convey information, immersive visuals might in fact prove to be a distraction in some cases. could be more fully exploited for information communication and navigation. It is important to characterize edge-blend performance based on factors such as gamma linearity and gamma matching between projectors, color balance, and geometric alignment. It is difficult to develop specifications for projector gamma linearity without knowing the sensitivity of the eye to deviations in linearity for an edgeblended image. Robust test patterns for display performance testing are required. It would also be useful to evaluate various projector types for fundamental limitations in key areas important to edge-blending such as gamma linearity and color balance. Another important issue in the commercialization of IPT is the development of user-friendly content production tools. Numerous tools exist for digital film and video production along with armies of trained professionals. Visual simulation is a highly specialized field with only a handful of experts and specialized software. This is changing, however, with the advent of real-time 3D computer games. Immersive multimedia productions will likely hybridize tools from the film and video, visual simulation and 3D video game industries. An immersive production environment would ideally be capable of handling any input format including live action film or video, HDTV, and prerendered animations. A major stumbling block is the lack of innovation in group interactive paradigms and hardware interfaces. Perhaps the most interesting work being done in this area is by Rob Fisher of the Studio for Creative Inquiry at Carnegie Mellon in conjunction with Cinematrix. Rob is creating a new research facility for the purpose of studying the educational effectiveness of group interactive technology. Topics include portraits of audience behavior designed to reveal features like emergent behavior, group dynamics, nearest neighbor effects, and changes in response time. One issue that remains open is the best means for involving the audience within an interactive, immersive experience. Pushbutton responders leave a lot to be desired. Group “majority rules” interactive experiences are not nearly as compelling as one-on-one video games. Even multidimensional controls such as joysticks do not provide a personal involvement in show outcome. Early systems will likely employ a trained navigator or “tour guide” to control the viewpoint for the audience. Immersive theaters will never succeed without a constant stream of compelling content. Quality content is needed now to demonstrate the viability of these theaters and to jump-start the market. Research Topics Large-scale immersive video-based theaters are finding applications in numerous special venues including planetaria. As the installed base of theaters increases, the need for compelling content and product developments will increase as will available funding. University research is expected to play a major role in the commercialization of IPT. Therefore this paper concludes with suggestions for IPT research. Renaissance painter Leonardo da Vinci considered natural perspective to be spherical [13]. Currently images are rendered as view planes and then warped or mapped to a cylinder or sphere. Practically every rendering engine in existence is based on planar perspective projection. It would be interesting to examine the possibility of a spherical rendering engine. Such a renderer would operate in polar coordinates and would require an efficient means of mapping multi-resolution images to a sphere [14,15]. IPT opens the doors for unique work in Human Factors Engineering and Psychophysics. The wide instantaneous field-of-view provided by IPT can be used to study the opto-vestibular response and other phenomena relying on peripheral vision [12]. The concept of presence may have to be revisited. If Information Visualization scientists better understood how the brain processes wide-field imagery then the increased visual bandwidth of IPT [1] Ed Lantz, Steve Bryson, David Zeltzer, Mark Bolas, Bertrand de La Chapelle, and David Bennett, “The Future of VR: Head Mounted Displays versus Spatially Immersive Displays,” Computer Graphics, Annual Conference Proceedings Series, pp. 485-486, 1996. 6 2nd Annual Immersive Projection Technology Workshop, May 11-12, 1998 [2] Ed Lantz, “Future Directions in Visual Display Systems,” Computer Graphics, 31(2), pp. 38-45, 1997 [12] Gunnar Johansson and Erik Börjesson, “Toward a New Theory of Vision: Studies in Wide-Angle Space Perception,” Ecological Psychology, 1, pp. 301-331, 1989. [3] Shen Ing, “High Gain Screen Technology,” Image VI Conference Proceedings, pp. 41-52, July 1992 [13] R. Patteson Kelso, “Perspective Projection: Artificial and Natural,” Engineering Design Graphics Journal, Vol. 56, No. 3, pp. 27-35, 1992. [4] Kevin Arthur, Kamal Hassan, and Hugh Murray, “Modelling Brightness, Contrast and 3D Coincidence in Dome Screen Theaters,” 134th SMPTE Technical Conference Proceedings, pp. 114, November 1992 [14] W. Fetter and C. Wittenberg, “A Progression of Wide-Angle Display Developments by Computer Graphics,” Displays, Vol. 5, No. 2, pp. 65-83, 1984 [5] Hemispheric pixel mapping estimate assumes the image originates from an equidistant polar format with equal-area pixels spanning the diameter (representing 180°). For 1 arcminute resolution, pixel size is 0.5 arcminutes (assuming ideal Nyquist sampling), requiring a 21,600 pixel diameter polar file size. Resulting frame contains 466,560 total pixels with 400,500 active pixels. [15] Gyorgy Fekete, “Rendering and Managing Spherical Data With Sphere Quadtrees,” IEEE Visualization ‘90 Proceedings, pp. 176-186, 1990 Ed Lantz is Product Development Manager at Spitz, Inc. in Chadds Ford, PA. He received the BS (1982) and MS (1984) degrees in Electrical Engineering from Tennessee Tech. He led optical signal processing research at Harris Corp. in Melbourne, Florida for seven years, after which he joined the Astronaut Memorial Planetarium and Observatory in Cocoa, Florida to develop advanced dome projection and control systems. In 1995 Mr. Lantz joined Spitz, Inc. to lead the development of video-based dome display systems leading to Spitz’s Immersive Visualization Environment product line. Spitz has manufactured planetaria, astronomical simulators and dome screens for over 50 years. Mr. Lantz’s interests include spherical rendering techniques, storytelling in immersive environments and advanced techniques for group interactivity. He is a member of ACM, SIGGRAPH, IEEE, SPIE, and IPS. Address: Spitz, Inc., US Route 1, Chadds Ford, PA 19317, E-mail: elantz@spitzinc.com, Work Phone: 610.459.5200, Fax: 610.459.3830, URL: www.spitzinc.com. [6] Achievable resolution of film-based systems depends on a variety of factors including film stock, exposure, and camera/projector optics. Between 50 and 150 line-pairs per millimeter are typical resolution values for film stock. [7] Theo Mayer, “New Options and Considerations for Creating Enhanced Viewing Experiences,” Computer Graphics, 31(2), pp. 3234, 1997 [8] Theo Mayer, “Design considerations and applications for innovative display options using projector arrays,” SPIE Proceedings, January 30, 1996. [9] Laurence J. Thorpe, “HDTV and Film - Issues of Video Signal Dynamic Range,” SMPTE Journal, pp. 780-795, October 1991 [10] Ed Lantz and Jon Shaw, “Spatially Immersive Displays for Group Information Visualization,” Workshop on New Paradigms in Information Visualization and Manipulation (NPIV’96), in conjunction with CIKM 96, pp. 3740, October 1996. [11] Thomas B. Sheridan and Thomas A. Furness III, ed., various articles in “Spotlight on Simulator Sickness”, Presence, Vol. 1, No. 3, 1992. 7