EUROGRAPHICS Area Papers 2011 / Andy Day, Roberto Scopigno, RafałMantiuk, Erik Reinhard
(Editors)
,
Enhancement of Low Dynamic Range Videos using High
Dynamic Range Backgrounds
Francesco Banterle, Matteo DellePiane, and Roberto Scopigno
Visual Computing Laboratory, ISTI-CNR, Italy
Abstract
In this paper, we present a practical system for enhancing the quality of Low Dynamic Range (LDR) videos using
High Dynamic Range (HDR) background images. Our technique relies on the assumption that the HDR information is static in the video footage. This assumption can be valid in many scenarios where moving subjects are the
main focus of the footage and do not have to interact with moving light sources or highly reflective objects. Another
valid scenario is teleconferencing via webcams, where the background is typically over-exposed not allowing the
users to perceive correctly the environment where the communication is happening.
Categories and Subject Descriptors (according to ACM CCS): I.4.1 [Image Processing and Computer Vision]:
Enhancement—Filtering I.3.3 [Computer Graphics]: Picture/Image Generation—Bitmap and framebuffer operations
1. Introduction
HDR images capturing is now a well known and solved
problem in many capturing conditions, see Reinhard et
al. [RWP∗ 10] for an overview. However, HDR video capturing still remains a challenge.
HDR video cameras are currently under release, but their
cost make them affordable only for specific high-standard
applications, like cinematography. On the other side, HDR
images can be obtained with an SLR camera following a
very simple procedure. In addition, some SLR and compact
cameras are now featuring HDR capturing for still images
using bracketing techniques followed by tone mapping.
Our work addresses the problem of enhancing LDR videos
using the HDR information in the scene (i.e. the background). This can be especially important when the environment contains light sources or reflective objects: if the
exposure of the camera is chosen to focus on the important
parts of he scene (i.e. a moving subject), part of the detail in
the background can be lost.
In the assumption that the subjects are not interacting with
light sources or reflective objects, we propose to acquire the
HDR information of the static scene in a preliminary stage.
Then the LDR video can be acquired in the usual way, by
setting the best exposure level for the moving subjects.
In a completely automatic post-processing phase, the HDR
c 2011 The Author(s)
Journal compilation c 2011 The Eurographics Association and Blackwell Publishing Ltd.
Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and
350 Main Street, Malden, MA 02148, USA.
information is used to recover detail in all the over-exposed
and under-exposed portions of each frame. In addition to the
increase in visual quality of the video, also the video shooting phase is greatly helped. Even in the case of complex environments, it is possible to focus on the main elements of
the scene, because the rest of the background will be enhanced in a subsequent phase.
The proposed approach can be useful in very different
contexts: from the acquisition of medium-to-high quality
videos, to the typical case of webcams, where a satisfying
trade-off in exposure between the subject in foreground and
the usually bright background is usually hard (if not impossible) to find.
The background enhancement technique is easy to obtain,
automatic and fast to process. It represents a way to add
HDR information to LDR videos, until the cost and usability of HDR video-cameras will make them usable for wide
public.
2. Related Work
We organized the related work in three categories: HDR
video acquisition using computational photography techniques, native HDR video-cameras, and techniques for enhancing videos from photographs.
Banterle et al. / Enhancement of Low Dynamic Range Videos using High Dynamic Range Backgrounds
2.1. Computational Photography
In computational photography, different approaches have
been proposed to tweak different features of LDR videocameras.
Aggarwal and Ahuja [AA01] proposed and implemented a
video-camera design, where the aperture is split into multiple parts and the beam exiting from each is directed in a different direction using mirrors. This solution has recently become a popular technique for creating HDR videos [Sov10].
However, it requires two video-cameras and accurate alignment has to take place, to avoid misalignments artifacts.
Moreover, practical solutions do not allow more than 23 cameras to be used. This limits the dynamic range that
can be captured. Kang et al. [KUWS03] achieved to capture HDR videos by varying the shutter speed of the videocamera at subsequent frames. They use warping to register different exposure frames. Then, the aligned exposure
frames are merged into a radiance map. Their method is effective, but a very high-speed camera needs to be employed,
otherwise alignment artifacts can appear. Narasimhan and
Nayar [NN02] proposed a Bayer-like pattern where exposure varies spatially. This method has the drawback that a
more complex de-mosaicing algorithm needs to be applied
and a trade-off in spatial resolution has to be found. Nayar
and Branzoi [NB03] proposed a controllable liquid crystal
light modulator in front of the camera, in order to have an
adaptive dynamic range camera. Their modulator adapts the
exposure of each pixel on the image detector, this allows to
acquire scenes with a very large dynamic range. In this case,
a perfect alignment between the sensor and the modulator is
difficult to achieve, and filtering needs to take place to avoid
artifacts.
2.2. HDR Video-cameras
with the aim of enhancing LDR videos using images. One
of the first works for enhancing LDR videos was presented
by Sand and Teller [ST04]. Their work matches different
videos allowing to create HDR videos from two or more
LDR videos. The system is prone to differences in the videos
to match, but it cannot cope with large-scale differences between the images, such as an actor that appears in one frame
but not the other.
Bennett and McMillian [BM05] introduced a system for enhancing low exposure LDR videos exploiting temporal coherence, the bilateral filter, and tone mapping. The main goal
of the paper is improving the visual quality of a LDR video
in terms of noise and dynamic range. However, the final dynamic range cannot be considered as HDR.
Ancuti [AHMB08] proposed a simple technique to transfer
detail from an high resolution image to the video. Gupta
[GBD∗ 09] extended the concept by designing a framework
that could be implemented in video-cameras which can acquire both images and videos. No HDR information is taken
into account.
In a similar fashion to our method, Bhat et al. [BZS∗ 07]
proposed a method to enhance videos using photographs of
the same scene. In their system, images can be taken from
different positions, and the geometry of the scene needs to
be estimated, possibly restraining the field of application.
Moreover, this system works only for static scenes, while
our approach deals with moving objects, from a fixed point
of view.
Wang et al. [WWZ∗ 07] proposed a method for increasing
the dynamic range and transferring the detail from a source
LDR image into a target LDR image. However, their method
can not be applied to videos because it needs user’s interaction for increasing the dynamic range and transferring detail
from a part of an image to the other one.
In last few years, HDR CCD/CMOS sensors have been introduced by several companies, such as the HDRc Sensors
line by IMS Chips [IC10]. These sensors allow to record
into 10/12-bit channels in the logarithmic domain, but they
typically capture at low resolutions, and can be very noisy.
Therefore, they are mainly used in security and industrial
manufacturing.
More recently, various HDR cameras have been presented such as the Gustavson’s camera [UG07], HDRv by
SpheronVR GmbH developed in collaboration with Warwick University [CBB∗ 09], and RED’s HDRx and easyHDR
technologies [RED10]. All these solutions are extremely expensive. Moreover, they can present a limited transportability as in the case of the HDRv due to the need of a 24 TB
array in order to acquire videos.
In our work, we automatically enhance LDR videos using HDR images in a set-up, where these are taken from the
same position and device. The camera is fixed, but the scene
is dynamic in contrast to Bhat et al.’s work [BZS∗ 07]. This
may be seen as the opposite of their methodology, where
the camera is dynamic but the scene is static. Compared to
Wang et al.’s work [WWZ∗ 07] our method works with a real
HDR background source, it uses straightforward techniques
for transferring detail and dynamic range, and it is fully automatic. The main contributions of this paper are an acquisition technique for enhancing the dynamic range of LDR
videos based on static camera, and a straightforward blending method for enhancing the input LDR videos which does
not need image warping and/or complex techniques. To our
knowledge, this kind of methodology, even if it seems to be
quite straightforward, is not present in the literature.
2.3. LDR Videos Enhancement using Photographs
3. The Acquistion Method
HDR video-cameras are extremely expensive and resourcedemanding: for this reason, some research has been made
Our capturing method consists of three straightforward
steps. Firstly, the acting scene is acquired in HDR capturc 2011 The Author(s)
Journal compilation c 2011 The Eurographics Association and Blackwell Publishing Ltd.
Banterle et al. / Enhancement of Low Dynamic Range Videos using High Dynamic Range Backgrounds
Figure 1: The proposed pipeline for augmenting LDR videos into
HDR videos.
(a)
(b)
(a)
(c)
Figure 2: An example of different blending algorithm: a) Blending
in the gradient domain. b) Blending in the spatial domain. c) The
difference between a) and b), note that gradients are slightly more
enhanced in a) and colors are slightly shifted towards a bluish tint.
ing all the dynamic range of it, from dark areas to bright
areas. Then, all different exposures are built into a radiance
map using Debevec and Malik’s method [DM97]. Secondly,
an LDR video footage, where actors play in the scene, is acquired. The exposure of this footage is manually set in order
to have actors well exposed. Finally, the background HDR
image and the LDR image are processed by a straightforward blending algorithm.
(b)
(c)
Figure 3: An example of the classification mask: a) A frame from
3.1. The Blending Algorithm
In our approach, we blend the HDR image and LDR video
in a straightforward way. The full pipeline is shown in Figure 1. Firstly, the LDR video footage is linearized applying
the inverse Camera Response Function (iCRF) of the LDR
video-camera. Assuming that the device that captured the
HDR image and the LDR video was the same, the iCRF that
is calculated during the creation of the radiance map can be
reused. After linearization, the LDR video is scaled by the
capturing shutter speed, obtaining a normalized LDR video
with absolute values. Linearization and scaling is important
in order to match intensities and colors with the reference.
Moreover, this allows to use less computationally expensive
techniques for the blending.
Secondly, the HDR image and the normalized and scaled
LDR frame are linearly blended in the logarithm domain
using a selection mask M which classifies background and
c 2011 The Author(s)
Journal compilation c 2011 The Eurographics Association and Blackwell Publishing Ltd.
an LDR video. b) The application of thresholding to a). c) The final
mask after the application of the bilateral filter.
actors. The blending is applied in the logarithm domain to
avoid seams at mask’s boundaries. Then, the blended image
is exponentiated to obtain the final radiance map. We found
out from our experiments that this straightforward blend is
enough to obtain no seams or other kind of artifacts. We
tested blending using Laplacian Pyramids [BA87] and Gradient Domain techniques [BZCC10, PGB03, FLW02], but
they produced similar results of the linear blending. Moreover, in some cases the colors are slightly shifted, see Figure
2 for a comparison. Therefore, we opted for the computationally cheaper solution of the linear blending.
Banterle et al. / Enhancement of Low Dynamic Range Videos using High Dynamic Range Backgrounds
3.2. The Classification Mask
The classification mask M is computed using thresholding on over-exposed and under-exposed pixels values on
the luminance channel for each frame of the video. In our
experiments, we found out that 0.95 and 0.05 are respectively good threshold for over-exposed and under-exposed
threholds (using normalized RGB values). Thresholding can
produce groups of single pixels in the image which are typically to be considered as noise. Therefore, we apply morphological operators such as erosion followed by dilation.
In our case, we empirically found that 3-5 iterations are
typically enough to obtain good results on full HD content
(1920 × 1080). Finally, the mask is cross bilateral filtered
with the original frame LDR luminance using the bilateral
grid [CPD07] (σs = 16 and σr = 0.1 for full HD content) in
order to smoothly extend the classification to strong edges.
Figure 3 shows an example of the different steps for calculating the mask.
(a)
4. Results
One of the goals of the proposed method was to create a
method that could be used with a wide range of videocamera types. The acquisition step is very simple, and
the processing is automatic. Hence, we decided to test
the system using both a low-end and a medium/high-end
camera.
In our experiments, we tested our capturing technique
with a Logitech QuickCam Pro 9000 and a Canon 550D
(or Rebel XTi) DSLR camera. The images depicting the
reference HDR were tone mapped using Reinhard’s operator
implemented in [BADC10].
The Canon 550D is able to acquire videos at 1920 × 1080
full HD resolution, and it can be adapted to difficult lighting
conditions. In the example shown in Figure 4, a tone mapped
version of the HDR background is shown on the top. The
middle image shows a frame of the original LDR video,
where most of the background and the sky are over-exposed.
Using the HDR background image, the enhanced video
(bottom image) recovers the appearance of the sky and of
several over-exposed parts.
A second example (Figure 5) shows an indoor environment,
where the light and part of the outdoor scene are lost.
The enhanced video recovers this information. The colors
in the enhanced frame are different from the ones in the
LDR frame because the linearization process matches color
curves between HDR background and LDR frame.
The second type of tested device (a medium-end webcam
with a 1280 × 720 HD Ready resolution) covers for a
different type of application: the enhancement of videos
which are not able to adapt to difficult conditions. Webcams
are usually not able to find a trade-off exposure between
the foreground subject and the background. The example
in Figure 6 allows to select the best exposure for the main
(b)
(c)
Figure 4: A frame of a video taken with Canon 550D (1920 ×
1080 resolution): a) A tone mapped version the HDR reference of
the background. b) A frame from the original LDR video. c) The
same frame in b) after enhancement and tone mapping.
subject, while background information is recovered during
the blending. The top image shows that a convincing HDR
can be obtained also with this device, although part of the
range is missing due its limitations.
Regarding timing, we ran our fully automatic enhancement algorithm, described in the previous Section, on an Intel Core 2 Duo at 2.33 Ghz equipped with 3Gb of memory
and Windows 7. The algorithm was implemented in Matlab,
and each frame took on average less than 9 seconds for a
1920 × 1080 frame. Note that the algorithm was run on a
single core using unoptimized Matlab code expect the Matc 2011 The Author(s)
Journal compilation c 2011 The Eurographics Association and Blackwell Publishing Ltd.
Banterle et al. / Enhancement of Low Dynamic Range Videos using High Dynamic Range Backgrounds
(a)
(a)
(b)
(b)
Figure 5: An example of indoor scene taken with a Canon 550D
(1920 × 1080): a) A frame from the original LDR video. c) The same
frame in a) after enhancement and tone mapping.
lab Bilateral Grid implementation by Chen et al. [CPD07]
which took around 4 seconds on average. All the operations
of the algorithm are straightforward image processing operations such as thresholding, morphological operations (i.e.
dilatation and erosion) which can be easily implemented on
graphics hardware. Moreover, the Bilateral Grid can be implemented efficiently in real-time on GPUs [CPD07]. Therefore, a real-time implementation of the whole algorithm
would be possible on graphics hardware, allowing to preview the final results.
The main limitation of our algorithm is that the camera
needs to be static. However, actors can play inside the scene,
allowing to enhance the experience during teleconferencing
or improve visual quality in movies. Another limitation is
that our straightforward classifier based on threshold can
have false positives. This means that an over-exposed moving object can be classified as background. However, this
typically happens for very few frames, because the main exposure was set for an overall good exposure for actors and
moving objects. Note that this artifacts cannot be noticed by
the human eye because it stands for only few frames, and it
can be perceived for a reflection. A better classifier based on
motion estimation or tracking would solve this problem that
can happen during the classification.
c 2011 The Author(s)
Journal compilation c 2011 The Eurographics Association and Blackwell Publishing Ltd.
(c)
Figure 6: A frame of a short sequence taken with a Logitech QuickCam Pro 9000 (1280 × 720 resolution): a) The tone mapped version
of the HDR background. b) A frame from an LDR video of the scene.
c) The same frame in a) after enhancement and tone mapping.
5. Conclusion and Future Work
In this paper, we presented a straightforward technique for
increasing the dynamic range of LDR videos using HDR
background images taken from the same camera and position. We showed that this technique is able to produce convincing videos at low cost. The method can be exploited for
video teleconferencing when challenging backgrounds (i.e.
with windows, and bright light sources) are present in the
environment in order to feel the users a true teleconferencing experience. Moreover, the method can be employed for
shooting movies in a similar manner as chroma key or bluescreen are used nowadays for enhancing the movie produc-
Banterle et al. / Enhancement of Low Dynamic Range Videos using High Dynamic Range Backgrounds
tions with visual effects.
In future work, we would like to improve the classification
algorithm in order to remove the minor problems in case of
over-exposed moving actors and/or objects. This classification could be coupled with reverse tone mapping operators
[MAF∗ 09, DMHS08, RTS∗ 07] in order to improve the overexposed areas of the actors and/or moving objects.
In addition, code re-factoring and GPU implementation
could decrease the processing time to the possibility or having quasi-realtime enhancement: one of the possible applications of this could be video-conferencing.
Acknowledgements. We thank Gianpaolo Palma and
Daniele Bernabei for their help in acquiring the video sequences. Marco Di Benedetto, Stefano Marras, and Daniele
Bernabei played in these videos, and their contribution is
gratefully acknowledged. We also thank the anonymous reviewers, whose suggestions improved the paper. The work
presented in this paper was founded by the EC IST IP project
“3D-COFORM” (IST-2008-231809).
References
[AA01] AGGARWAL M., A HUJA N.: Split aperture imaging for
high dynamic range. Computer Vision, IEEE International Conference on 2 (2001), 10. 2
[AHMB08] A NCUTI C., H ABER T., M ERTENS T., B EKAERT P.:
Video enhancement using reference photographs. The Visual
Computer 24, 7-9 (2008), 709–717. 2
[BA87] B URT P. J., A DELSON E. H.: Readings in computer vision: issues, problems, principles, and paradigms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1987, ch. The
Laplacian pyramid as a compact image code, pp. 671–679. 3
[BADC10] BANTERLE F., A RTUSI A., D EBATTISTA K.,
C HALMERS A.: Advanced High Dynamic Range Imaging: Theory and Practice, first edition ed. AK Peters, Ltd, 2010. 4
[BM05] B ENNETT E. P., M C M ILLAN L.: Video enhancement
using per-pixel virtual exposures. ACM Trans. Graph. 24 (July
2005), 845–852. 2
[BZCC10] B HAT P., Z ITNICK C. L., C OHEN M., C URLESS B.:
Gradientshop: A gradient-domain optimization framework for
image and video filtering. ACM Trans. Graph. 29 (April 2010),
10:1–10:14. 3
[BZS∗ 07] B HAT P., Z ITNICK C. L., S NAVELY N., AGARWALA
A., AGRAWALA M., C URLESS B., C OHEN M., K ANG S. B.:
Using photographs to enhance videos of a static scene. In Rendering Techniques 2007 (Proceedings Eurographics Symposium
on Rendering) (jun 2007), Kautz J., Pattanaik S., (Eds.), Eurographics, pp. 327–338. 2
[CBB∗ 09] C HALMERS A., B ONNET G., BANTERLE F., D UBLA
P., D EBATTISTA K., A RTUSI A., M OIR C.: High-dynamicrange video solution. In ACM SIGGRAPH ASIA 2009 Art Gallery
& Emerging Technologies: Adaptation (New York, NY, USA,
2009), SIGGRAPH ASIA ’09, ACM, pp. 71–71. 2
[CPD07] C HEN J., PARIS S., D URAND F.: Real-time edge-aware
image processing with the bilateral grid. ACM Trans. Graph. 26,
3 (2007), 103. 4, 5
and interactive techniques (New York, NY, USA, 1997), ACM
Press/Addison-Wesley Publishing Co., pp. 369–378. 3
[DMHS08] D IDYK P., M ANTIUK R., H EIN M., S EIDEL H.-P.:
Enhancement of bright video features for HDR displays. In Proceeding of Eurographics Symposium on Rendering 2008 (2008),
Computer Graphics Forum, Eurographics, Blackwell Ltd. 6
[FLW02] FATTAL R., L ISCHINSKI D., W ERMAN M.: Gradient
domain high dynamic range compression. ACM Trans. Graph.
21, 3 (2002), 249–256. 3
[GBD∗ 09] G UPTA A., B HAT P., D ONTCHEVA M., C URLESS B.,
D EUSSEN O., C OHEN M.: Enhancing and experiencing spacetime resolution with videos and stills. In International Conference on Computational Photography (2009), IEEE. 2
[IC10] IMS-C HIPS:
HDRc Sensors.
http://www.imschips.de/home.php?id=a3b4c2en (December 2010). 2
[KUWS03] K ANG S. B., U YTTENDAELE M., W INDER S.,
S ZELISKI R.: High dynamic range video. ACM Trans. Graph.
22 (July 2003), 319–325. 2
[MAF∗ 09] M ASIA B., AGUSTIN S., F LEMING R. W., S ORKINE
O., G UTIERREZ D.: Evaluation of reverse tone mapping through
varying exposure conditions. ACM Trans. Graph. 28, 5 (2009),
1–8. 6
[NB03] NAYAR S., B RANZOI V.: Adaptive Dynamic Range
Imaging: Optical Control of Pixel Exposures over Space and
Time. In IEEE International Conference on Computer Vision
(ICCV) (Oct 2003), vol. 2, pp. 1168–1175. 2
[NN02] NAYAR S., NARASIMHAN S.: Assorted Pixels: MultiSampled Imaging With Structural Models. In European Conference on Computer Vision (ECCV) (May 2002), vol. IV, pp. 636–
652. 2
[PGB03] P ÉREZ P., G ANGNET M., B LAKE A.: Poisson image
editing. ACM Trans. Graph. 22 (July 2003), 313–318. 3
[RED10]
RED: Hdrx. In http://www.red.com/ (2010). 2
[RTS∗ 07] R EMPEL A. G., T RENTACOSTE M., S EETZEN H.,
YOUNG H. D., H EIDRICH W., W HITEHEAD L., WARD G.:
Ldr2hdr: on-the-fly reverse tone mapping of legacy video and
photographs. ACM Trans. Graph. 26, 3 (2007), 39. 6
[RWP∗ 10] R EINHARD E., WARD G., PATTANAIK S., D EBEVEC
P., H EIDRICH W., M YSZKOWSKI K.: High Dynamic Range
Imaging, Second Edition: Acquisition, Display, and Image-Based
Lighting (The Morgan Kaufmann Series in Computer Graphics).
Morgan Kaufmann Publishers Inc., San Francisco, CA, USA,
2010. 1
[Sov10] S OVIET M ONTAGE P RODUCTIONS:
HDR Video
Demonstration Using Two Canon 5D mark II’s.
In
http://www.sovietmontage.com/ (San Francisco, CA, USA,
2010). 2
[ST04] S AND P., T ELLER S.: Video matching. ACM Trans.
Graph. 23 (August 2004), 592–599. 2
[UG07] U NGER J., G USTAVSON S.: High dynamic range video
for photometric measurement of illumination. In In Proceedings
of Sensors, Cameras and Systems for Scientific/Industrial Applications X, IS&T/SPIE 19th Inernational Symposium on Electronic Imaging (2007), vol. 6501. 2
[WWZ∗ 07] WANG L., W EI L.-Y., Z HOU K., G UO B., S HUM
H.-Y.: High dynamic range image hallucination. In Proceedings
of Eurographics Symposium on Rendering (Jun 2007). 2
[DM97] D EBEVEC P., M ALIK J.: Recovering high dynamic
range radiance maps from photographs. In SIGGRAPH ’97: Proceedings of the 24th annual conference on Computer graphics
c 2011 The Author(s)
Journal compilation c 2011 The Eurographics Association and Blackwell Publishing Ltd.