USOO833O829B2

(12) United States Patent (10) Patent No.: US 8,330,829 B2

Goh et al. (45) Date of Patent: Dec. 11, 2012
(54) PHOTOGRAPHC FLICKER DETECTION 7,248,289 B2 7/2007 Katoh et al.
AND COMPENSATION 7.289,144 B2 10/2007 Arazaki
7,502,054 B2 * 3/2009 Kalapathy et al. ......... 348,226.1
7,538,799 B2 5, 2009 Yanof
(75) Inventors: Roy Goh, Redmond, WA (US); Andrew 2004/0080630 A1* 4/2004 Kim ........................... 348,226.1
Juenger, Woodinville, WA (US); Yi He, 2005/0225.649 A1 * 10/2005 ShinotSuka. ................ 348,226.1
Issaquah, WA (US): Stuart William 2007/0O859 12 A1 4/2007 Hakola et al.
Milton, Woodinville, WA (US) 2008/0309791 A1 12/2008 Nishiwaki et al.
2009.0128660 A1* 5/2009 Dai et al. ................... 348,228.1
2009/01471.03 A1* 6/2009 Chao .......................... 348,226.1
(73) Assignee: Microsoft Corporation, Redmond, WA
(US) OTHER PUBLICATIONS

(*) Notice: Subject to any disclaimer, the term of this Poplin, Dwight, "An Automatic Flicker Detection Method for
patent is extended or adjusted under 35 Embedded Camera Systems'. Retrieved at <<http://ieeexplore.ieee.
U.S.C. 154(b) by 236 days. org/stamp? stamp.jsp?arnumber=01649642). IEEE Transactions on
Consumer Electronics, May 2006, vol. 52, No. 2, pp. 308-311.
(21) Appl. No.: 12/755,952 “International Search Report'. Mailed Date: Aug. 3, 2011, Applica
tion No. PCT/US2010/059595, Filed Date: Dec. 9, 2010, pp. 6.
(22) Filed: Apr. 7, 2010
* cited by examiner
(65) Prior Publication Data
US 2011 FO157415A1 Jun. 30, 2011 Primary Examiner — Sinh Tran
Assistant Examiner — Christopher K Peterson
Related U.S. Application Data (74) Attorney, Agent, or Firm — Alleman Hall McCoy
Russell & Tuttle LLP
(60) Provisional application No. 61/291,607, filed on Dec.
31, 2009. (57) ABSTRACT
(51) Int. C. One disclosed embodiment includes detecting flicker in a
H)4N 9/73 (2006.01) photographic setting. A plurality of samples are obtained
(52) U.S. Cl. .................................. 348/226.1: 348/227.1 from a light detector circuit of a camera, each of which has a
(58) Field of Classification Search ..... 348/226.1-228.1 value that is dependent upon intensity of light incident upon
See application file for complete search history. the camera. The method further includes processing the
samples to identify whether the incident light is varying in
(56) References Cited intensity at one or more pre-selected frequencies. The pro
cessing may include multiplying the samples with one or
U.S. PATENT DOCUMENTS more periodic signals to generate scalar product outputs, and
5,272.539 A 12/1993 Kondo analyzing the scalar product outputs to ascertain the presence
6,999,118 B2 2/2006 Suzuki of periodic flicker.
7,034,870 B2 4/2006 Nagaoka et al.
7,187.405 B2 3/2007 Poplin et al. 18 Claims, 3 Drawing Sheets

WIDEO RECORD NG SYSTEMIAPPARATUS

100- DISPLAY NON WOLATLE
36 STORAGE

RAM

PROCESSOR
OPTICAL IMAGE
SYSTEM | SENSOR
10 20
112

3.
U.S. Patent Dec. 11, 2012 Sheet 1 of 3 US 8,330,829 B2

VIDEO RECORDING SYSTEM/APPARATUS

DISPLAY NON WOLATLE
36 STORAGE

RAM

PROCESSOR
OPTICAL MAGE
SYSTEM | SENSOR
10 20

210 -220 230

T1 T3 T4
FIG. 2
U.S. Patent Dec. 11, 2012 Sheet 2 of 3 US 8,330,829 B2

SET SAMPLE NUMBER = 0;

CONFIGURE GPIO1 TO OUTPUTAZERO
310
o
CONFIGURE GPIO2 TO OUTPUTAZERO;
INITIALIZE SAMPLE, CHARGE, AND DISCHARGETIMERS:
315
AMPLE NUMBERE YES
MAXIMUM? -1 END

CONFIGURE GPIO1 TO OUTPUTAONE

CONFIGURE GPIO2ASAN INPUT:
START CHARGETIMER AND SAMPLE TIMER

YES 330
STORE SAMPLE TIMERVALUE

337
WAIT
CHARGETIMER =
FINAL VALUE-1 No
START DISCHARGETIMER;
CONFIGURE GPIO1 TO OUTPUTAZERO;
CONFIGURE GPO2 TO OUTPUTAZERO

347
WAIT
DISCHARGETIMERE sa

FINAL VALUE?
YES 350
INITIALIZE SAMPLE, CHARGE, AND DISCHARGETIMERS:
INCREMENT SAMPLE NUMBER

FIG. 3
U.S. Patent Dec. 11, 2012 Sheet 3 of 3 US 8,330,829 B2

LSTTOIV/

o !”

S(HEZÁ?TV)N zH0(z410y’6_)/8

D
Iºvis
C.
 US 8,330,829 B2
1. 2
PHOTOGRAPHC FLICKER DETECTION FIG.2 shows a timing diagram of signals from the nodes of
AND COMPENSATION the light detector circuit.
FIG. 3 shows an embodiment of a method for sampling
CROSS REFERENCE TO RELATED electrical signals that vary in response to ambient light inten
APPLICATIONS S1ty.
FIG. 4 shows an embodiment of a method for processing
This application claims priority to U.S. App. Ser. No. samples to identify whether ambient light associated with the
61/291,607, filed Dec. 31, 2009, the entirety of which is samples varies in intensity at one of a plurality of pre-selected
hereby incorporated herein by reference. frequencies.
10
BACKGROUND DETAILED DESCRIPTION

Fluorescent lamps typically employ ballast mechanisms As indicated above, it will often be desirable to configure
that can cause light output to vary in intensity at a rate twice Video recording devices and other cameras to account for
the frequency of the alternating current (AC) power Supplied
15 periodic changes in the ambient light environment. Fluores
to the fluorescent lamp. The varying intensity, or flicker, of the cent lights are a common Source of periodic variation in light
intensity, or flicker. Fluorescent light sources produce light as
lamps is generally not detectable by the human eye, but it can a result of being driven by electrical components known as
interact with a camera's image sensors. In some cases, flicker ballasts. In particular, the ballast causes excitation of gas
may cause noticeable artifacts to appearin video images, such within the light fixture in order to produce light. The AC
as non-static banding or fluctuating brightness. A camera may power that drives the ballast causes the excitation to vary
Substantially reduce these artifacts by using a set of exposure slightly. Specifically, there are two peaks of excitation for
timings that are matched to the AC frequency Supplied to the each period of the AC waveform, resulting inflicker occurring
fluorescent lamps. However, there are two main AC frequen at twice the AC power frequency. The flicker typically is
cies used throughout the world, 50 Hz and 60 Hz. This pre 25 undetectable by the human eye, though it can interact with the
sents the issue of identifying the frequency that is employed, image sensors of a camera to produce undesirable artifacts. To
and then setting exposure timings on the camera to make the reduce Such artifacts, the exposure timings of a camera may
appropriate compensation. This may be performed manually be synchronized with the periodic intensity variations of the
by the user, but Such an approach involves an extra manual ambient light. Since there are two main AC frequencies used
step and the possibility of operator error. An alternate 30 throughout the world, 50 Hz and 60 Hz, causing intensity
approach is to pre-configure the camera during manufacture variations at 100 Hz and 120 Hz, respectively, it will often be
to employ compensation for a specific AC frequency (e.g., 60 desirable to configure the camera with the exposure timings
HZ). With this approach, however, a manufacturer would have correlated to the local AC power frequency.
to make different cameras for use in different parts of the One solution for configuring the camera is to have a user of
world. Also, a camera designed for a particular frequency 35 the camera manually configure the camera with the local AC
wouldn't perform as well if taken to region employing a power frequency. For example, the user may select the local
different AC frequency. AC power frequency from a menu presented on a display of
the camera. A second solution is to make one model of the
SUMMARY camera for 50Hz AC power and a second for 60Hz AC power.
The different models could then be identified by different
40
Accordingly, various embodiments are disclosed herein stock-keeping units (SKU).
that relate to detecting and compensating for the presence of In an alternate solution, the camera may include a detector
periodic flicker in photographic settings. For example, one for automatically detecting intensity variations in the ambient
disclosed embodiment includes obtaining a plurality of light. One potential advantage of this solution is that a user
samples from a light detector circuit of a camera, each of 45 cannot inadvertently configure the camera with the wrong AC
which is an electrical signal having a value that is dependent power frequency. Another potential advantage is eliminating
in some fashion upon intensity of light to which the camera is the need to make different camera models. Yet another benefit
exposed. The method further includes processing the samples is that the device can automatically adjust and operate cor
to identify whether the incident light is varying in intensity at rectly when placed into a different flicker environment, such
one or more pre-selected frequencies. Such processing may 50 as when a user of the camera travels to a region with a
include multiplying the samples with one or more periodic different AC power frequency. Accordingly, various systems
signals to generate Scalar product outputs, and analyzing the and methods are disclosed for automatically detecting and
Scalar product outputs to ascertain the presence of periodic adjusting or compensating for periodic flicker.
flicker. FIG. 1 shows an example embodiment of a camera 100
This Summary is provided to introduce a selection of con 55 including optical system 10, image sensor 20, light detector
cepts in a simplified form that are further described below in circuit 40, and a logic Subsystem Such as processor 32. Cam
the Detailed Description. This Summary is not intended to era 100 may operate in an ambient light environment that
identify key features or essential features of the claimed sub includes a flickering light Source, such as fluorescent lamp
ject matter, nor is it intended to be used to limit the scope of 110. When the camera is introduced into such a photographic
the claimed subject matter. Furthermore, the claimed subject 60 setting, light from fluorescent lamp 110 may be incident upon
matter is not limited to implementations that solve any or all image sensor 20 and light detector circuit 40, as indicated by
disadvantages noted in any part of this disclosure. rays 112 and 114, respectively.
Optical system 10 may comprise a lens system for focusing
BRIEF DESCRIPTION OF THE DRAWINGS an image onto image sensor 20. Optical system 10 may
65 include filters, such as an infrared blocking filter, or color
FIG. 1 shows an embodiment of a camera including a light filters for directing red, green, and blue light toward selected
detector circuit. pixels of image sensor 20, for example. Image sensor 20
 US 8,330,829 B2
3 4
converts the image from optical system 10 into a set of elec indicated above, or one or more display devices utilizing
trical signals that are sent to processor 32. Image sensor 20 virtually any type of display technology, Such as a liquid
may be a charge-coupled device (CCD), a complementary crystal display (LCD).
metal-oxide-semiconductor (CMOS) active pixel sensor, or The logic subsystem of camera 100 may be configured to
any other suitable device. Image sensor 20 typically com communicate via one or more signal pins of GPIO 33. In
prises an array of pixels arranged in a series of rows and particular, as explained in detail below, GPIO 33 provides an
columns. example of a mechanism that may be used to sample electrical
Image sensor 20 may include an electronic shutter con signals from a light detector circuit. Such as light detector
trolled by processor 32, or another mechanism to control the 10
circuit 40. GPIO 33 includes logic connected to a first GPIO
integration time of each row of pixels. “Integration time' signal pin, GPIO1, and to a second GPIO signal pin, GPIO2.
refers to the amount of time that an image sensor (or portion A GPIO signal pin may be programmed to be an input or an
thereof) is exposed to light, and is thus analogous to the output. When programmed as an output, GPIO 33 may drive
photographic term "exposure.” When the integration time of a logic Zero or a logic one onto a signal pin. For example, the
the pixels is not synchronized with flicker present in the 15 Voltage at ground may be a logic Zero, and the power Supply
ambient light, artifacts such as non-static banding or fluctu Voltage, V of the logic Subsystem may be a logic one.
ating brightness may appear in video images. Brightness When programmed as an input, GPIO 33 may read a logic
variation artifacts may appear within a given frame or static Zero or a logic one from a signal pin. The Voltage on a GPIO
image, or may occur frame-to-frame in a video stream. One signal pin may be between ground and V. When the GPIO
Source offlicker may be a fluorescent lamp flickering at a rate signal pin is configured as an input, the Voltage on the GPIO
twice the frequency of the alternating current (AC) power signal pin may be interpreted as a logic one when the Voltage
supplied to the ballast that drives the fluorescent lamp. exceeds the V. Voltage and as a logic Zero otherwise. The
Another source offlicker may be light from a computer moni V, and V parameters are a function of the devices used to
tor flickering at the monitor's refresh rate. If it can be deter implement the logic subsystem. The PIC18F2550 manufac
mined that the recording environment includes light intensity 25 tured by Microchip TechnologyTM, Inc., Chandler, Ariz., is an
that varies at a particular frequency (e.g., periodic flicker example of an inexpensive controller that includes GPIO
resulting from a fluorescent light source), the integration time logic and signal pins.
of the pixels may be adjusted to reduce or eliminate potential Generally, light detector circuit 40 is responsive to the
artifacts. intensity of light that is incident on the camera (e.g., the video
In addition to processor 32, camera 100 may be provided 30 recording device). For example, the circuit may be configured
with various other components to perform flicker detection/ to respond to ambient light intensity by yielding an electrical
compensation and to carry out other tasks. As an example, signal proportional to the intensity of the incident light. In this
camera 100 may include general purpose input and output manner, the intensity of ambient light may be measured. In
logic (GPIO) 33 (e.g., a chip or chipset) and a data-holding one example, an output of light detector circuit 40 may be
35 sampled at different points in time. As explained below, the
Subsystem, which may include non-volatile storage 30, ran sampling may be conducted via control signals applied to
dom access memory (RAM)34, and/or removable media 35. light detector circuit 40 from GPIO 33. The obtained samples
In one embodiment of camera100, the electronic components may then be processed (e.g., via instructions executed by
may be integrated into one or more common devices, such as processor 32) to identify periodic intensity variations in the
an application specific integrated circuit (ASIC) or a system 40 incident light.
on a chip (SOC). Camera 100 may optionally include display In some cases, it may be desirable to employ processing
36 and/or other components not shown in FIG. 1. which seeks to identify variations in intensity that occur at
Processor 32 may be configured to execute one or more one of a plurality of pre-selected discrete frequencies. For
instructions stored in the data-holding Subsystem. For example, given that the large majority of the world employs
example, processor 32 may be configured to process electri 45 AC power causing fluorescent lamp flicker to occur at either
cal signals produced in response to light exposure and, in 100 Hz or 120 Hz, one approach would be to conduct the
response, adjust or otherwise control the exposure timings of processing to look for just these frequencies. Such an
image sensor 20. Non-limiting examples of nonvolatile Stor approach may be advantageous from an efficiency standpoint,
age 30 include flash memory and a hard disc drive. RAM34 and may enable the use of low-cost components and compu
may also be configured to hold instructions for execution by 50 tational methods that consume minimal resources. As dis
processor 32. Processor 32 may be configured to cause image cussed in more detail below, looking for specific discrete
data from image sensor 20 to be stored in RAM 24, nonvola frequencies (whether two or otherwise) is a non-limiting
tile storage 30, and/or removable media 35. Non-limiting example, and variety of other implementations may be used.
examples of removable media 35 include videotape, record Light detector circuit 40 typically includes a light sensor
able optical discs, and memory cards. In an alternate embodi 55 with an output that varies according to the intensity of the
ment, processor 32 may be configured to cause image data light incident upon light sensor. Non-limiting examples of
from image sensor 20 to be transmitted to a computing device light sensors include phototransistors, photodiodes, photore
via a video cable and/or a wired or wireless network. sistors, and charge-coupled devices. The light sensor may be
When included, display subsystem 36 may be used to a low-cost discrete component, producing output soon after
present a visual representation of data held by the data-hold 60 camera 100 is powered on. For example, the light sensor may
ing Subsystem. For example, display Subsystem 36 may be produce an output before other components of camera 100 are
used to display image data from image sensor 20, Such as on initialized. In this manner, the light sensor output may be used
a display Screen and/or through an optical viewfinder. In to determine exposure timings of image sensor 20 quickly
another example, display Subsystem 36 may be used to dis upon start-up, prior to a user operating camera 100. In an
play image data stored in removable media35. In this manner, 65 alternate embodiment, the light sensor may be integrated with
a viewer may view images that are stored in the camera. other electronic components of camera 100 into one or more
Display Subsystem 36 may include optical components, as common devices, such as an ASIC or a SOC.
 US 8,330,829 B2
5 6
In the example embodiment of FIG.1, light detector circuit GPIO2 may increase until the circuit reaches steady-state or
40 is connected to GPIO 33 via GPIO signal pins, GPIO1 and until the voltage of GPIO1 is driven with a logic Zero. In FIG.
GPIO2. Phototransistor 42 converts light into an electrical 2, the voltage of GPIO2 reaches steady-state and the voltage
signal proportional to the intensity of the light incident upon of GPIO2 exceeds V, at time T2. Rise time 210, which is the
phototransistor 42. Light may be directed toward phototrans- 5 difference between time T2 and T1, may be measured as a
istor 42 by optical system 10. For example, optical system 10 representation of the amount (intensity) of light Striking pho
may include lenses, filters, mirrors and/or other optical com totransistor 42. When the light striking phototransistor 42 is
ponents to direct light to image sensor 20. In some embodi increased (i.e., higher intensity), the parasitic capacitor will
ments, phototransistor 42 is placed in the optical pathway charge faster and rise time 210 will be decreased. Thus, there
formed by these components. Alternatively, light may reach 10 is an inverse relationship between the intensity of ambient
phototransistor 42 through direct exposure, or otherwise light and rise time 210. Charge time 220, which is the differ
independently of optical system 10, for example, by situating ence between time T3 and T1, may be set so that rise time 210
the phototransistorator near an external housing of the cam is less than charge time 220 for expected ranges of ambient
era. Phototransistors may be manufactured with varying sen light intensity and tolerances of the resistance of resistors 44
sitivities to different wavelengths of light. Phototransistor 42 15 and 46, V, and capacitance of phototransistor 42.
may be manufactured to be sensitive to the visible spectrum of At time T3, light detector circuit 40 is returned to the
light, from approximately 380 nanometers (nm) to 750 nm. discharge state. GPIO1 and GPIO2 are configured as outputs
Phototransistors are a specialized form of bipolar junction driving logic Zeroes. This may enable parasitic capacitor of
transistor including a base, a collector, an emitter and a para phototransistor 42 to quickly discharge through Small resistor
sitic capacitor between the collector and emitter. The 20 44. By discharging the parasitic capacitor, light detector cir
OSRAMTM SFH 310 photoransistor (OSRAM Opto Semi cuit 40 is enabled for the next charge state at time T4. Dis
conductorsTM, Regensburg, Germany) may be suitable for charge time 230, which is the difference between time T4 and
phototransistor 42, for example. T3, may be shorter than charge time 220. Sample period 240
Resistors 44 and 46 bias phototransistor 42 so that current is the sum of charge time 220 and discharge time 230.
generated by light striking the base-collector region of pho- 25 From the above, it will be appreciated that the timing
totransistor 42 may be amplified as current flowing from diagram of FIG. 2 provides an example of how an individual
collector C to emitter E. Resistor 44 is connected between the sample may be obtained from the light detector circuit. It will
GPIO1 signal pin and the collector of phototransistor 42. be appreciated, however, that multiple samples typically are
Resistor 46 is connected to ground on one side and to the obtained. Specifically, a series of samples may be collected
GPIO2 signal pin and the emitter of phototransistor 42 on the 30 from light detector circuit 40 via multiple cycles of the charge
other side. The resistance of resistor 46 may be much larger and discharge operation described above. The samples may
than the resistance of resistor 44. For example, the values of be retained by recording each rise time 210 as it occurs, for
resistors 44 and 46 may be 10 ohms and 510 kilohms, respec example by writing the rise time into a storage location (e.g.,
tively. in RAM 34 or non-volatile storage 30).
FIG. 2 is a timing diagram that illustrates one example of 35 Turning now to FIG. 3, an example method will be
how light detector circuit 40 may be used to detect the inten described for obtaining a plurality of samples. The method is
sity of light incident upon phototransistor 42. In the example applicable to obtaining any desired number of samples. As
of FIG. 2, light detector circuit 40 is alternately discharged will be explained in more detail below, the desired number of
and charged so that a rise time of light detector circuit 40 may samples and other sampling parameters may be dependent
be measured successively to generate a plurality of rise time 40 upon various factors. Generally, it will be desirable to mini
samples. The rise time of light detector circuit 40 is propor mize the amount of sampling, while at the same time provid
tional to the intensity of the light incident upon light detector ing a number of samples that provides high-confidence output
circuit 40. In alternative embodiments, other properties of and that is well-suited to the computational methods that are
light detector circuit 40 may be measured. Such as a current employed to discern flicker intensity. In addition, employing
through phototransistor 42 or a Voltage across resistor 46. As 45 an appropriate number of samples will in some cases facilitate
indicated above, numerous possibilities exist for generating use of efficient methods and processes that reduce the com
and sampling electrical signals that vary in response to ambi plexity and time associated with processing the samples.
ent light intensity. FIG. 3 shows an example embodiment of a method for
Continuing with the example of FIG. 2, at time T0, light obtaining a plurality of rise time samples from light detector
detector circuit 40 is in the discharge state. GPIO signal pins 50 circuit 40. In describing FIG. 3, various references will also
GPIO1 and GPIO2 are configured as outputs driven to a logic be made to components and signal elements shown in FIGS.
Zero. At time T1, light detector circuit 40 is switched to the 1 and 2. At 310, the sample number is set to zero and the GPIO
charge state by the low-to-high transition of GPIO1. GPIO1 is signal pins, GPIO1 and GPIO2, are configured as outputs
configured as an output driving a logic one and GPIO2 is driving a logic Zero. In this way, the parasitic capacitor of
configured as an input. In steady-state, when enough light is 55 phototransistor 42 may be discharged as illustrated at T0 of
incident upon phototransistor 42, the voltage of GPIO2 will FIG. 2. The sample timer, charge timer and discharge timer
be given by the Voltage divider formed by resistors 44 and 46. are initialized to their starting value. The sample timer, charge
When resistor 46 has a much larger resistance value than timer and discharge timer may be implemented via execut
resistor 44, the voltage of GPIO2 will exceed V and the able instructions carried out by processor 32 (FIG. 1). In an
GPIO2 input will be interpreted as a logic one. 60 example embodiment, the timers may be initialized to Zero so
However, in the transient state beginning at time T1, the the timers may be incremented to show the passage of time. In
parasitic capacitor of phototransistor 42 prevents the Voltage an alternative embodiment, the timers may be written with a
from changing instantaneously across the collector and emit desired value associated with a timing interval and the timers
ter of phototransistor 42. The parasitic capacitor charges at a may be decremented to show the passage of time.
rate determined by its capacitance, the resistance of resistors 65 Next, at 315, the sample number is compared to a maxi
44 and 46, V, and the amount of light incident on pho mum number of samples. If the sample number equals the
totransistor 42. As the capacitor charges, the Voltage of maximum number of samples, all samples have been
 US 8,330,829 B2
7 8
obtained and processing ends. If the sample number is less power supply or a 60 Hz AC power supply, with resulting
than the maximum number of samples, more samples need to intensity variation occurring at 100 Hz or 120 Hz, respec
be obtained and method 300 continues at 320. tively.
At 320, GPIO1 is configured to drive a logical output of one At 410, Samples are obtained, for example using the sys
and GPIO2 is configured as an input. The charge timer and the tems and methods described with respect to FIGS. 1-3. In one
sample timer may be started. The timers may be implemented embodiment, N samples of the signal are obtained over the
with dedicated hardware timers or software loops, for entire timing window before sample processing begins. In an
example. In this manner, light detector circuit 40 may begin a alternative embodiment, sampling may occur in parallel with
charge state, such as at time T1 in FIG. 2. At 325, it is sample processing. At least one sample is obtained before
determined if the voltage on GPIO2 exceeds V, and is a logic 10 processing may begin and N samples are obtained before
one. If GPIO2 is a logic one, method 300 continues at 330. processing may end. Once processing is to begin, samples of
Otherwise, routine 300 continues at 327. At 327, method 300 the signal are passed to blocks 430, 432, 434, and 436 to
generate scalar products, as explained below.
waits before continuing to 325. Steps 325 and 327 may be At 420, 422, 424, and 426, square waves are generated for
performed by actively polling on the GPIO2 value or by 15 calculating a set of scalar products. In the present example,
waiting for an interrupt associated with the GPIO2 value, for two square waves are employed for each discrete frequency of
example. intensity variation to be detected. Each square wave alternates
At 330, the voltage on the GPIO2 signal pin is greater than between one and Zero at the frequency to be detected. In some
or equal to V and the sample timer value can be stored. This embodiments, the duty cycle of the waves may be 50%, such
corresponds to time T2 in FIG. 2. The value of the sample that the square wave is high (e.g., one) for half of the period
timer measures the rise time of the GPIO2 signal from a logic and low (e.g., Zero) for half of the period. In an alternate
Zero to a logic one. The sample timer value may be used as a embodiment, the square wave may alternate between positive
single sample of the ambient light intensity signal. At 335, it and negative values (e.g., positive one and negative one).
is determined if the charge timer is equal to the final value of Typically, the two square waves used for a given frequency
the charge timer. The final value of the charge timer may be 25 are deployed 90 degrees out of phase relative to one another.
Zero when the timer is decremented to show the passage of In some embodiments, such use of phase-shifted waveforms
time. In an alternative embodiment, the final value of the can enable frequency detection at an arbitrary phase, since the
charge timer may be another value when the timer is incre Source flicker and the sampling in the camera are generally
mented to show the passage of time. The charge timer values asynchronous.
are selected so that the charge timer reaches its final value 30 The present example is presented in the context of identi
when charge time 220 has elapsed, corresponding to time T3 fying whether nearby fluorescent light sources are powered
in FIG. 2. If the charge timer is equal to the final charge timer by 50Hz or 60Hz AC power. Accordingly, four square waves
value, method 300 continues at 340. If the charge timer is not are employed two at 100 Hz, and two others at 120 Hz. In
particular, at 420, a first square wave is generated with a
equal to the final charge timer value, method 300 continues at 35 frequency of 100 Hz. At 422, a second square wave is gener
337. At 337, method 300 waits before continuing to 335. ated having the same frequency as the first but 90 degrees out
Steps 335 and 337 may be performed by actively polling on a of phase. At 424, a third Square wave is generated with a
value generated by software or a hardware timer, or by wait frequency of 120 Hz. At 426, a fourth square wave is gener
ing for an interrupt associated with a hardware timer, for ated with same frequency as the third but 90 degrees out of
example. 40 phase. As shown at blocks 430, 432, 434, and 436, the square
At 340, GPIO1 and GPIO 2 are configured as outputs waves are mixed with samples 410 to generate Scalar prod
driving logic Zeroes. The discharge timer may be started. In uctS.
this manner, light detector circuit 40 may begin a discharge Specifically, at 430, 432, 434 and 436, scalar products are
state, such as at time T3 in FIG. 2. During the discharge state, calculated by mixing (multiplying) the samples individually
the parasitic capacitor of phototransistor 42 may be dis 45 with each of the four square waves. The resulting value of the
charged. At 345, it is determined if the discharge timer is summations (i.e., the S0, S1, S2 and S3 scalar products) will
equal to the final value of the discharge timer. The discharge depend on the particular implementation of the square waves,
timer values are selected so that the discharge timer reaches Such as whether the low portion of the square wave is nega
its final value when discharge time 230 has elapsed, corre tive, Zero or some other value.
sponding to time T4 in FIG. 2. If the discharge timer is equal 50 At 440 and 442, intermediate sums are generated for the
to the final value of the dischargetimer, method 300 continues Scalar products at each frequency. In particular, at 440, a 100
at 350. Otherwise, method 300 continues at 347 where HZ scalar product output is calculated by adding the absolute
method 300 waits before continuing at 345. value of scalar product S0 to the absolute value of scalar
At 350, method 300 prepares the timers for another itera product S1. At 442, a 120 Hz scalar product output is calcu
tion of the sample loop. The sample timer, charge timer, and 55 lated by adding the absolute value of scalar product S2 to the
discharge timers are initialized to their initial values. The absolute value of scalar product S3. The relative magnitudes
sample number is incremented and method 300 continues at of the 100 Hz intermediate sum and the 120 Hz intermediate
315. In this manner, a plurality of rise time samples from light Sum may indicate whether the underlying light Source is
detector 40 may be obtained. varying in intensity (flickering) at 100 Hz or 120 Hz, or
The plurality of rise time samples may be processed as 60 neither. This analysis may be performed, for example, at
illustrated in the example embodiment of FIG. 4 to identify block 450.
periodic intensity variations. More particularly, the example In particular, at 450, the intermediate sums are compared to
illustrates processing that is used to determine whether the determine whether the underlying light source is varying in
light that was incident on the detector was varying in intensity intensity at one of the pre-selected frequencies, i.e., 100 Hz or
at one of a plurality of pre-selected frequencies. For example, 65 120 Hz in the current example. For example, a relatively
the flickeroflight from a fluorescent lamp may be analyzed to larger intermediate Sum may indicate that flicker is occurring
determine if the fluorescent lamp is powered by a 50 Hz AC at the frequency associated with that intermediate Sum. More
 US 8,330,829 B2
10
particularly, depending on the number of samples and experi performed by first running a single-frequency search for 100
mental observation, it might be established that whenever the Hz flicker, and then successively running a search for 120 Hz
100 Hz, intermediate sum is larger than the 120 Hz, interme flicker.
diate sum by a factor of four, then the setting is a 50 Hz AC When searching for flicker occurring at a particular fre
environment. Conversely, a 60 Hz AC environment would be 5 quency, it may also be desirable to search in a range Surround
inferred when the 120 Hz intermediate sum was at least four ing the specific frequency of interest. For example, if the main
times larger than the 100 Hz intermediate sum. If neither frequency of interest is 100 Hz, the method might be imple
condition existed, then it could be inferred, for example, that mented to yield a positive determination in the event offlicker
the light environment contained no appreciable flicker (e.g., anywhere in the range from 97-103 Hz. Such tolerances may
no nearby fluorescent light source). 10 be achieved by looking specifically for the neighboring fre
The above example is but one possibility for the compari quencies (e.g., by employing square waves and Scalar prod
son performed at block 450. Each intermediate sum can be ucts for those frequencies), and/or through empirically-deter
compared to a pre-determined threshold, to a relative thresh mined thresholds for the scalar product values of the center
old, or to other values. As indicated above, thresholds or other frequency.
triggering values may be derived through experimentation. 15 Still further, the methods described herein may also be used
Theoretical methods may also be employed. in situations involving flicker at multiple different frequen
A relative threshold may improve detection accuracy in cies. In Such a setting, square waves and Scalar products may
certain lighting conditions. In certain settings, for example, be generated for each frequency or range of interest. Thresh
the inventors have observed that large discrepancies between olds could then be consulted to determine whether or not
the intermediate sums correlate strongly with the presence of flicker was present at a particular frequency. As in the other
flicker at a particular intensity. Specifically, when a recording examples, the thresholds could be arrived at through theoreti
device is exposed to an environment with a fluorescent light cal methods or empirical testing and observation.
powered by 60 Hz AC, use of the present methods results in a The methods described in FIGS. 3 and 4 may be imple
120 Hz, intermediate sum that is significantly larger than the mented on a digital signal processor (DSP) or other suitable
100 Hz intermediate sum. As briefly mentioned above, a 25 device. DSPs typically employ special purpose hardware and
factor of four has been employed in one embodiment to instructions to efficiently process signals of this nature. How
establish relative thresholds as follows: (1) when the 100 Hz ever, the special purpose hardware and instructions may come
intermediate sum is at least four times larger than the 120 Hz at increased cost and/or higher power than other processors,
intermediate Sum, the ambient environment is deemed to such as the PIC18F2550, for example. The methods described
include 100 Hz flicker (e.g., 50 Hz AC power); and (2) when 30 in FIGS. 3 and 4 are concerned with detecting one or more
the 120 Hz intermediate sum is at least four times larger than specific pre-selected discrete frequencies. This task typically
the 100 Hz intermediate sum, the ambient environment is is less computationally expensive than obtaining samples of a
deemed to include 120 Hz flicker (e.g., 60 Hz AC power). waveform and reconstructing the waveform from the set of
As will be appreciated from the above example, in some samples. Complexity may also be reduced as a result of
cases the comparison at block 450 will not be conclusive. One 35 searching for a limited number of frequencies, as opposed to
intermediate sum may not be significantly larger than the employing a method geared to comprehensive, multi-fre
other. In particular, the video recording device may be used quency detection and analysis. Therefore, the methods
outdoors, or in a room where natural light is much more described in FIGS. 3 and 4 may have an added advantage that
intense than that produced by fluorescent lighting or another they may be implemented on an inexpensive and/or relatively
flickering source. In such a case, the comparison at 450 may 40 simple processor.
conclude with a determination that no appreciable flicker is Use of square waves can also yield various efficiency
present. related benefits. Square waves can simplify calculations, rela
Regardless of the particular way in which determinations tive to other waveforms, which can enable software/firmware
are reached, the comparison at 450 may be followed by an implementations to carry out tasks using reduced code space
appropriate setting of the exposure timings. For example, 45 and execution time. Square waves may also be simpler to
upon determining at block 450 that the light source was synthesize in Some cases, for example, compared to sine or
flickering at 100 Hz, the exposure time would be adjusted cosine waves.
accordingly at block 460, e.g., by adjusting the integration The particulars of the sampling operation may affect the
time of the image sensor. As previously discussed, adjusting ability to detect flicker in the ambient light. Typically, the
the exposure timings to account for known flicker can Sub 50 sampling rate will be selected to satisfy the Nyquist-Shannon
stantially reduce visual artifacts in recorded images. sampling theorem, which requires sampling to occurat a rate
Though the above examples are discussed primarily in the greater than twice the highest frequency to be detected.
context of 100 Hz, and 120 Hz flicker, many other possibilities Accordingly, the components of light detector circuit 40
exist. Instead of searching for intensity variations at one of (FIG. 1) and the associated processing should be configured
two pre-selected frequencies, variations may be searched for 55 to sample at a rate of at least 240 Hz in order to identify 120
at only a single pre-selected frequency, or at three or more HZ intensity variations. In addition, the performance of the
frequencies. Still further, the methods herein may be scalar product technique described above may be affected by
employed to search for flicker in an entire range of frequen the number of samples that are collected via the method of
cies, for example by stepping through a range at regular FIG. 3 and employed at 410 (FIG. 4). Generally, it will be
frequency increments. An example of searching at a single 60 desirable to use fewer samples, in order to conserve comput
frequency would be to search for the frequency of the refresh ing resources and reduce processing time. On the other hand,
rate of a computer monitor, in order to compensate for light the sample should be large enough to generate meaningful
intensity variations contributed by the computer monitor. and reliable results.
Also, searching for flicker at multiple different frequencies In the context of the scalar product method described
may be achieved by performing multiple iterations of a 65 above, it has been determined that a desirable number of
single-frequency search, but successively at different fre samples may be calculated based on divisors/factors of the
quencies. Indeed, the above examples could alternately be pre-selected frequencies of interest. Specifically, in the case
 US 8,330,829 B2
11 12
of two frequencies, a greatest common factor is identified. methods and processes are implemented, the state of the
Then, the remaining factors of the two frequencies are mul data-holding Subsystem may be transformed (e.g., to hold
tiplied together, and that result is multiplied by four. This different data). The data-holding subsystem may include
example method yields a whole number of periods for both removable media and/or built-in devices. The data-holding
frequencies of interest, which can potentially avoid spectrum 5 Subsystem may include optical memory devices, semicon
leakage and the appearance of spurious frequencies. The ductor memory devices, and/or magnetic memory devices,
resulting figure, denoted below as N, forms one basis for a among others. The data-holding Subsystem may include
beneficial number of samples. Specifically, if roughly N devices with one or more of the following characteristics:
samples are employed (or a multiple therof), it has been found Volatile, nonvolatile, dynamic, static, read/write, read-only,
that the scalar product calculations may be performed effi 10 random access, sequential access, location addressable, file
ciently and rapidly, and that the results strongly and accu addressable, and content addressable. In some embodiments,
rately identify when flicker is present at one of the two fre the logic Subsystem and the data-holding Subsystem may be
quencies. In addition, doubling the base sample number of integrated into one or more common devices, such as an
samples in some cases can provide an improved signal to application specific integrated circuit (ASIC) or a system on
noise ratio. 15 a chip (SOC).
Referring again to the specific context of an ambient light It is to be understood that the configurations and/or
environment that might include fluorescent light sources approaches described hereinare exemplary in nature, and that
powered by 50 Hz or 60 Hz AC, the potential intensity varia these specific embodiments or examples are not to be consid
tions would occur at either 100 Hz or 120 Hz. Thus the ered in a limiting sense, because numerous variations are
method would entail processing the samples to determine possible. The specific routines or methods described herein
whether the incident light was varying in intensity at either may represent one or more of any number of processing
100 Hz or 120 Hz. Using the above methodology, the greatest strategies. As such, various acts illustrated may be performed
common factor is 20. The remainder factor for 100 Hz, is 5 in the sequence illustrated, in other sequences, in parallel, or
(i.e., 100/20) and the remainder factor for 120 Hz is 6 (i.e., in some cases omitted. Likewise, the order of the above
120/20). The remainder factors are then multiplied together, 25 described processes may be changed.
and further multiplied by a factor of 4, thereby yielding 120. The subject matter of the present disclosure includes all
Thus, the desirable base number of samples N is 120. novel and nonobvious combinations and Subcombinations of
More generally, when attempting to identify intensity the various processes, systems and configurations, and other
variations occurring at either a first frequency, f1, or a second features, functions, acts, and/or properties disclosed herein,
frequency, f2, the base number of samples N may be calcu 30 as well as any and all equivalents thereof.
lated as follows: For integer frequencies, each frequency may The invention claimed is:
be factored into a greatest common factor c (i.e., common to 1. A method for detecting flicker in photographic settings,
each frequency f1 and f2), and a remainder factor. In particu comprising:
lar, fl=ca, and f2=cb, wherea and b are remainder factors. obtaining approximately an integer multiple of N rise time
The number of samples may be calculated as the remainder 35 samples from a light detector circuit of a camera, N
factors multiplied by four: Number of samples-ab4. In being derived using a greatest common factor of a first
Some settings, it may be sufficient to use an approximate frequency and a second frequency, a value of each of the
multiple of Nsamples (e.g., within 10% or 15% of a multiple rise time samples being dependent upon light that is
of Nsamples) to obtain suitable results. incident upon the camera; and
The multiplier of four is employed in certain examples 40 processing the rise time samples to determine whether the
because 360 degrees divided by 90 degrees is four, which may light that is incident upon the camera is varying in inten
simplify the calculation of a square wave that is 90 degrees sity at one of a plurality of frequencies including the first
out of phase. frequency and the second frequency.
As described above in FIG. 1, camera 100 may include 2. The method of claim 1, wherein each of the rise time
computing system type components that may perform one or 45 samples is generated by charging the light detector circuit and
more of the above described methods and processes. For measuring a rise time of the light detector circuit.
example, the camera may include a logic Subsystem and a 3. The method of claim 1, wherein the first frequency is 100
data-holding Subsystem. HZ and the second frequency is 120 Hz.
When included, a logic Subsystem may include one or 4. The method of claim 1, wherein N is equal to a product
more physical devices configured to execute one or more 50 of four, a first remainder factor and a second remainder factor,
instructions. For example, the logic Subsystem may be con wherein the first remainder factor is the first frequency
figured to execute one or more instructions that are part of one divided by the greatest common factor and wherein the sec
or more programs, routines, objects, components, data struc ond remainder factor is the second frequency divided by the
tures, or other logical constructs. Such instructions may be greatest common factor.
implemented to perform a task, implement a data type, trans 55 5. The method of claim 1, wherein processing the rise time
form the state of one or more devices, or otherwise arrive at a samples includes generating a first scalar product output by
desired result. The logic Subsystem may include one or more multiplying the rise time samples by a periodic signal having
processors that are configured to execute software instruc a frequency equal to the first frequency, and generating a
tions. Additionally or alternatively, the logic Subsystem may second Scalar product output by multiplying the rise time
include one or more hardware or firmware logic machines 60 samples by a periodic signal having a frequency equal to the
configured to execute hardware or firmware instructions. The second frequency.
logic Subsystem may include general purpose input and out 6. The method of claim 5, wherein the periodic signal
put logic to communicate with circuitry of camera 100. having a frequency equal to the first frequency is a square
When included, a data-holding Subsystem may include one wave, and wherein the periodic signal having a frequency
or more physical devices configured to hold data and/or 65 equal to the second frequency is a square wave.
instructions executable by the logic Subsystem to implement 7. The method of claim 5, wherein processing the rise time
the herein described methods and processes. When such samples further includes comparing the first scalar product
 US 8,330,829 B2
13 14
output to the second scalar product output to determine product output further includes multiplying the plurality of
whether the light that is incident upon the camera is varying in samples with an additional 120 HZ square wave that is phase
intensity at either the first frequency or the second frequency. shifted by 90 degrees.
8. The method of claim 1, further comprising setting an 13. The method of claim 9, where the light detector circuit
exposure timing of the camera in response to determining 5 includes a phototransistor from which the plurality of
whether the light that is incident upon the camera is varying in samples is obtained.
intensity at one of the plurality of pre-selected frequencies. 14. A camera configured to automatically compensate for
9. A method for detecting for detecting flicker in photo periodic variations in light intensity, comprising:
graphic settings, comprising: a light detector circuit;
obtaining a plurality of Samples from a light detector cir 10 a logic Subsystem operatively coupled with the light detec
cuit of a camera, where a value of each of said plurality tor circuit; and
of samples is dependent upon intensity of light to which a data-holding Subsystem including instructions execut
the camera is exposed; able by the logic subsystem to:
generating a first Scalar product output by multiplying the 15 obtain a plurality of samples from the light detector
plurality of Samples with a square wave having a first circuit, where a value of each of said plurality of
frequency; samples is dependent upon intensity of light to which
generating a second scalar product output by multiplying the camera is exposed;
the plurality of samples with a square wave having a generate a scalar product output by multiplying the plu
second frequency; rality of Samples with a periodic signal having a pre
comparing the first scalar product output and the second selected frequency; and
scalar product output to determine whether the light process the scalar product output to determine whether
incident upon the camera is varying in intensity at either light that is incident upon the light detector circuit is
the first frequency or the second frequency; and varying in intensity at the pre-selected frequency.
setting an exposure timing of the camera in response to 25 further configured toofgenerate
15. The camera claim 14, where the instructions are
a plurality of Scalar product
determining whether the light incident upon the camera outputs by multiplying the plurality of samples with each of a
is varying in intensity at either the first frequency or the plurality of periodic signals having frequencies selected from
second frequency. a group of pre-selected frequencies.
10. The method of claim 9, wherein each of the plurality of 16. The camera of claim 15, where each of the plurality of
samples is a measured rise time associated with a capacitance 30 periodic signals is a square wave.
charging of the light detector circuit.
11. The method of claim 9, wherein generating the first configured tocamera
17. The
process
of claim 15, where the instructions are
the plurality of scalar product outputs to
Scalar product output includes multiplying the plurality of determine whether light
samples with a 100 Hz square wave, and generating the sec tor circuit is varying in that is incident upon the light detec
intensity at any of the pre-selected
ond scalar product output includes multiplying the plurality 35 frequencies.
of samples with a 120 Hz square wave. 18. The camera of claim 14, wherein the instructions are
12. The method of claim 11, wherein generating the first further
Scalar product output further includes multiplying the plural responseconfigured
to the
to set an exposure timing of the camera in
processing of the Scalar product output.
ity of samples with an additional 100 Hz square wave that is
phase-shifted by 90 degrees, and generating the second scalar k k k k k
 UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
PATENT NO. : 8,330,829 B2 Page 1 of 1
APPLICATIONNO. : 12/755952
DATED : December 11, 2012
INVENTOR(S) : Goh et al.

It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:

In the Specification
In column 11, line 9, delete “therof and insert -- thereof.--, therefor.

In the Claims

In column 13, line 8, In Claim 9, after “detecting delete “for detecting.

Signed and Sealed this

Seventeenth Day of February, 2015
74-4-04- 2% 4 Michelle K. Lee
Deputy Director of the United States Patent and Trademark Office