Digital Camera in A Nutshell Image Sensor
Digital Camera in A Nutshell Image Sensor
Digital Camera in A Nutshell Image Sensor
Image sensor:
An image sensor is a device that converts an optical image to an electric signal. It is used mostly
in digital cameras and other imaging devices. Early sensors werevideo camera tubes but a modern one is
typically a charge-coupled device (CCD) or a complementary metal–oxide–semiconductor (CMOS) active
pixel sensor.
Using still more intelligence (a digital camera can contain several processors and other chips, including
application-specific processors--ASICs--and a master CPU), the data is analyzed according to internal,
brand and model-specific programming ("algorithms") and reassembled into a file that can be
recognized and read as a visual image. This image file is saved to some sort of built-in or external
electronic memory system. After this point, the image file can be downloaded to a computer, output to
a printer, or displayed on a television set. Or it may be internally accessed, or sub-sampled, to be
viewed on the camera's own LCD viewfinder, where the user has the option to apply still more
algorithms to it, using the onboard operating system interface (accessed, usually, on the LCD), or to
trash (delete) it and start over again.
Throughout this multi-step process, the camera's intelligence continuously polls the operating system,
to be able to instantly execute and integrate the photographer's settings (which he/she inputs using
the numerous dials, buttons, switches, LCD and control panel). It's a complex system, with lots of
data and instructions moving over numerous paths--all jammed into a small, lightweight battery-
powered box that you can hold in the palm of your hand.
The above describes just the bare bones outline of the digital image capture process. It is the details
that distinguish different digital cameras. So, let's look more closely at each step of this process in a
typical digital camera. (Exceptions to the basic design will be covered in future installments, as we
look more deeply at each component of the digital camera system.)
There are several reasons for this discrepancy. Some are so-called dark pixels, or, defective, inactive
pixels inevitably created during the manufacturing process (Creating a 100% perfect image sensor is
virtually impossible, using current sensor fabrication technology.) In addition, portions of the sensor
can be used for other purposes, such as to calibrate the signals read from the sensor. By masking a
small number of pixels at the edges so they are not exposed to light, the background dark current (or
noise) generated by the pixels can be determined, and this noise can be subtracted from the actual
image data. Also, portions of the sensor may be masked to create images with a specific aspect ratio.
(That's the proportion of horizontal to vertical dimensions.)
Incidentally, increases in image sensor size are logarithmic, not linear. Going from a 3MP to 4MP
sensor doesn't increase image size 25%, but by a smaller increment. That's why the newer digital
cameras with higher density image sensors offer only evolutionary, incremental image size boosts,
which may or may not be of importance to the user.
click on image for full view
Presently, all consumer digital cameras employ a single CCD or CMOS sensor. Some high-end
professional digital cameras, as well as many better camcorders, use multiple sensors, with the
incoming light equally divided among them via an optical beamsplitter (prism). Multiple sensors can
eliminate color aliasing, or the tendency for edges of red, blue, and green to separate in an image.
However, multiple sensor cameras require greater precision to build, and because of the beamsplitter,
they tend to be bulkier and less rugged. They also require advanced optics and more precise
manufacturing processes, so the overall cost (not to mention the greater engineering challenge) is
typically more than a single sensor camera.
Interestingly, multiple sensor devices do not always follow simple linear mathemathics. In the
standard scenario (which most multiple sensor camcorders follow), there are three separate red,
green and blue CCD or CMOS full resolution sensors, which contribute 1/3rd of the color information
for each pixel. That means, for instance, in a 3MP three-sensor camcorder, each of the sensors will be
a 3MP sensor. However, in still digital cameras, how the information from multiple sensors is used
varies with each manufacturer, and in fact, with each model.
Some three-sensor cameras will use sensors that each have 1/3rd the resolution of the full picture,
and use interpolation. But other multiple sensor cameras may use combinations of the primary colors
on each sensor, and develop complicated algorithms for combining them. For instance, the now
discontinued Minolta RD-175 had three CCDs, two of which were green, and the third, which was red
and blue. (This doubling of the green is similar to the Bayer pattern color filter array for single sensor
devices that is described below.) Each of the RD-175's sensors contained less than ½ MP, but through
mathematics, they were combined to create a rather good quality 1.7 MP image.
In many digital cameras, only a portion of the pixel is photosensitive, so it is important to direct as
much light as possible to the area that can capture it (a phenomenon called "fill factor"). For this
purpose, most consumer image sensors have a "microlens" directly over each pixel to direct the
photons down into the photosensitive "well" of the pixel. The photons are actually converted to
electrons by a silicon photodiode positioned on top of the well, and the well behaves like a capacitor,
capable of storing an electrical charge.
Digital Capture Of Your Image
Because image sensors are inherently grayscale devices and have no color to them, sensors used in
digital cameras typically employ a color filter array (CFA), wedged between the microlens and the
pixel well. The CFA is constructed to assign a single color to each pixel. Digital camera manufacturers
choose among a variety of CFA architectures, usually based on different combinations of primary
colors (red, green, blue) or complementary colors (cyan, magenta, yellow). Whatever type of CFA is
used, the idea behind all of them is to transfer only the color of interest, so each pixel sees only one
color wavelength. All CFAs also attempt to reduce color artifacts and interference between neighboring
pixels, while helping with accurate color reproductions. (Below, we explain how later in the data
stream, the camera's image processors reassemble the image from all these individual bits of colors.)
It also produces what appear to be sharper images. The science of perceived color versus scientifically
measured color is a complex issue that has generated numerous solutions. And different
manufacturers subscribe to various color models and algorithms for defining what they consider to be
the best color for digital cameras.
All digital cameras have the electronic equivalent of a shutter (it's not a traditional film-based
mechanical shutter) built into the image sensor. Its purpose is to regulate the precise length of time
that light strikes the image sensor. The electronic shutter is an on/off switch that closes off (or allows
access to) the image sensor to the incoming photon stream. Some digital cameras also incorporate a
more expensive mechanical shutter, not for redundancy, but to prevent additional light from hitting
the image sensor after the initial exposure is completed. This helps eliminate after-exposure artifacts
like ghosting, streaking or fogging (known in the industry as "smear").
When you press a digital camera shutter button halfway, its focus and exposure are frozen, in
anticipation of an imminent image capture. Those are exactly the same steps that occur when you
press the shutter button halfway on a typical point-and-shoot film camera. What happens next,
however, is radically different from a film camera. When the shutter button is pressed fully on a digital
camera, a number of events, or sequences, occur almost simultaneously.
1. The first event is the mechanical shutter closes (if the digital camera has one) and the sensor
is instantly flushed of any electrical charge. That is because the image sensor is always active,
allowing electrical charges to continuously fill up the sampling points. (On some better digital
cameras, the image sensor might be in a sleep state immediately prior to image capture, to help
reduce heat and improve the signal-to-noise ratio.). If no instructions are received, the image
sensor will also continuously flush those same electrical charges about every 1/60th of a second.
Therefore, before the image sensor can be ready to capture your picture, all residual electrical
charges must be flushed.
Interestingly, some digital cameras (such as the Olympus Camedia E-100RS) will hold the most
recently flushed data in a temporary memory buffer, and can display it to you after you shoot, in
case you prefer it to the scene you composed and captured. This ESP-like pre-shoot capture mode
is great for capturing kids or animals that tend to blink or move every time they hear the click of a
camera.
2. Whether the camera throws away the pre-shoot electrical charge or puts it into a temporary
memory buffer for possible selection by the photographer, one of the camera's several processors
makes use of the data to adjust and set the parameters for the photo you are about to take. For
instance, the camera's processor that controls white balance may use those values to try to
determine what pixels in the current image should be white. And it may try to adjust the overall
colors to remove any hue shift from that "white point." Similarly, the controls for focus, flash and
other pre-shoot determinations are set or activated, based upon the electronic charges flushed out
of the image sensor immediately before the picture is taken. These parameters are also held in a
buffer, so that they may be used later in the image-processing phase. If the LCD viewfinder is being
used to compose the picture, it will also receive this image data.
3. Once all electrical charge has been flushed from the sensor and the shooting parameters set,
the image sensor is ready to accept the image that you told the camera you wanted to capture,
when you pressed the "shutter" button. So, the camera opens the mechanical shutter and activates
the electronic shutter; both remain open or active according to the exposure time determined
earlier, with the mechanical shutter closing at the end of the exposure.
4. The shutter then opens again while the camera is recycling, and will close only when the
photographer presses the shutter button and starts the flushing process for preparing to take
another picture. If the processor (or the photographer) had decided to trigger an electronic flash for
the scene (usually the digital camera's built-in strobelight), it will illuminate the scene until a
separate light sensor decides that the flash has produced enough lumens for that specific exposure
and turns it off.
Note: Olympus envisions the digital capture process to take the following graphic form:
click on image for full view
Because of the time it takes for the image sensor flush, as well as to read and set the shooting
controls, there is always a brief, unavoidable, but often annoying delay, or lag, between when you
fully press the shutter button and when the picture is actually taken. In a typical consumer digital
camera, this delay can take anywhere from 60 milliseconds (that's so fast that it is virtually
instantaneous) to as long as 1½ seconds.
Integrating larger memory buffers and faster processors can reduce shutter lag, which is why most
expensive digital cameras tend to shoot faster than most budget devices. Among more expensive,
professional cameras, the new Nikon DH1 has a 128MB buffer, and some cameras, like Kodak's DCS
520, 620, and the Fuji S1, have 64MB buffers. There are only a handful of prosumer and higher end
consumer cameras with buffers as large as 16 MB or 32MB.
In addition, some image sensors (most specifically CMOS) are multi-functional chips that have
intelligence built into them, which helps reduce the time involved in transmitting and using
information captured by the sensor. Like any other digital systems, digital cameras will operate faster
when their internal bandwidth is improved.
On Becoming Digital
Backing up a bit in the process, when the image sensor converts incident photons to electrons, it is
dealing with analog data at this stage. The next step is to move the stored electrical charges out of
the sensor's pixels and convert them to a stream of voltages via a built-in output amplifier, and then
send the stream of voltages to an external or on-chip analog to digital converter (ADC).
One of the major differences between CMOS and CCD image sensors is that CMOS can have the ADC
on the sensor, while the CCD must use an external ADC chip. CMOS image sensors in particular are
inherently noisy and benefit from integrated ADCs. The ADC converts the various voltage levels to
binary digital data. The digital data is further processed and ultimately organized according to color bit
depth for the red, green, and blue channels, reflecting the color and intensity (or shade) associated
with a specific pixel.
Nomenclature Primer
Unfortunately, digital camera bit-depth nomenclature is inherently confusing. To understand it, we
have to look at the basics of digital color. All colors in a digital camera are created by combining the
light or signals of the three primary colors--red, green, and blue. These three primary colors are also
called channels.
Bit depth of color may be stated for each of the three channels (as in 10-bit, 12-bit, etc), or for the
entire spectrum by multiplying the channel value by three (30-bit, 36-bit, etc.). However, the
conventions of bit-depth nomenclature sometimes go beyond logic, and you just have to know certain
things. For instance, 24-bit color (which is sometimes also called True Color, because it is the closest
the digital world can get to the number of colors the human eye can perceive) is defined as 8 bits per
channel.
But 24-bit color is never called 8-bit color. When you hear someone talking about 8-bit color, it does
not relate to 8 bits per channel, but to 8-bits for the entire spectrum, or a total of 256 different colors,
which is a very limited spectrum indeed. In contrast, 24-bit color offers 16.7 million different colors.
One way to remember the naming convention is to think of 24-bit color as a dividing line. Any bit
depth over 24-bit can be named either according to the per channel or the full spectrum numbers.
From 24-bit depth and less, you'll seldom hear of anyone using anything other than the full spectrum
bit depth.
Until last autumn, almost all consumer digital cameras were 24-bit color devices (using 8-bit ADCs).
There are now some models, like the Olympus E-10 and HP PhotoSmart 912, that can generate 30 or
36-bit color data (10 or 12-bit ADCs). However, some digital cameras capable of capturing a higher bit
depth have an 8-bit ADC, which means that they can save only 24-bit color. (A few cameras, like the
Canon PowerShot G1, can save a RAW 36-bit image, but it is a proprietary file format that cannot be
read directly by any imaging program. While Photoshop can read up to 16-bits per channel, most of its
functions are not available for such files. Canon's upload software first must convert the data into a
TIFF file that Photoshop can use. What's more, most output devices usually are unable to use all that
data.)
In addition, the DSP defines the image resolution. While most digital cameras can be set to create
various resolution images, most of them will capture all the data that their respective image sensors
can deliver. For instance, when shooting in the VGA mode with a 3 megapixel digital camera, instead
of capturing only a 640x480 image, the camera will shoot and capture a full 20486x1548. Then, the
DSP will sample down or interpolate up to the resolution that the photographer selected in the
operating system (on the LCD or control panel) or by pushing a button when the shot was first set up.
However, some image sensors (usually CMOS) can selectively turn off pixels rather than down-
sampling or interpolating-up, thereby setting lower or higher resolutions at the time of the initial
capture. CMOS sensors have this ability, because they are direct access devices similar to RAM, and
they can quickly select the desired data by fast row/column access. That's in opposition to CCDs,
which are serial output devices, in which all data must be delivered, and the processor must then do
its sampling/interpolating process. Obviously, a CMOS chip that captures only the amount of data that
is wanted can speed up processing times.
In the same region of the image-processing stream where the data is converted to an image with a
specified resolution, each manufacturer applies its "secret sauce"--the algorithms that are brand
specific. In other words, DSPs add image enhancements based on the manufacturer's image
characterization. Thus the image processing performed by every camera is unique, and incorporates
its own color balance and saturation settings to produce what the manufacturer feels are ideal
pictures.
Similarly, sharpness or softness is applied, a preprogrammed white balance used, etc. It is here, in
the image processing stage, that you'll find the greatest differences among various digital cameras
produced by different companies.
The advantage of using removable memory is that you can swap out the memory card or device when
it is full, and replace it with a fresh one. This allows you to keep shooting, without taking the time to
upload all the pictures to your computer and then erase the memory. In addition, removable memory
gives the user the flexibility of upgrading to higher storage capacities.
Viewfinders
At the same time that the image is processed and saved to memory, it also may be displayed on the
LCD viewfinder or electronic eye-level viewfinder. Most LCD viewfinders are 1.8" or 2" TFT color panels
consisting of between 65,000 and 220,000 pixels, with a refresh rate between 1/8th and 1/30th of a
second. They are designed to be viewed at an optimum distance of 8" to 18".
It is almost always preferable to use your camera's eyelevel viewfinder for composing your images,
and use the LCD primarily for setting parameters and then viewing the captured image. Even when
using high-resolution LCD viewfinders, digital cameras sample the image so you don't see a direct 1-
to-1 resolution on the viewfinder. Therefore, they cannot be easily used for detailed focusing or
framing. Worse yet, LCDs are voracious power consumers and can quickly drain a set of batteries if
used heavily. Another major disadvantage is that because of their proximity to the CCD or CMOS
image sensor, in a typical camera design, LCDs can produce noise that translates into unwanted visual
artifacts. (That's one of the major benefits of the articulated LCD viewfinders that swivel away from
the camera body, such as on the Canon G1. It moves the LCD, and its potential to generate noise on
the sensor, even further from the sensor.)
A much less expensive and less complicated eyelevel viewing solution is the ubiquitous optical glass
viewfinder, which is used in most consumer digital cameras. It's made of clear glass, and rather than
showing exactly what the lens sees (through-the-lens, or TTL viewing), it sights along the top or side
of the lens. Its advantages are that it uses no power, has no moving parts, and is brighter than any
TTL system. However, it tends to be very inaccurate (it generally shows much less than is actually
captured, so you can end up with unwanted material along the edges of your picture), and introduces
parallax.
Parallax occurs because the viewfinder is positioned 1" or 2" away from your lens, and therefore you
are viewing the scene at a slightly different angle than the lens. This doesn't matter when you are
shooting far-away scenes, but as you get closer, the difference in angle between what you see and
what is actually captured increases. When shooting macros (12" or closer to your subject), glass
viewfinders are virtually useless because the parallax error is so great.
The electronic eyelevel viewfinder is a newer technology that replaces the optical viewfinder with a
tiny, low-powered, high-resolution color monitor that you view by holding the camera up to your eye.
In addition to direct, detailed viewing that can clearly show whether a subject is in focus or not, most
electronic viewfinders also display important data about the photographer's settings--f-stop, shutter
speed, flash status, etc. The major disadvantage is that this technology, while popular in camcorders,
is relatively new and primitive in still digital cameras and, therefore, may not always be as bright,
clear or responsive as a traditional optical viewfinder.
As with an LCD viewfinder, an electronic eye-level viewfinder also can display a lower resolution
version of the saved image, which has been sub-sampled by a processor. Or, it may be an electronic
thumbnail from the TIFF or JPEG file header. As the technology improves, we expect electronic
eyelevel viewfinders will also replace LCD viewfinders in many models.
The number of processors, DSPs, ASICs and other chips varies widely among digital camera brands
and models. However, one increasingly popular trend is to consolidate as many functions as possible
onto fewer chips, to save cost and space in the design.
All this on-board processing requires lots of electrical power. In previous years, when we tested digital
cameras, we needed to stock up with scores of AA alkalines. This is because digital camera power
drain was so significant that it was necessary to change batteries after taking relatively few pictures.
The current generation of digital cameras has improved, both in electrical efficiency and in their ability
to better utilize battery power. Many digital cameras have abandoned alkalines for more efficient and
advanced battery technologies, such as rechargeable nickel metal hydride or lithium-ion batteries. And
a handful of manufacturers, like Sony, have developed "smart" batteries for their digital cameras that
inform the user, to the minute, precisely how much power is left (To learn more about Battery
technology, please read="Batteries: History Present and Future of Battery Technology").
As cameras become more sophisticated, with more components and greater speed requirements,
power consumption and efficiency will continue to be an important area for future developments.
Digital Camera Quality... Much More than Just Pixels
It is important to understand that digital cameras are true systems, in which the picture is a result of
a sum of the parts and how well they function together. No one component alone determines image
quality or speed or efficiency, though a single bottleneck in the process can throw the whole system
off and negatively impact on overall image quality.
In early digital cameras, the most significant limiting factor was the comparatively poor quality and
tiny size of the image sensors (about the size of a pea, back then). Camera manufacturers realized
that fine, high-quality lenses would be wasted if used in such cameras given that their image sensors
were not able to capture high quality images. So, the first consumer digital cameras used small,
plastic lenses with relatively poor optical quality.
At the same time that sensor densities increase, everything else probably will become smaller and
more compact, so that the cameras themselves can be miniaturized. At present, smaller cameras are
technological compromises that must do without some functionality in order to fit into such a compact
size. But as chips are consolidated for multiple functions and the various technologies become more
efficient, future miniaturized cameras will have full functionality.
click on image for full view
Another approach to miniaturization will involve radical re-engineering of the camera itself. For
example, the new Olympus Brio D-100 is unusually slim for a digital camera. The only way they could
fit in the optics and all the components into such a skinny package was to position the CCD at 90
degrees to the lens, using a mirror system in between the lens elements to angle the light. It's a
simple, though optically revolutionary idea that involved some significant new designs.
Conversely, larger, pro-like cameras will continue to invade the consumer price range. At the lower
end of the market, less expensive, low-resolution cameras will come into their own. Despite their
relatively low resolutions, basic image quality could equal or even surpass some current high-
resolution devices. (Remember, the number of pixels is only one aspect of digital capture, and image
quality is a synthesis of the entire process.)
Each succeeding generation of digital cameras will exhibit more intelligence than ever, as they edge
towards true multifunction devices. Convergence will become the catchword, as digital cameras, digital
video camcorders, voice recorders, videoconferencing cameras, PDAs, and cellular telephones begin to
blur into single devices. So, we'll see even greater ingenuity applied in image processing and camera
engineering to counter noise and other problems inherent in jamming so much electronics into such a
small space. And, of course, the prices will come down, while quality and performance will rise. It will
be an exciting time for digital photographers...which means for all of us.