Milbeaut ISP for Digital Cameras

 Kentaro Endo

The Milbeaut image signal processor (ISP) enables a digital camera system to be implemented
on a chip. It can process signals from a variety of image sensors, perform enhanced-quality
image processing, and store 20-megapixel-plus still images and full high-definition (Full HD)
video. This paper introduces the MB91696AM ISP, the sixth generation of the Milbeaut series.
It features high-speed continuous shooting and Full HD video capture thanks to a newly
developed high-speed signal-processing platform. It provides enhanced noise-reduction
performance and optical-correction functions that have been improved with every Milbeaut
generation and incorporates general-purpose image-processing macros to satisfy market de-
mands. As a processor that provides high-performance and high-quality processing of still
images to Full HD video as desired by today’s camera users, the MB91696AM has come to
be used in many digital cameras that require a high pixel count, high-speed processing, and
multi-functionality.

1. Introduction the Milbeaut1), note1) series of ISPs. Beginning with the

The appearance of mirrorless interchangeable- M-1 series in 2000, Milbeaut represents an evolution of
lens cameras and the further evolution of camera image processing algorithms that are now in their sixth
functions such as video capture have helped to expand generation with the MB91696AM.
the digital-camera market to a scale of about 150 The MB91696AM system LSI is designed for use
million units. A digital camera basically consists of an with digital cameras. It features two ARM processors
image sensor and image signal processor (ISP), and from ARM Ltd. as independent CPU cores, sensor
there is a growing demand for digital cameras with signal processing, JPEG compression/decompression,
advanced image sensors and high-speed, enhanced- HD-video compression/decompression, a DDR3/DDR2-
image-quality, multi-functional ISPs. It would be no compatible SDRAM controller, a video interface, a USB
exaggeration to say that ISP performance is now the 2.0 interface, and a variety of card interfaces. With a
determining factor in digital-camera performance. significantly improved image processing algorithm and
Fujitsu’s MB91696AM ISP, which is introduced a newly developed H.264 Full HD codec engine, the
here, can perform noise removal and other types of MB91696AM provides a high-performance, high-im-
enhanced-image-quality processing for image sizes age-quality digital camera system-on-a-chip that meets
in excess of 20 megapixels. Combined with a high- the needs of today’s camera users, from still images to
speed image sensor, this ISP supports high-speed Full HD video.
continuous shooting at about 100 megapixel/s and full A block diagram of the MB91696AM is shown
high-definition (Full HD) video capture at 1920 × in Figure 1, and its external appearance is shown in
1080 pixels and 30 frames per second (fps). Figure 2. The main specifications are listed below.
• Processor technology: 65-nm CMOS process
2. Processor overview
note1) "Milbeaut" is a trademark of Fujitsu Semiconductor
The MB91696AM processor is the latest version of
denoting its series of ISPs for digital cameras.

10 FUJITSU Sci. Tech. J., Vol. 49, No. 1, pp. 10–16 (January 2013)
 K. Endo: Milbeaut ISP for Digital Cameras

MB91696AM
Image processing block Codec processing block External output block
Pre- Color LCD I/F

OSD control
processing D-LCD I/F
Sensor I/F processing Resolution Image
Demosaic
processing conversion filtering
JPEG codec H.264 codec HDMI I/F HDMI Tx
AE/AF/AWB Color-space C-DAC
conversion Video

SDRAM
Power I/F PWC AD I2C
GPIO CPU CPU Card
Battery RTC UDC PWM Card interface block
Line output Peripheral UART DMAC PC/USB
Audio codec CSIO
Speaker output macro block ARM

Microphone input

Figure 1
MB91696AM block diagram.
Figure 1
MB91696AM block diagram.
techniques for enhancing image quality, Full HD video
support, and multi-functionality.

3. High-speed processing
As the pixel count of image sensors increases, the
amount of processing required for high-speed continu-
ous shooting of still images and the capturing of Full
HD video increases at an exponential rate. Achieving
such processing at high speed with a high level of
image quality is difficult, however, with existing tech-
nologies. To resolve this problem, we developed a new
image-processing platform and thoroughly optimized
image processing for the MB91696AM ISP.
Figure 2
MB91696AM external appearance. 3.1 Two ARM processors
We developed the base firmware for the
MB91696AM by using an ARM platform, which is widely
• CPU-core operating frequency (max.): 288 MHz used in the development of many kinds of portable
• Image processing rate (max.): 216 megapixel/s devices. This approach provides the following benefits.
• Video codec: H.264/MPEG-4 advanced video 1) Increases development efficiency across multiple
coding customers
• HDMI interface: Yes 2) Simplifies customer-based customization
• Package type: FBGA-385 3) Facilitates implementation of diverse applications
• Package size: 13 mm square 4) Reduces development labor hours through the
The following sections introduce the key features use of extensive middleware and a rich develop-
of the MB91696AM ISP, namely, high-speed processing, ment environment

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 11

K. Endo: Milbeaut ISP for Digital Cameras

The MB91696AM incorporates two ARM proces- We also made advances in converting applica-
sors as independently operating cores. The base tions to hardware, which, in addition to speeding up
firmware can treat these processors as a main CPU and processing, is expected to reduce power consumption
a sub-CPU. It can allocate system control, operating sys- by reducing the processing load on the CPUs. This is
tem (OS) control, and sub-CPU control to the main CPU particularly effective for face-detection processing
and some control tasks to the sub-CPU. It can also treat when capturing Full HD video, which already places a
the latter as a dedicated CPU for executing image-pro- heavy load on the CPUs.
cessing applications. This scheme results in a number
of advantages. When shooting still images, it enables 3.3 High-speed continuous shooting
multiple image applications such as face recognition, High-speed continuous shooting requires high
scene recognition, panorama shooting, and high throughput for a series of signal-processing tasks con-
dynamic range (HDR) shooting to be launched simul- sisting of the input of Bayer-arranged signals from the
taneously so that parallel processing and high-speed image sensor, the processing of signals to enhance
image processing can be performed. Furthermore, this image quality, and the storage of data.
double-core approach makes it possible to capture Full While previous ISP processors featured only par-
HD video at 1920 × 1080 pixels and 30 fps while shoot- allel interfaces for the image sensor, the MB91696AM
ing video and running applications like face detection, is also equipped with a differential signal serial inter-
face recognition, and image stabilization (camera- face to connect with new types of image sensors. The
shaking compensation) simultaneously. MB91696AM can input Bayer-arranged signals at a
maximum rate of 216 megapixel/s when connecting
3.2 Optimized image processing to a high-speed image sensor, and it incorporates an
An ISP receives signals in a Bayer arrangementnote2) SDRAM controller supporting DDR3/DDR2 interfaces.
from the image sensor and performs various types of When combined with DDR3 SDRAM, the MB91696AM
image processing. High-speed continuous shooting can achieve a data bandwidth about 2.5 times that of
and the capturing of Full HD video, which demands previous processors and an image-processing perfor-
high-speed processing, requires that signals input from mance of about 100 megapixel/s. On top of that, it
the image sensor be processed one after the other in provides an SD-card interface supporting SDR50 mode
rapid succession. This requirement calls for pipeline of the UHS-I specification, enabling high-speed storage
processing. The MB91696AM’s image-processing block of JPEG data.
is divided into smaller blocks for processing signals Thanks to the above features, the MB91696AM
including a pre-processing block for processing signals can achieve high-speed continuous shooting of about
in a Bayer arrangement, a color-processing block that 8 fps (for 14-megapixel images), which is about
includes demosaic processing,note3) a resolution-con- 5 times faster than with previous processors.
version block, and an image-filtering block. We have
optimized these processes so that they do not require 4. Techniques for enhancing image
time-division processing, thereby enabling them to be quality
pipeline processed. Image sensors with higher pixel counts are a cur-
Circuits that were previously processed in different rent trend in digital cameras. Compact digital cameras
macro blocks within the image-processing block have frequently use 1/2.3-inch image sensors. These image
been incorporated into one macro block. This means sensors comprise over 15 million photosensitive ele-
that data storage to SDRAM and reference process- ments in an area of 6.2 × 4.6 mm, which is about the
ing that had previously been performed between two size of a fingertip. The trends in area/pixel and total
macro blocks can now be omitted. pixel count in image sensors used in compact digital
cameras are shown in Figure 3. This data shows how
note2) A mosaic arrangement of red (R), green (G), and pixel features have become finer year after year, reach-
blue (B) color filters
ing an area/pixel of 1.7 μm2 in 2011 models. Greater
note3) Interpolation processing to obtain an RGB signal
from a Bayer-arranged signal integration, however, is increasing the amount of heat

12 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013)

 K. Endo: Milbeaut ISP for Digital Cameras

50 18

Total pixel count Area/pixel

16

40
14

Total pixel count (megapixels)

12
Area/pixel (µm2)

30
10

8
20

4
10

0 0
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Sales year

Figure 3
Change in area/pixel and total pixel count of image sensors.

generated, resulting in the generation of thermal noise The first is defect pixel correction. Defect pixels
Figureof3the S/N ratio for data output
and the degradation included in a Bayer-arranged signal can be corrected
from the image sensor.in Compact
Change digitaland
area/pixel cameras,
total by count
pixel using a ofstatic process
image that identifies noise pixels
sensors.
moreover, make optical design difficult since they call from an image frame captured under light-blocking
for a compact and thin configuration while requiring conditions and then corrects them on the basis of pixel-
low-cost optical components to keep the sales price defect information set in SDRAM. Defect pixels can also
low. There is consequently a high possibility that opti- be corrected by using a dynamic process in real time.
cal problems like distortion and shading will occur. Contiguous defects can be corrected as well.
In light of the above, the demand for enhanced The second process is noise removal. Multiple
noise-removal performance and optical correction func- noise-removal filters can be used to remove not only
tions in ISPs is high. The Milbeaut series of ISPs has noise coming from the image sensor and hardware
been improving performance and functionality with but also noise unique to Bayer-arranged signals
every generation. In the following sections, we de- such as color moiré, block noise, and color uneven-
scribe how the MB91696AM enhances image quality. ness. A feature of the MB91696AM algorithm is that
long-wavelength luminance noise can be removed
4.1 Enhancing image quality against Bayer- while preserving the high-frequency components of
arranged signals the target object. In short, noise components can be
Optimal signal processing in the image-process- efficiently removed without losing a sense of high reso-
ing block differs depending on whether it is performed lution (Figure 4). The MB91696AM can also remove
before or after demosaic processing. First, we present noise by detecting the optical zero level and taking the
three main processes in the pre-processing block that optical-zero-level portion of the input signal into ac-
come before demosaic processing. count. This raises correction accuracy for dark areas of

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 13

K. Endo: Milbeaut ISP for Digital Cameras

Before correction After correction

Figure 4
Noise-removal effect.
Figure 4
Noise-removal effect.

Before correction After correction

Figure 5
Distortion-correction effect.

Figure 5
the image, where the signal level is low. achieves both high noise-removal performance and a
Distortion-correction
The third process effect
is the application of correction sense of high resolution can therefore be performed.
functions with respect to sensor output and the lens At the same time, applying a low pass filter (LPF) only
itself. The MB91696AM has built-in correction func- to the chrominance signal makes it possible to remove
tions for pixel gap, linearity, shading, distortion, and false colors or conspicuous color noise in low-saturation
chromatic aberration of magnification; it can therefore areas while maintaining contours.
deal with the optical problems associated with com- In addition, the MB91696AM adds contour en-
pact digital cameras mentioned above. An example hancement/correction components to an image after
of the distortion correction effect achieved with the it has been subjected to noise-removal processing. In
MB91696AM is shown in Figure 5. the past, components targeted for correction were ex-
tracted through the use of a high pass filter (HPF); for
4.2 Enhancing image quality after the MB91696AM, we have developed an algorithm that
demosaic processing reduces disturbance in the correction components with
After demosaic processing, the ISP converts the respect to contour pixels. This approach achieves im-
RGB signal into a YC signal, separates that into a lu- ages with sharp edges.
minance (Y) signal and a chrominance (C) signal, and
performs noise removal for each signal. For example, 5. Full HD video support
processing of the luminance signal may focus on the 5.1 Codec engine for digital cameras
signal level (brightness). In bright areas of the image, The first single-lens reflex camera equipped with
the level of the luminance signal is high and the noise a function for capturing Full HD video went on sale
level is relatively high as well. Adaptive processing that in 2008. It was then predicted that compact digital

14 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013)

 K. Endo: Milbeaut ISP for Digital Cameras

cameras would one day be equipped with such a func- great advantage over processors not equipped with this
tion, and, as it turned out, the MB91696AM was the color-noise-removal process for video capture.
first ISP in the Milbeaut series to support Full HD video
capture at 1920 × 1080 pixels and 30 fps. Needless 6. Multi-functionality
to say, capturing Full HD video requires the processing Recent years have seen an ongoing drop in the
and storing of a huge amount of data compared to still price of compact digital cameras despite gains in
images. With this in mind, we developed an H.264 Full performance and functionality. As a result, camera
HD codec engine specialized for digital cameras that makers find it necessary to place camera products with
is based on Fujitsu Laboratories Limited’s proprietary new functions on the market in relatively short prod-
H.264 codec algorithm, which is widely used in many uct cycles while keeping costs down. In addition to
audio-visual devices. This engine features low power software-based functions that can be easily updated,
consumption due to the use of an architecture that the MB91696AM comes equipped with general-pur-
substantially reduces the amount of image data that pose image-processing macros that can be executed
needs to be transferred. By using this engine, which through hardware processing that excels in terms of
efficiently compresses and encodes video data through processing time and power consumption. This scheme
dedicated hardware, the MB91696AM achieves clear, provides for a flexible response to ongoing demands
Full HD video capture with little noise. This function can for multi-functionality. We here introduce a few of the
operate simultaneously with a variety of applications many applications provided by the MB91696AM.
such as face detection, camera-shaking compensation, 1) Face detection
noise removal, motion detection, and wide dynamic Face detection is now considered to be an essen-
range (WDR) correction. tial function of digital cameras. The MB91696AM not
only can detect faces but can also distinguish between
5.2 Noise removal adults and children and detect smiles and blinking. It
The MB91696AM ISP provides the dedicated provides, moreover, a face-enhancement function that
hardware described above with a time-axis filter automatically corrects for facial blemishes such as fine
and removes noise during H.264 video compression. wrinkles, age spots, and dull skin. These processes are
Specifically, it compares information between consecu- achieved by dedicated hardware in the MB91696AM.
tive frames along the time axis and performs adaptive 2) Automatic scene recognition
alpha blending, which has the effect of reducing tem- The MB91696AM recognizes scenes by comparing
porally fluctuating random noise. Moreover, since the an input image with scene data stored in a database.
bit rate that had been consumed by random noise can It can then control exposure, focus, flash, and other
now be assigned to the video signal itself, this process parameters as needed for the recognized scene. It can
also has the effect of improving compression efficiency. recognize, for example, portrait, landscape, and macro
The color-noise-removal process performed in scenes as well as shooting conditions such as twilight,
still-image processing was expanded so that it can night, backlight, and low-light.
also be used in video capture. With the approach 3) HDR correction
previously used, performance of this processing by a HDR correction expands the dynamic range by
single hardware macro, there was no margin in the combining multiple images taken with different shut-
processing band, and the processing speed was insuf- ter speeds. For example, shooting with a shutter
ficient for capturing Full HD video. In response to these speed appropriate for highlights can capture tones in
problems, we added a faster version of this process that a highlighted area, and shooting with a shutter speed
removes color noise generated during video capture. appropriate for shadows can capture tones in a shadow
With this new approach, color noise in low-frequency area. Images shot with these different shutter speeds
bands, which is normally very difficult to remove, is re- can be combined using image filter processing in the
moved. This approach is quite effective when shooting MB91696AM so that highlight areas do not become
video in a high-sensitivity shooting environment such saturated and shadow areas become more visible,
as an indoor location. The MB91696AM ISP thus has a thereby expanding the dynamic range.

FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013) 15

K. Endo: Milbeaut ISP for Digital Cameras

7. Conclusion expected to drive expectations for network support and

In this paper, we introduced Fujitsu’s MB91696AM other advanced functions that go beyond the conven-
ISP, which was developed for digital cameras. The need tional framework of digital cameras.
is growing for higher processing performance to keep Looking forward, we will strive to determine
up with the trend toward higher pixel counts, faster market needs as accurately as possible and continue
processing speeds, and multi-functionality in digital to develop and market the high-performance, high-
cameras, as discussed in this paper. This trend toward picture-quality Milbeaut series of ISPs featuring ever
higher performance also has the effect of increasing evolving image-processing algorithms.
power consumption, so developing measures for reduc-
ing power has become an urgent matter. While not References
directly touched upon in this paper, we are also expend- 1) Fujitsu: Milbeaut Imaging Processors.
ing efforts on developing technologies for reducing http://www.fujitsu.com/global/services/
power consumption. At the same time, the deploy- microelectronics/product/assp/milbeaut/

ment of a high-speed communications infrastructure is

Kentaro Endo
Fujitsu Semiconductor Ltd.
Mr. Endo is engaged in the development
of image processing algorithms.

16 FUJITSU Sci. Tech. J., Vol. 49, No. 1 (January 2013)