See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/277653033

Separation of Manga Line Drawings and Screentones

Conference Paper · May 2015

DOI: 10.2312/egsh.20151018

CITATIONS READS

7 2,299

4 authors, including:

Toshihiko Yamasaki Kiyoharu Aizawa

The University of Tokyo The University of Tokyo
260 PUBLICATIONS   1,758 CITATIONS    627 PUBLICATIONS   4,936 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

PQk-means View project

Approximate Nearest Neighbor Search with Selective K-means Tree View project

All content following this page was uploaded by Kiyoharu Aizawa on 03 June 2015.

The user has requested enhancement of the downloaded file.

EUROGRAPHICS 2015/ B. Bickel and T. Ritschel Short Paper

Separation of Manga Line Drawings and Screentones

Kota Ito1 , Yusuke Matsui1 , Toshihiko Yamasaki1 and Kiyoharu Aizawa1

1 The Universitiy of Tokyo, Japan

Abstract
Screentones are unique expressions of Japanese comics (manga), which enrich their visual expression. However,
such screentones have a very different visual nature from that of line drawing areas; this prevents us from applying
various kinds of image processing techniques to manga. We propose a method for extracting line drawings and
removing screentones. We employ Laplacians of Gaussian filters and flow-based differences of Gaussian filters,
one for removing screentones and the other for preserving lines, and make a binary mask for separating line
drawings from manga by merging the results of the two filters. We show that the proposed method successfully
separates line drawings and is better than existing methods in comparative studies.
Categories and Subject Descriptors (according to ACM CCS): I.4.6 [Image Processing and Computer Vision]:
Segmentation—Edge and feature detection

1. Introduction
Manga (Japanese comics) are popular all over the world. A
unique trait of manga is their use of screentones, which are
preprinted patterns that express visual effects such as tex-
tures and shadows (Fig. 1(a)). Manga authors draw manga by
drawing lines and filling areas with screentones (Fig. 1(b)).
Currently, comic books are increasingly being pub-
lished in electronic form as well as in print, and research
on new applications of manga image processing has ap-
peared, e.g., manga retrieval [MAJ14], interactive segmen-
tation [AMYA14], and element composition for drawing as-
(a) Examples of various (b) A manga image consists of a line
sistance [CLC14]. kinds of screentones. drawing layer and a screentone layer.
However, it is not clear how to handle screentones in dig-
ital images. The nature of screentones is very different from Figure 1: Manga components. Note that a black-filled area
that of line drawings, which might cause undesired effects is considered to be a screentone in this paper. Junichiro
c
if we do not take the screentones into consideration when Akabi
manga images are processed. Therefore, we want to separate
screentones and line drawings in advance before the main
processing is applied, to make full use of the visual structure
(Fig. 1(a)), it is hard to extract all kinds of screentones with
of the manga image. Further, the separation enables us to ap-
a single filter. To handle them, we create two masks: (1) a
ply image processing methods developed for line drawings.
screentone removal mask, which deletes all screentone ar-
We propose a method for extracting line drawings and re- eas, and (2) a line preserving mask, which preserves lines
moving screentones. The method consists of applying sev- as much as possible. By appropriate merging of the results
eral filters and merging the results. Because screentones have from these two masks, we compute the final mask, which
a wide variety of statistical natures, including large/small, specifies line drawings. Our method does not require any
isotropic/anisotropic, and low/high frequencies of textures parameter tuning and is very robust to the kinds of screen-

c The Eurographics Association 2015.

 Kota Ito & Yusuke Matsui & Toshihiko Yamasaki & Kiyoharu Aizawa / Separation of Manga Line Drawings and Screentones

(a)

Figure 2: Work flow. Hotaru

c Yoshizawa

tones, whereas previous line extracting methods for manga

are suitable only for a single kind of texture, e.g., clean filled (b)
areas [HYLW10], or simple dot patterns [KL12]. Note that
Figure 3: Effect of LoG parameter: (a) LoG masks with var-
Qu and colleagues proposed a robust extraction method for
ious values of i Takuji.
c (b) Change of criteria with i: top:
screentones [QWH06], but it requires manual interactions.
NCC(MLoGi ); middle: STC, and bottom: CCC.

2. Method
We show the flow of our method in Fig. 2. First, we create Bin(IˆLoGi ) is a binary of IˆLoGi by Otsu’s method [Ots75]. We
a screentone removal mask Mrm , which removes screentone omit a description of input x for readability.
areas as much as possible, and a line preserving mask Ml p ,
which retains as many lines as possible. Then, we combine The output of the LoG filter changes with i as shown in
these masks, taking account of connected components. Fig. 3(a). We must select an appropriate value for i to remove
screentones; i = 19 is the best in this case. To select i, we
introduce two criteria, the Connected Component Criteria
2.1. Screentone removal mask (CCC) and the Stop Criteria (STC), which are defined as
The purpose of a screentone removal mask is to delete all follows:
screentone areas while allowing the removal of some por- NCC(MLoGi )
CCCi = 1 − , (3)
tion of line drawings. To create a screentone removal mask, NCC(MLoGi−2 )
we employ a Laplacian of Gaussian (LoG) filter, which acts
as a bandpass filter in the Fourier domain, and works as a NCC(MLoGi )
line detector. Because screentones tend to have periodicity or STCi = × |(NCC(MLoGi−2 ) − NCC(MLoGi ))|,
NCC(MLoG1 )
high-frequency components, they can be removed by a LoG (4)
filter while preserving line drawings. We found two traits of where NCC(M) is the number of connected components of
outputs of LoG filtering: (1) the LoG value is relatively high black pixels in a mask M. CCCi is a ratio showing the de-
at line drawings, and low in other areas; (2) a pixel that has crease in the number of connected components when i is in-
a negative LoG value hardly ever belongs to a line drawing. cremented. If CCCi has its highest value at i∗ , it means the
By considering these observations, we create an LoG mask most screentones are removed at i∗ . We empirically found
MLoGi as shown below: that i values a little larger than i∗ are best for our purpose.

0 if ILoGi (x) < 0 If we increment i much more, the lines become too blurred.
IˆLoGi (x) = (1) Examples are shown in Fig. 3(a) and Fig. 3(b).
ILoGi (x) otherwise,
STCi is used for the stopping condition of the process.
MLoGi = Bin(IˆLoGi ), (2) If STCi is smaller than a predefined threshold, the process
stops. STCi denotes the amount of change of NCC(MLoGi )
where ILoGi is the output of a LoG filtering of an input image
(compared with the ratio between NCC(MLoG1 ) and
I, and i is the window size of a Gaussian filter. i must be odd.
NCC(MLoGi ), i.e., NCC(MLoGi )/NCC(MLoG1 )), as shown in
Note that i automatically determines a standard deviation pa-
Fig. 3(b).
rameter of the filter. IˆLoGi is an image in which each pixel
IˆLoGi (x) is ILoGi (x) if ILoGi (x) is positive and 0 otherwise. We show an algorithm for creating a removal mask Mrm in

c The Eurographics Association 2015.

 Kota Ito & Yusuke Matsui & Toshihiko Yamasaki & Kiyoharu Aizawa / Separation of Manga Line Drawings and Screentones

create visually plausible line drawings. FDoG extracts a set

of coherent isotropic lines. It is useful for manga because
they have clear isotropic line drawings.
We must select two parameters to compute FDoG; a
threshold of binarization τ, which is set as 0.05, and a win-
dow size for filtering wDoG . If wDoG is smaller than the
(a) Result without a (b) Result with a line width, the FDoG filter does not work well. Therefore,
base mask (bold). base mask (thin). we select wDoG automatically, by using ibase , which denotes
a scale space of the input image and was computed when
Figure 4: The effect of a base mask MLoGibase . Motoi
c Tak-
making the screentone removal mask. Experimentally we se-
enaka
lected wDoG as wDoG = ibase − 2.

2.3. Mask merging for refining line extraction

Alg. 1. α is a threshold of STC, and set to 1. β is used to con-
trol a search range. If β = 1, i of max CCCi is simply found. After generating the two masks, we restore the line draw-
Empirically, we found larger i is better if there are several ings from the line preserving mask by using the screentone
similar peaks of CCCi . We therefore select a relaxation fac- removal mask as a guide. The algorithm is shown in Alg. 2.
tor β, i.e., if there are similar peaks, we select the last one. β First, we make a combined mask Mcomb by taking a pixel-
is set as 0.8. By checking CCCi and STCi in the while loop, wise logical OR between Mrm and Ml p (line 3). R is a set of
the best parameter iLoG is found. This procedure works well pixels that belong to Mrm . Then we separate Mcomb into con-
except when there are black-filled screentones and iLoG is nected components. CCn (M) in line 8 represents the nth con-
too large. In such cases, lines around the black-filled areas nected component of M. We consider a connected compo-
tend to be bold as shown in Fig. 4(a). To handle such excep- nent as a unit for the restoration process. Each CCn is stored
tions, we set ibase , which produces a relatively “thin” border in L if the intersection of CCn and R is not empty. Finally,
mask (line 12), and take a pixel-wise logical AND of the two we obtain the final mask M f inal from L.
masks, computed from iLoG and ibase (line 13). An example
of the effect of ibase is shown in Fig. 4(b). Algorithm 2 Combined masks.
1: Input: Mrm , Ml p
2: Output: M f inal
Algorithm 1 Create a removal mask. W
3: Mcomb ← Mrm Ml p
1: Input: I
4: R ← {x | Mrm (x) = 1}
2: Output: ibase , Mrm
5: L ← ∅
3: i ← 3, CCCmax ← 0, iLoG ← 1
6: N ← NCC(Mcomb )
4: while STCi > α do
7: for n ← 1...N do
5: if CCCi ≥ β ×CCCmax then
8: if CCn (Mcomb ) ∩ R 6= ∅ then
6: iLoG ← i
9: L ← L ∪CCn (Mcomb )
7: CCCmax ← max(CCCmax ,CCCi )
10: end if
8: end if
11: end for
9: i ← i+2 
1 if x ∈ L
10: end while 12: M f inal (x) ←
0 otherwise
11: iLoG ← iLoG + 4
12: ibase ← min( 2i , iLoG )
13: Mrm ← MLoGi ∧ MLoGi
LoG base 3. Experiment
To evaluate our method, we manually created ground truth
masks from 13 manga images. These were chosen from
six different titles so that they have different screentones.
2.2. Line preserving mask
The evaluation
V
value, overlap ratio, is defined as roverlap =
The line preserving mask is intended to extract line draw- |M f inal Mgt |
W ,
where Mgt represents the ground truth. This
ings. This is a difficult task, so this mask cannot avoid the |M f inal Mgt |
is a standard criterion for evaluation in an object detecting
capture of small amounts of screentones. Such unnecessary
task [EEG∗ 14]. We compared our method with a binariz-
screentones will be deleted by combining the result from
ing operation [Ots75], Canny edge, LoG filter, FDoG fil-
Mrm in the merging step.
ter [KLC07], and an edge-enhanced isotropic nonlinear fil-
To create a line preserving mask Ml p , we use a Flow- ter [HYLW10], which is a stroke detection method for clear-
based Difference of Gaussian (FDoG) filter [KLC07], which background cartoon images. The parameters of these meth-
was originally proposed for non-photorealistic rendering to ods were selected to maximize the average overlap ratio,

c The Eurographics Association 2015.

 Kota Ito & Yusuke Matsui & Toshihiko Yamasaki & Kiyoharu Aizawa / Separation of Manga Line Drawings and Screentones

(a) Original size (b) ×2 (c) ×4

Figure 6: An example of scale variation. Takuji

c

(a) Original (b) Removal mask (c) Output mask

Figure 7: An example of failure of merging. Tetsuya

c Kuro-
sawa, Hidehisa Masaki

Figure 5: Examples of comparative evaluation.

4. Conclusion
We have presented a method for extracting line drawings
from manga images. The algorithm is based on a combined
and fixed for all images. The results are shown in Table 3
LoG filter and FDoG filter. Experimental results show that
and Fig. 5. Our method outperforms all the other methods.
our proposed method outperforms conventional approaches
Please see supplementary material for more details.
for 13 manga images.

Table 1: Results of the comparative studies: accuracy. Acknowledgment

We thank authors of manga for the use of their images.
Binarization Canny LoG This work was partially supported by the Strategic Infor-
ro overlap 0.41 ± 0.15 0.23 ± 0.071 0.47 ± 0.068 mation and Communications R&D Promotion Programme
FDoG [HYLW10] Ours (SCOPE) and JSPS.
roverlap 0.51 ± 0.081 0.48 ± 0.080 0.56 ± 0.073
References
LoG-based methods (LoG, [HYLW10]) and FDoG have [AMYA14] A RAMAKI Y., M ATSUI Y., YAMASAKI T., A IZAWA
different advantages. LoG-based methods can remove non- K.: Interactive segmentation for manga. In ACM SIGGRAPH
2014 Posters (2014), ACM, p. 66. 1
periodic or sparse screentones with large window sizes, but
they cannot preserve lines in such cases. In contrast, the [CLC14] C AO Y., L AU R. W. H., C HAN A. B.: Look over here:
Attention-directing composition of manga elements. ACM TOG
FDoG filter cannot remove such screentones, but it preserves 33, 4 (2014), 1214–1220. 1
line drawings strongly. Our method can make the best use of
[EEG∗ 14] E VERINGHAM M., E SLAMI S., G OOL L. V.,
both advantages, and shows stable efficiency. In addition, be- W ILLIAMS C. K., W INN J., Z ISSERMAN A.: The pascal visual
cause our method determines all parameters automatically, object classes challenge: A retrospective. IJCV (2014), 1–39. 3
it is robust for both kinds of screentones and scale variation. [HYLW10] H UANG M., YANG M., L IU F., W U E.-H.: Stroke
An example of results with different scale images is shown extraction in cartoon images using edge-enhanced isotropic non-
in Fig. 6, where our method works well even when the scale linear filter. In Proc. VRCAI (2010), ACM, pp. 33–38. 2, 3, 4
of the image is changed. Our method is robust to the scale, [KL12] KOPF J., L ISCHINSKI D.: Digital reconstruction of
especially for periodic screentones, but it cannot handle non- halftoned color comics. ACM TOG 31, 6 (2012), 140. 2
periodic screentones well. Please see the supplementary ma- [KLC07] K ANG H., L EE S., C HUI C. K.: Coherent line drawing.
terial for more examples. Note our run-time cost was 6.4 sec In Proc. Non-photorealistic Animation and Rendering (2007),
pp. 43–50. 3
for 827×1170 images on average.
[MAJ14] M ATSUI Y., A IZAWA K., J ING Y.: Sketch2manga:
Limitations: If the screentone consists of dense long Sketch-based manga retrieval. In ICIP (2014), IEEE. 1
curves, the removal mask tends to fail to delete screentones [Ots75] OTSU N.: A threshold selection method from gray-level
(Fig. 7(b)). Because we consider connected components as a histograms. Automatica 11, 285-296 (1975), 23–27. 2, 3
unit in the merging stage, our method restores pixels that do [QWH06] Q U Y., W ONG T.-T., H ENG P.-A.: Manga coloriza-
not exist in the removal mask (Fig. 7(c)). tion. ACM TOG 25, 3 (2006), 1214–1220. 2

c The Eurographics Association 2015.

View publication stats