Hi, I’m Dave, and I’m here to talk about StrokeStrip:
an algorithm for joint parameterization and fitting of stroke clusters.
This is joint work with Chenxi Liu, Nicholas Vining, Mikhail Bessmeltsev, and Alla Sheffer.
Free-hand sketching is ubiquitously used by artists to communicate shapes and ideas.
When creating free-form drawings artists often use multiple overdrawn strokes that,
together, convey single intended curves.
Artists overdraw strokes to refine the shapes of their intended curves,
emphasize or thicken the curves, or as in this example to add interest and texture to the drawing.
We refer to groups of strokes that jointly define individual curves as clusters;
on this rough sketch, each cluster is visualized using a different colour.
Viewers can easily imagine the intended curves conveyed by each one of these clusters.
When capturing these strokes, artists increasingly use digital vector tools.
More and more sketches are now captured in vector form through tools like Wacom SDK,
Illustrator, or Blender’s grease pencil.
While multiple methods exist for grouping strokes that jointly depict intended curves,
there had been relatively little focus on the problem of fitting intended curves to such clusters.
We present StrokeStrip, a new method for fitting intended curves to vector stroke clusters.
It outperforms prior art and can robustly generate aggregate curves
consistent with human expectations on arbitrarily complex inputs.
Replacing clusters by intended curves not only produces aesthetically pleasing clean drawings,
but enables users to process their drawings using
downstream applications that operate on clean, overdrawing-free vector drawings
where each stroke is assumed to depict an intended, meaningful curve.
If we have an algorithm to extract a meaningful curve out of a cluster of strokes, then we can do things like:
apply gradient colours to whole stroke clusters based on their positions along the aggregate curve,
edit and deform whole clusters by editing just the single aggregate curve, or
combine the shape of one cluster with the stroke style of another to produce a new result.
Our work builds on prior work across a number of domains.
Prior methods take whole sketches as inputs and group strokes into clusters that depict individual aggregate curves.
Our method operates on clusters formed by any of these methods.
Starting from overdrawn raster inputs, Simo-Serra et al. use a neural network based solution
to generate overdrawing-free raster outputs.
These methods do not preserve artist intended drawing connectivity and
do not guarantee that when you have input that represents a single intended curve, you get a single curve as output.
Vectorizing both raw raster drawings or cleaned-up drawings produced by these methods results in outputs with non-curve connectivity.
Vector-space methods enable users to fit single curves to stroke clusters.
Generic methods for fitting curves to unordered points sampled from strokes fail dramatically on even medium complexity inputs.
Orbay and Kara, and Liu et al., propose methods specifically designed for fitting curves to stroke clusters.
Orbay and Kara order sample points using Laplacian spectral embedding, and then fit a curve to the ordered points.
Liu et al., apply a modified Moving Least Squares fitting to stroke sample points,
and then compute a directed spanning tree of the smoothed points and fit a curve to the longest path.
Unfortunately both methods fail to obtain correct point orderings
(shown here using a blue to red scale) resulting in undesirable fitting outcomes.
StrokeStrip robustly fits curves to clusters on which prior methods fail
by leveraging a number of insights about artist and viewer perception of overdrawn stroke clusters.
We observe that viewers perceive clusters of overdrawn strokes as strips along a path which follow the artist intended curve.
Human observers mentally delineate the outline of these strips, and can distinguish points
that are next to one another within the strip from ones that are nearby in Euclidean space but located in different sections of the strip.
A strip can be parameterized by assigning smoothly varying isovalues to all points on the strip.
We are specifically interested in a natural arclength 1D strip parameterization, which is defined by
arclength parameterizing the strip’s path and extending the parameterization to the strip
using path-orthogonal isolines, shown here in white.
The parameterization isolines are consequently straight and are on average evenly spaced.
They cross the strips from one side to the other forming cross-sections.
The restriction of this parameterization to the cluster strokes provides a natural mapping
from the strokes to the a priori unknown strip path, or intended curve.
In this mapping each isoline corresponds to a point on this curve.
Given this joint parameterization of the strokes on the strip, we can directly compute
a parametric fitted curve whose shape at each parameter value reflects the average shape of the input strokes –
their position, tangent, and curvature – at that parameter value.
This means we can recast the fitting problem as a parameterization problem:
we first compute a joint arc-length parameterization of the strokes in the cluster,
and then fit the desired aggregate curve to the cluster using this parameterization.
We seek a parameterization that has similar parameter values at points that are adjacent within the strip.
However, since we are only given input strokes, we don’t actually know the shape of the strip,
and thus do not know which points on different strokes
that are adjacent in Euclidean space are also adjacent within the strip.
Looking only at local cues, it’s hard to tell if two close-by points are close within the strip or not.
So take the points in green here: they are close together within the strip
but although the red points are the same distance apart in 2D, they lie on far apart locations on the strip.
So, in order to parameterize a cluster of strokes, we need to solve two problems at once:
We need to compute optimal parameter values at all points on the strokes,
which requires assigning similar values to within-the-strip adjacent points.
To do this, we need to determine which points on the strokes are within-the-strip adjacent.
Our method addresses both challenges simultaneously using a variational approach.
Before describing our method, lets consider the properties of the natural arclength cluster parameterizations we seek.
First, we look for strips whose path is aligned with the direction of the input strokes.
This property is satisfied when the gradient of the parameterization is aligned with the tangents of the cluster strokes.
Second, we want the parameterization to be arc length preserving,
facilitating even sampling of stroke points during subsequent fitting.
Third, we expect the parameterization to be strictly monotonic along each stroke,
as we expect the fitted curves to follow the shape of the strokes.
And finally, we want parameter values at points on different strokes that viewers perceive as adjacent within the strip to be similar.
We refer to this property as ”isoline span.”
Failing to satisfy this requirement results in unexpected fitted curve geometry.
We simultaneously compute the joint cluster parameterization and implicitly reconstruct the unknown strip geometry
by casting the computation of the desired parameterization as a minimization of a constrained variational problem.
We start by considering the arc length parameterization of a hypothetical strip:
that is, the function u that minimizes the following double integral,
defined in terms of the arc length of the strip and the isoline cross-sections for each parameter value t.
In a natural arclength parameterization, these isolines are expected to be straight and perpendicular to the strip’s path.
Unfortunately, applying this formulation to strokes as-is is problematic since the gradient of u is not clearly defined.
We know that the gradient of an arc length parameterization is always orthogonal to the path tangent,
and its average over each gradient orthogonal cross-section has unit length.
Consequently, we can rewrite this parameterization in terms of the projection
of the expression inside the norm onto the centerline’s tangent and normal.
We now seek the function u whose domain is the strip restricted to the set of strokes in our strip,
whose range is the arc length, and that minimizes the functionals E_length and E_align
while also satisfying a monotonicity constraint.
Our functionals satisfy two goals:
First, we want the parameterization to be arc length.
Second, we want the tangents of the aggregate curve – and consequently the gradient of the parameterization that defines it –
to be aligned with the stroke tangents.
In other words, parameterization isolines should be approximately orthogonal to the strokes.
Finally, since the arc length parameterization of a strip is strictly monotonic,
its restriction to each stroke must be constrained to be strictly monotonic as well.
Each of these functionals also has a clear geometric interpretation.
For our arc length preserving term, we integrate along the length of the strip,
and across each cross-section; we expect that at each point,
u should change at a speed of 1 in the direction of the aggregate curve.
For our tangent alignment term, we again integrate along the length of the strip
and across each cross-section, and at each point,
we want the gradient of u to be orthogonal to the normal of the aggregate curve.
If they are orthogonal everywhere, then the isolines of the parameterization will be orthogonal to the curve,
and the tangents will be aligned.
However, C(t) is still unknown.
Since the strip geometry is unknown, we cannot integrate across cross-sections,
since we have no strip geometry to define a cross-section on.
This is the core chicken and egg problem we face.
It is further complicated by our monotonicity constraint for each stroke.
Finding the minimizer of our functional requires solving for three sets of variables.
First, the parameter values at all stroke points.
Second, the gradient orientations of each stroke, and finally,
a set of isoline cross-sections that are defined for a dense set of parameter values for u,
and the set of stroke points that lie on each cross-section.
This creates a mixed discrete-continuous optimization problem,
which cannot be addressed using standard off-the-shelf optimization techniques.
We address the parameterization problem using a two step process.
We observe that we can reliably predict the orientations O_s of the parameterization gradients along each stroke,
that jointly allow for optimal cluster parameterization.
We therefore first compute optimal gradient orientations across all strokes in the cluster.
We then simultaneously compute both the parameterization cross-sections and the parameter values along them using a variational framework.
We use this parameterization to define parametric fitted curves whose shape reflects
the geometric properties of the cluster strokes at the corresponding isovalues.
Lets now look at each of these step in more detail.
To best satisfy our tangent alignment and isoline span properties,
we aim to maximize gradient direction similarity for adjacent roughly parallel strokes.
Specifically, we expect most pairs of adjacent and roughly parallel strokes to be next to one another within the strip,
and thus to have similar gradient orientations.
Given pairs that allow for multiple strip interpretations, we prefer orientations consistent with more narrow strips.
Lastly, we avoid orientation choices which introduce isolines with diverging gradient directions,
prioritizing tangent alignment over isoline span. prioritizing tangent alignment over isoline span.
We use these cues to compute Boolean pairwise orientation compatibilities for each pair of adjacent and roughly parallel strokes.
We then assess the impact that satisfying each stroke pair’s most compatible orientation has
on the global solution based on the amount of perceived adjacency,
degree of parallelism at adjacent points, and the inter-stroke distances.
Finally, we compute global stroke orientations that maximize overall compatibility between stroke pairs, weighted by their impact.
We formulate and solve this optimization problem using integer programming.
Lets now look at the details of our core parameterization step.
We discretize the continuous strip parameterization formulation and restrict it to the strokes, arriving at the following energy functional.
For details please see the paper.
The core challenge in optimizing this functional is that we do not know which points on different strokes
that are adjacent in Euclidean space are perceived as adjacent within the strip.
Specifically, we do not know which points belong to the same cross-section.
Nor do we know the parameterization gradient along these cross-sections.
Purely local analysis of point neighborhoods is insufficient to distinguish
between points which are adjacent within the strip, and ones that are adjacent only in Euclidean space.
While human observers can easily envision the aggregate strips and curves corresponding to each cluster,
and hence the isoline cross-sections orthogonal to these curves,
our algorithm needs to compute these cross-sections given only the raw strokes as input.
At the same time in many cases, adjacency in Euclidean space DOES translate into WTS adjacency,
and tangent orthogonality suggests that stroke tangents are a good proxy for parameterization gradients.
Thus given a ray from a point on one stroke, that is orthogonal to the stroke tangent at that point,
the isovalues at the intersections of this ray with other strokes are likely to be similar to the isovalue at its origin.
We refer to such rays as stroke-orthogonal cross-sections.
While each stroke-orthogonal cross-sections may contain stroke points that are not within-the-strip adjacent;
most points on the same stroke-orthogonal cross-sections are likely to be within-the-strip adjacent.
The natural question to ask is: “How likely?”
To answer this question, we introduce within-strip adjacency likelihood as an additional variable into our formulation.
We then solve for both parameter values and within-strip adjacencies simultaneously.
Since stroke orthogonal cross-sections only approximate our target cross-sections we break the computation into several steps.
We first identify a set of side-by-side points that are reasonably likely to be within-strip adjacent
by generating a set of stroke-orthogonal cross-sections.
We use these cross-sections to compute an approximate parameterization,
and determine the likelihood that pairs of points on current cross-sections are, indeed, within-the-strip adjacent.
We use an iterative computation process that alternates between parameter value and likelihood update steps.
We use the relaxed parameterization as an initial close guess for computing the final arclength cluster parameterization.
We aim to maximize the span of each cross-section while also
keeping them contained as much as possible within the perceived strip.
We therefore start shooting rays orthogonally from sample points along the input strokes.
We then filter out any points that are unlikely to be within-the-strip adjacent using local and contextual filters.
These include tangent similarity
and outlier inter-stroke distances.
Here we see the final result.
We use these stroke-orthogonal cross-sections to solve for parameter values at all cross-section points,
and to compute the likelihoods that pairs of points on the same cross-sections are within-strip-adjacent.
We use a formulation whose first term maximizes arclength preservation along each stroke,
and whose second term minimizes parameter value differences between points on the same cross-section.
The entries in this second term are multiplied by a likelihood estimator
that reflects how likely the points are to be within-strip adjacent.
We set this likelihood term to be a function of the similarity of the parameter values at the two points.
The more similar the values are the more likely the points are to be within-the-strip adjacent.
Starting with a set of fixed small likelihood values,
we alternate between computing a parameterization based on those likelihoods
and then updating the likelihoods based on our new parameterization.
Here we see how after each update the parameterization isolines straighten
and get closer to the cross-sections we expect from our final parameterization.
The minimizer of our relaxed formulation closely approximates the optimal solutions we seek,
so we use it to form new isoline cross-sections by directly utilizing the isolines of our relaxed parameterization.
We use these new cross-sections to optimize for the desired cluster arc-length parameterization,
again using an iterated solver.
We solve for parameter values along the cross-sections,
then form cross-sections yet again from the resulting parameterization, and repeat this process until we converge.
Finally, with a parameterization, we can fit a parametric curve!
We compute a curve whose shape at each parameter value u reflect the shape of the input strokes at this u value.
Our fitting accounts for three input stroke properties: positions, tangents, and curvatures.
We prioritize curvature and tangent preservation over position approximation
in accordance with prior work, and our own observations of human preferences.
So let's take a look at the results!
We show the output of our fitting method on 80 vector format drawings,
containing a total of one thousand, eight hundred and thirty two clusters,
in the paper and supplementary material, as well as 31 additional single-cluster inputs.
These examples include self-intersecting clusters, narrow U-shapes, spirals,
clusters with sudden curvature changes, and clusters with large gaps and tangent differences between participating strokes.
In all cases, we successfully compute arc length parameterizations
that satisfy our core criteria and fit curves that are well aligned with viewer expectations.
In this alphabet example, we see that StrokeStrip can correctly parameterize clusters
with significant overdraw, and single clusters with multiple self-intersections.
This allows us to produce high quality fitted curves even on complex inputs like the cursive letters in this alphabet.
Prior methods fail on many of the letters that we succeed on.
Here is another example.
While StrokeStrip correctly fits all clusters in this input, prior methods dramatically fail on the two highlighted clusters.
In general, StrokeStrip outputs are significantly better aligned
with human observer preferences than those produced by prior methods.
Being robust to curvature and self-intersection allows us to handle complex inputs
such as this geometric design, where other methods fail.
To evaluate our method, we ran a comparative perceptual study.
We asked viewers to look at input drawing and fitting results generated by our method
and one alternative method, either Orbay and Kara 2011 or Liu et al. 2018.
Users were asked which output drawing more accurately depicted the rough drawing.
We asked 80 questions comparing StrokeStrip to Orbay and Kara 2011, and 76 against Liu et al. 2018,
using a total of 36 participants, each of whom answered at most 32 questions.
In comparisons against the closest competitor, participants preferred our method 60% of the time,
judged the results as on par 34% of the time, and preferred the alternative only 5% of the time.
We also tested our method on raster inputs.
After running raster clusters through a commercial vectorization tool, we produce fitted curves that match expectations,
with only minimal changes to the algorithm to account for vectorization artifacts.
Having a parameterization enables us to edit sketches more easily.
Here, we color whole stroke clusters by interpolating colours along each strip,
using our joint parameterization.
Note the gradients throughout.
Our joint parameterization also enables users to edit all strokes in a cluster at once using cluster level controls.
To summarize, StrokeStrip is a robust method for fitting and parameterizing stroke clusters that dramatically outperforms prior art.
We achieve this goal by casting stroke cluster parameterization as the computation
of the restriction to the strokes of the natural arclength parameterization of the intended strips these clusters depict.
Thank you so much for your time!