research-article

Calligraphic Stylisation Learning with a Physiologically Plausible Model of Movement and Recurrent Neural Networks

Authors:

Frederic Fol Leymarie,

Réjean PlamondonAuthors Info & Claims

MOCO '17: Proceedings of the 4th International Conference on Movement Computing

Article No.: 25, Pages 1 - 8

https://doi.org/10.1145/3077981.3078049

Published: 28 June 2017 Publication History

Abstract

We propose a computational framework to learn stylisation patterns from example drawings or writings, and then generate new trajectories that possess similar stylistic qualities. We particularly focus on the generation and stylisation of trajectories that are similar to the ones that can be seen in calligraphy and graffiti art. Our system is able to extract and learn dynamic and visual qualities from a small number of user defined examples which can be recorded with a digitiser device, such as a tablet, mouse or motion capture sensors. Our system is then able to transform new user drawn traces to be kinematically and stylistically similar to the training examples. We implement the system using a Recurrent Mixture Density Network (RMDN) combined with a representation given by the parameters of the Sigma Lognormal model, a physiologically plausible model of movement that has been shown to closely reproduce the velocity and trace of human handwriting gestures.

References

[1]

J. Bergstra and Y. Bengio. 2012. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research 13 (2012), 281--305.

Digital Library

[2]

D. Berio, S. Calinon, and F. Fol Leymarie. 2016. Learning dynamic graffiti strokes with a compliant robot. In Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS). Daejeon, Korea, 3981--6.

[3]

D Berio, S. Calinon, and F. Fol Leymarie. 2017. Dynamic Graffiti Stylisation with Stochastic Optimal Control. In ACM Proceedings of the 4th International Conference on Movement and Computing. London, UK.

Digital Library

[4]

D Berio, S. Calinon, and F. Fol Leymarie. 2017. Generating Calligraphic Trajectories with Model Predictive Control. In Proceedings of Graphics Interface. Canadian Human-Computer Communications Society, Edmonton, Canada.

[5]

D. Berio and F. Fol Leymarie. 2015. Computational Models for the Analysis and Synthesis of Graffiti Tag Strokes. In Proceedings of the Workshop on Computational Aesthetics. Eurographics Association, 35--47.

Digital Library

[6]

H. Bezine, A.M. Alimi, and N. Sherkat. 2004. Generation and analysis of handwriting script with the Beta-elliptic model. Proc. of International Workshop on Frontiers in Handwriting Recognition (IWFHR) 8, 2 (2004), 515--20.

Digital Library

[7]

CM. Bishop. 1994. Mixture density networks. (1994). Technical Report NCRG/4288, Neural Computing Research Group, Aston University, U.K.

[8]

L. Crnkovic-Friis and L. Crnkovic-Friis. 2016. Generative Choreography using Deep Learning. arXiv preprint arXiv:1605.06921 (2016).

[9]

Kingma D. and Ba J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014), 1--13.

[10]

J. De Winter and J. Wagemans. 2008. Perceptual saliency of points along the contour of everyday objects: A large-scale study. Perception & Psychophysics 70, 1 (2008), 50--64.

[11]

A. Dempster, N. Laird, and D. Rubin. 1977. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. Royal Statistical Society B 39, 1 (1977), 1--38.

[12]

J. Feldman and M. Singh. 2005. Information along contours and object boundaries. Psychological review 112, 1 (2005), 243.

[13]

A. Fischer, R. Plamondon, C. O'Reilly, and Y. Savaria. 2014. Neuromuscular Representation and Synthetic Generation of Handwritten Whiteboard Notes. In Proc. of Frontiers in Handwriting Recognition (ICFHR). 222--7.

[14]

T. Flash and A.A. Handzel. 2007. Affine differential geometry analysis of human arm movements. Biological cybernetics 96, 6 (2007), 577--601.

Digital Library

[15]

T. Flash and B. Hochner. 2005. Motor primitives in vertebrates and invertebrates. Current opinion in neurobiology 15, 6 (2005), 660--6.

[16]

FA. Gers, J. Schmidhuber, and F. Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural Computation 12, 10 (2000), 2451--2471.

Digital Library

[17]

A. Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013).

[18]

K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. 2015. DRAW: A Recurrent Neural Network For Image Generation. arXiv preprint arXiv:1502.04623 (2015).

[19]

D. Ha. 2015. Generative Handwriting Demo using TensorFlow. (2015). https://github.com/hardmaru/write-rnn-tensorflow

[20]

D. Ha, A. Dai, and Q.V. Le. 2016. HyperNetworks. arXiv preprint arXiv:1609.09106 (2016).

[21]

A. Hertzmann, N. Oliver, B. Curless, and S.M. Seitz. 2002. Curve Analogies. In Proc. 13th EuroGraphics Workshop on Rendering (EGRW). Pisa, Italy, 233--46.

Digital Library

[22]

S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735--80.

Digital Library

[23]

E. Indermühle, M. Liwicki, and H. Bunke. 2010. IAMonDo-database: An Online Handwritten Document Database with Non-uniform Contents. In Proc. 9th IAPR Int'l Workshop on Document Analysis Systems (DAS). 97--104.

Digital Library

[24]

H.S.R. Kao, R. Hoosain, and G.P. Van Galen. 1986. Graphonomics: Contemporary research in handwriting. Elsevier.

[25]

Graffiti Research Lab. 2009. Graffiti Analysis Database (000000book.com). Web. (2009). http://000000book.com

[26]

K. Lang and M. Alexa. 2015. The Markov pen: Online synthesis of free-hand drawing styles. In Proc. of the workshop on Non-Photorealistic Animation and Rendering (NPAR). Eurographics Association, 203--15.

Digital Library

[27]

H.W. Lin and M. Tegmark. 2016. Why does deep and cheap learning work so well? arXiv preprint arXiv:1608.08225 (2016).

[28]

M. Ltaief, H. Bezine, and A.M. Alimi. 2012. A neuro-Beta-elliptic model for handwriting generation movements. In Proc. of International Conference on Frontiers in Handwriting Recognition (ICFHR). 803--8.

Digital Library

[29]

H. Nagasaki. 1989. Asymmetric velocity and acceleration profiles of human arm movements. Experimental Brain Research 74, 2 (1989), 319--26.

[30]

V. Nair and G. Hinton. 2005. Inferring motor programs from images of handwritten digits. In Advances in neural information processing systems. 515--22.

Digital Library

[31]

C. O'Reilly and R. Plamondon. 2008. Automatic extraction of Sigma-Lognormal parameters on signatures. In Proc. of 11th International Conference on Frontier in Handwriting Recognition (ICFHR).

[32]

R. Pascanu, T. Mikolov, and Y. Bengio. 2013. On the difficulty of training Recurrent Neural Networks. In Proc. of ICML, Vol. 28. 1310--8.

Digital Library

[33]

V. Pham, T. Bluche, C. Kermorvant, and J. Louradour. 2014. Dropout Improves Recurrent Neural Networks for Handwriting Recognition. In Proc. of ICFHR. IEEE, 285--90.

[34]

A. Pignocchi. 2010. How the Intentions of the Draftsman Shape Perception of a Drawing. Consciousness and Cognition 19, 4 (2010), 887--98.

[35]

R. Plamondon. 1995. A Kinematic Theory of Rapid Human Movements. Part I. Movement Representation and Generation. Biological cybernetics 72, 4 (1995), 295--307.

Digital Library

[36]

R. Plamondon and others. 1993. Modelling velocity profiles of rapid movements: A comparative study. Biological cybernetics 69, 2 (1993), 119--28.

Digital Library

[37]

R. Plamondon and others. 2014. Recent developments in the study of rapid human movements with the kinematic theory. Pattern Recognition Letters 35 (2014), 225--35.

Digital Library

[38]

R. Plamondon and C. Privitera. 1996. A neural model for generating and learning a rapid movement sequence. Biological cybernetics 74, 2 (1996), 117--30.

Digital Library

[39]

B. Rohrer and N. Hogan. 2006. Avoiding spurious submovement decompositions II. Biological cybernetics 94, 5 (2006), 409--14.

Digital Library

[40]

D.A. Rosenbaum and others. 1995. Planning reaches by evaluating stored postures. Psychological Review 102, 1 (1995), 28--67.

[41]

M. Schuster. 1999. Better Generative Models for Sequential Data Problems: Bidirectional Recurrent Mixture Density Networks. In Proc. of NIPS. 589--95.

Digital Library

[42]

I. Sutskever. 2013. Training Recurrent Neural Networks. Ph.D. Dissertation. University of Toronto.

Digital Library

[43]

M. Umilta and others. 2012. Abstract Art and Cortical Motor Activation: An EEG Study. Frontiers in Human Neuroscience 6, 311 (2012), 1--9.

[44]

T. Varga, D. Kilchhofer, and H. Bunke. 2005. Template-based Synthetic Handwriting Generation for the Training of Recognition Systems. In Proc. of 12th Conf. of the International Graphonomics Society (IGS). 206--11.

[45]

X.-Y. Zhang and others. 2016. Drawing and Recognizing Chinese Characters with Recurrent Neural Network. arXiv preprint arXiv:1606.06539 (2016).

Cited By

Plamondon RBensalah ALebel KSalameh RSéguin de Broin GO’Reilly CBegon MDesbiens OBeloufa YGuy ABerio DLeymarie FBoyoguéno-Bidias SFischer AZhang ZMorin MAlamargot DRémi CFaci NFortin RSimard MBazinet C(2023)Lognormality: An Open Window on Neuromotor ControlGraphonomics in Human Body Movement. Bridging Research and Practice from Motor Control to Handwriting Analysis and Recognition10.1007/978-3-031-45461-5_15(205-258)Online publication date: 9-Oct-2023
https://doi.org/10.1007/978-3-031-45461-5_15
Hutson JLively JRobertson BCotroneo PLang MHutson JLively JRobertson BCotroneo PLang M(2023)Painting by Numbers: A Brief History of Art and TechnologyCreative Convergence10.1007/978-3-031-45127-0_3(37-85)Online publication date: 15-Nov-2023
https://doi.org/10.1007/978-3-031-45127-0_3
Tian NTanwani AGoldberg KSojoudi S(2022)Mitigating Network Latency in Cloud-Based Teleoperation Using Motion Segmentation and SynthesisRobotics Research10.1007/978-3-030-95459-8_56(906-921)Online publication date: 17-Feb-2022
https://doi.org/10.1007/978-3-030-95459-8_56
Show More Cited By

Index Terms

Calligraphic Stylisation Learning with a Physiologically Plausible Model of Movement and Recurrent Neural Networks
1. Computing methodologies
  1. Computer graphics
    1. Animation
      1. Procedural animation
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input

Recommendations

FRMDN: Flow-based Recurrent Mixture Density Network
Abstract
The class of recurrent mixture density networks is an important class of probabilistic models used extensively in sequence modeling and sequence-to-sequence mapping applications. In this class of models, the density of a target sequence in each ...
Highlights
- Two aspects of RMDNs have been explored for efficient density estimation.
- A normalizing flow is employed to increase the flexibility of RMDNs.
- A parameter-sharing approach for GMM is applied that decomposes the precision matrix.
Sales Demand Forecast based on Recurrent Neural Network∗
EITCE '22: Proceedings of the 2022 6th International Conference on Electronic Information Technology and Computer Engineering

In this paper, we compare the performance of RNN, LSTM and GRU, which are the most popular cyclic neural networks, in predicting the total sales of products in each store next month. A dropout layer is added to the model to reduce over-fitting. The ...
A recurrent neural network based deep learning model for text and non-text stroke classification in online handwritten Devanagari document
Abstract
It is very common for human beings to include both text and non-text data in a handwritten document. Text portion contains alphabets, digits, and mathematical symbols, whereas non-text portion includes various graphical entities like flow chart, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MOCO '17: Proceedings of the 4th International Conference on Movement Computing

June 2017

206 pages

ISBN:9781450352093

DOI:10.1145/3077981

Conference Chair:
Marco Gillies,
Editor:
Kiona Niehaus

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

University of Surrey

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MOCO '17

MOCO '17: 4th International Conference on Movement Computing

June 28 - 30, 2017

London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 85 of 185 submissions, 46%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
186
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Plamondon RBensalah ALebel KSalameh RSéguin de Broin GO’Reilly CBegon MDesbiens OBeloufa YGuy ABerio DLeymarie FBoyoguéno-Bidias SFischer AZhang ZMorin MAlamargot DRémi CFaci NFortin RSimard MBazinet C(2023)Lognormality: An Open Window on Neuromotor ControlGraphonomics in Human Body Movement. Bridging Research and Practice from Motor Control to Handwriting Analysis and Recognition10.1007/978-3-031-45461-5_15(205-258)Online publication date: 9-Oct-2023
https://doi.org/10.1007/978-3-031-45461-5_15
Hutson JLively JRobertson BCotroneo PLang MHutson JLively JRobertson BCotroneo PLang M(2023)Painting by Numbers: A Brief History of Art and TechnologyCreative Convergence10.1007/978-3-031-45127-0_3(37-85)Online publication date: 15-Nov-2023
https://doi.org/10.1007/978-3-031-45127-0_3
Tian NTanwani AGoldberg KSojoudi S(2022)Mitigating Network Latency in Cloud-Based Teleoperation Using Motion Segmentation and SynthesisRobotics Research10.1007/978-3-030-95459-8_56(906-921)Online publication date: 17-Feb-2022
https://doi.org/10.1007/978-3-030-95459-8_56
Godøy R(2021)Constraint-Based Sound-Motion Objects in Music PerformanceFrontiers in Psychology10.3389/fpsyg.2021.73272912Online publication date: 21-Dec-2021
https://doi.org/10.3389/fpsyg.2021.732729
Lyu RZhang TYuan Z(2021)Imaginary Stroke Movement Measurement and VisualizationProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/34656254:2(1-12)Online publication date: 2-Aug-2021
https://dl.acm.org/doi/10.1145/3465625
Godøy R(2021)Understanding Musical InstantsThe Oxford Handbook of Time in Music10.1093/oxfordhb/9780190947279.013.9(197-C11.P135)Online publication date: 8-Dec-2021
https://doi.org/10.1093/oxfordhb/9780190947279.013.9
Wu RZhou CChao FYang LLin CShang C(2020)GANCCRobotInformation Sciences: an International Journal10.1016/j.ins.2019.12.079516:C(474-490)Online publication date: 1-Apr-2020
https://dl.acm.org/doi/10.1016/j.ins.2019.12.079
Ahmadian AFouladi KAraabi B(2020)Model-based Persian calligraphy synthesis via learning to transfer templates to personal stylesInternational Journal on Document Analysis and Recognition (IJDAR)10.1007/s10032-020-00353-1Online publication date: 18-Jun-2020
https://doi.org/10.1007/s10032-020-00353-1
Kotani ATellex STompkin J(2020)Generating Handwriting via Decoupled Style DescriptorsComputer Vision – ECCV 202010.1007/978-3-030-58610-2_45(764-780)Online publication date: 7-Oct-2020
https://doi.org/10.1007/978-3-030-58610-2_45
Schacher JWei LColeman G(2019)Gesture-Ink-SoundProceedings of the 6th International Conference on Movement and Computing10.1145/3347122.3347136(1-8)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1145/3347122.3347136
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten