Formatting Open Science: agilely creating multiple document formats for academic manuscripts with Pandoc Scholar

Preprints and e-prints

In some areas of science a preprint culture, i.e., a paper-based exchange system of research ideas and results, already existed when Paul Ginsparg in 1991 initiated a server for the distribution of electronic preprints—‘e-prints’—about high-energy particle theory at the Los Alamos National Laboratory (LANL), USA (Ginsparg, 1994). Later, the LANL server moved with Ginsparg to Cornell University, USA, and was renamed as arXiv (Butler, 2001). Currently, arXiv (https://arxiv.org/) publishes e-prints related to physics, mathematics, computer science, quantitative biology, quantitative finance and statistics. Just a few years after the start of the first preprint servers, their important contribution to scientific communication was evident (Ginsparg, 1994; Youngen, 1998; Brown, 2001). In 2014, arXiv reached the impressive number of 1 million e-prints (Van Noorden, 2014).

In more conservative areas, such as chemistry and biology, accepting the publishing prior peer-review took more time (Brown, 2003). A preprint server for life sciences (http://biorxiv.org/) was launched by the Cold Spring Habor Laboratory, USA, in 2013 (Callaway, 2013). PeerJ preprints (https://peerj.com/preprints/), started in the same year, accepts manuscripts from biological sciences, medical sciences, health sciences and computer sciences.

The terms ‘preprints’ and ‘e-prints’ are used synonymously, since the physical distribution of preprints has become obsolete. A major drawback of preprint publishing are the sometimes restrictive policies of scientific publishers. The SHERPA/RoMEO project informs about copyright policies and self-archiving options of individual publishers (http://www.sherpa.ac.uk/romeo/).

Open Access

The term ‘Open Access’ (OA) was introduced 2002 by the Budapest Open Access Initiative and was defined as:

“Barrier-free access to online works and other resources. OA literature is digital, online, free of charge (gratis OA), and free of needless copyright and licensing restrictions (libre OA).” (Suber, 2012)

Frustrated by the difficulty to access even digitized scientific literature, three scientists founded the Public Library of Science (PLoS). In 2003, PLoS Biology was published as the first fully Open Access journal for biology (Brown, Eisen & Varmus, 2003; Eisen, 2003).

Thanks to the great success of OA publishing, many conventional print publishers now offer a so-called ‘Open Access option’, i.e., to make accepted articles free to read for an additional payment by the authors. The copyright in these hybrid models might remain with the publisher, whilst fully OA usually provide a liberal license, such as the Creative Commons Attribution 4.0 International (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/).

OA literature is only one component of a more general open philosophy, which also includes the access to scholarships, software, and data (Willinsky, 2005). Interestingly, there are several different ‘schools of thought’ on how to understand and define Open Science, as well the position that any science is open by definition, because of its objective to make generated knowledge public (Fecher & Friesike, 2014).

Cost of journal article production

In a recent study, the article processing charges (APCs) for research intensive universities in the USA and Canada were estimated to be about 1,800 USD for fully OA journals and 3,000 USD for hybrid OA journals (Solomon & Björk, 2016). PeerJ (https://peerj.com/), an OA journal for biological and computer sciences launched in 2013, drastically reduced the publishing cost, offering its members a life-time publishing plan for a small registration fee (Van Noorden, 2012); alternatively the authors can choose to pay an APC of $1,095 USD, which may be cheaper, if multiple co-authors participate.

Examples such as the Journal of Statistical Software (JSS, https://www.jstatsoft.org/) and eLife (https://elifesciences.org/) demonstrate the possibility of completely community-supported OA publications. Figure 1 compares the APCs of different OA publishing business models.

JSS and eLife are peer-reviewed and indexed by Thomson Reuters. Both journals are located in the Q1 quality quartile in all their registered subject categories of the Scimago Journal & Country Rank (http://www.scimagojr.com/), demonstrating that high-quality publications can be produced without charging the scientific authors or readers.

In 2009, a study was carried out concerning the “Economic Implications of Alternative Scholarly Publishing Models”, which demonstrates an overall societal benefit by using OA publishing models (Houghton et al., 2009). In the same report, the real publication costs are evaluated. The relative costs of an article for the publisher are presented in Fig. 2.

Conventional publishers justify their high subscription or APC prices with the added value; for example, journalism (stated in the graphics as ‘non-article processing’). However, stakeholder profits, which could be as high as 50%, also must be considered, and are withdrawn from the science budget (Van Noorden, 2013).

Generally, the production costs of an article could be roughly divided into commercial and academic/technical costs (Fig. 2). For nonmarket production, the commercial costs such as margins/profits, management etc. can be drastically reduced. Hardware and services for hosting an editorial system, such as Open Journal Systems of the Public Knowledge Project (https://pkp.sfu.ca/ojs/) can be provided by public institutions. Employed scholars can perform editor and reviewer activities without additional cost for the journals. Nevertheless, ‘article processing’, which includes the manuscript handling during peer review and production represents the most expensive part.

Therefore, we investigated a strategy for the efficient formatting of scientific manuscripts.

Current standard publishing formats

Generally speaking, a scientific manuscript is composed of contents and formatting. While the content, i.e., text, figures, tables, citations etc., may remain the same between different publishing forms and journal styles, the formatting can be very different. Most publishers require the formatting of submitted manuscripts in a certain format. Ignoring this Guide for Authors, (for example, by submitting a manuscript with a different reference style), gives a negative impression with a journal’s editorial staff. Manuscripts which are too carelessly prepared can even provoke a straight ‘desk-reject’ (Volmer & Stokes, 2016).

Currently DOC(X), LATEX and/ or PDF file formats are the most frequently used formats for journal submission platforms. However, even if the content of a submitted manuscript might be accepted during the peer review ‘as is’, the format still needs to be adjusted to the particular publication style in the production stage. For the electronic distribution and archiving of scientific works, which is gaining more and more importance, additional formats (EPUB, (X)HTML, JATS) need to be generated. Table 1 lists the file formats which are currently the most relevant ones for scientific publishing.

Although the content elements of documents, such as title, author, abstract, text, figures, tables, etc., remain the same, the syntax of the file formats is rather different. Table 2 demonstrates some simple examples of differences in different markup languages.

Documents with the commonly used Office Open XML (DOCX Microsoft Word files) and OpenDocument (ODT LibreOffice) file formats can be opened in a standard text editor after unzipping. However, content and formatting information is distributed into various folders and files. Practically speaking, those file formats require the use of special word processing software.

From a writer’s perspective, the use of What You See Is What You Get (WYSIWYG) programs such as Microsoft Word, WPS Office or LibreOffice might be convenient, because the formatting of the document is directly visible. But the complicated syntax specifications often result in problems when using different software versions and for collaborative writing. Simple conversions between file formats can be difficult or impossible. In a worst-case scenario, ‘old’ files cannot be opened any more for lack of compatible software.

In some parts of the scientific community therefore LATEX, a typesetting program in plain text format, is very popular. With LATEX, documents with highest typographic quality can be produced. However, the source files are cluttered with LATEX commands and the source text can be complicated to read. Causes of compilation errors in LATEX are sometimes difficult to find. Therefore, LATEX is not very user friendly, especially for casual writers or beginners.

In academic publishing, it is additionally desirable to create different output formats from the same source text:

• For the publishing of a book, with a print version in PDF and an electronic version in EPUB.

• For the distribution of a seminar script, with an online version in HTML and a print version in PDF.

• For submitting a journal manuscript for peer-review in DOCX, as well as a preprint version with another journal style in PDF.

• For archiving and exchanging article data using the Journal Article Tag Suite (JATS) (National Information Standards Organization, 2012), a standardized format developed by the National Library of Medicine (NLM).

Some of the tasks can be performed with LATEX, but an integrated solution remains a challenge. Several programs for the conversion between documents formats exist, such as the e-book library program calibre (http://calibre-ebook.com/). But the results of such conversions are often not satisfactory and require substantial manual corrections.

Therefore, we were looking for a solution that enables the creation of scientific manuscripts in a simple format, with the subsequent generation of multiple output formats. The need for hybrid publishing has been recognized outside of science (Kielhorn, 2011), but the requirements specific to scientific publishing have not been addressed so far. Therefore, we investigated the possibility to generate multiple publication formats from a simple manuscript source file.

Markdown editors

Due to MD’s simple syntax, basically any text editor is suitable for editing markdown files. The formatting tags are written in plain text and are easy to remember. Therefore, the author is not distracted by looking around for layout options with the mouse. For several popular text editors, such as vim (http://www.vim.org/), GNU Emacs (https://www.gnu.org/software/emacs/), atom (https://atom.io/) or geany (http://www.geany.org/), plugins provide additional functionality for markdown editing; for example, syntax highlighting, command helpers, live preview or structure browsing.

Various dedicated markdown editors have been published as well. Many of those are cross-platform compatible, such as Abricotine (http://abricotine.brrd.fr/), ghostwriter (https://github.com/wereturtle/ghostwriter) and CuteMarkEd (https://cloose.github.io/CuteMarkEd/).

The lightweight format is also ideal for writing on mobile devices. Numerous applications are available on the App stores for Android and iOS systems. The programs Swype and Dragon (http://www.nuance.com/) facilitate the input of text on such devices by guessing words from gestures and speech recognition (dictation).

Figure 5 shows the editing of a markdown file, using the cross-platform editor Atom with several markdown plugins.

Online editing and collaborative writing

Storing manuscripts on network drives (The Cloud) has become popular for several reasons:

• Protection against data loss.

• Synchronization of documents between several devices.

• Collaborative editing options.

Markdown files on a Google Drive (https://drive.google.com) for instance can be edited online with StackEdit (https://stackedit.io). Figure 6 demonstrates the online editing of a markdown file on an ownCloud (https://owncloud.com/) installation. OwnCloud is an Open Source software platform, which allows the set-up of a file server on personal webspace. The functionality of an ownCloud installation can be enhanced by installing plugins.

Even mathematical formulas are rendered correctly in the HTML live preview window of the ownCloud markdown plugin (Fig. 6).

The collaboration and authoring platform Authorea (https://www.authorea.com/) also supports markdown as one of multiple possible input formats. This can be beneficial for collaborations in which one or more authors are not familiar with markdown syntax.

Document versioning and change control

Programmers, especially when working in distributed teams, rely on version control systems to manage changes of code. Currently, Git (https://git-scm.com/), which is also used for the development of the Linux kernel, is one of the most employed software solutions for versioning. Git allows the parallel work of collaborators and has an efficient merging and conflict resolution system. A Git repository may be used by a single local author to keep track of changes, or by a team with a remote repository; for example, on Github (https://github.com/) or Bitbucket (https://bitbucket.org/). Because of the plain text format of markdown, Git can be used for version control and distributed writing. For the writing of the present article, the co-authors (Germany and Mexico) used a remote Git repository on Bitbucket. The plain text syntax of markdown facilitates the visualization of differences of document versions, as shown in Fig. 7.

Tables

There are several options to write tables in markdown. The most flexible alternative—which was also used for this article—are pipe tables. The contents of different cells are separated by pipe symbols (—):

 
 
 Left | Center | Right | Default 
:-----|:------:|------:|--------- 
 LLL  | CCC    | RRR   | DDD    

gives

The headings and the alignment of the cells are given in the first two lines. The cell width is variable. The Pandoc parameter –columns=NUM can be used to define the length of lines in characters. If contents do not fit, they will be wrapped.

Complex tables (for example, tables featuring multiple headers or those containing cells spanning multiple rows or columns), are currently not representable in Markdown format. However, it is possible to embed LATEX and HTML tables into the document. These format-specific tables will only be included in the output if a document of the respective format is produced. This is method can be extended to apply any kind of format-specific typographic functionality which would otherwise be unavailable in Markdown syntax.

Figures and images

Images are inserted as follows:

 
 
![alt text](image location/ name)    

e.g.,

 
 
![Publishing costs](fig-hybrid-publishing-costs.png)    

The alt text is used e.g., in HTML output. Image dimensions can be defined in braces:

 
 
![](fig-hybrid-publishing-costs.png){width=50mm}    

As well, an identifier for the figure can be defined with #, resulting e.g., in the image attributes {#figure1 height=30%}.

A paragraph containing only an image is interpreted as a figure. The alt text is then output as the figure’s caption.

Symbols

Scientific texts often require special characters; for example, Greek letters, mathematical and physical symbols, and so on.

The UTF-8 standard, developed and maintained by the Unicode Consortium, enables the use of characters across languages and computer platforms. The encoding is defined as RFC document 3629 of the Network Working group (Yergeau, 2003) and as ISO standard ISO/IEC 10646:2014 (International Organization for Standardization, 2014). Specifications of Unicode and code charts are provided on the Unicode homepage (http://www.unicode.org/).

In Pandoc Markdown documents, Unicode characters such as ∘, α, ä, Å can be inserted directly and passed to the different output documents. The correct processing of MD with UTF-8 encoding to LATEX/PDF output requires the use of the –latex-engine=xelatex option and the use of an appropriate font. The Times-like XITS font (https://github.com/khaledhosny/xits-math), suitable for high quality typesetting of scientific texts, can be set in the LATEX template:

 
 
∖usepackage{unicode-math} 
∖setmainfont 
[    Extension = .otf, 
   UprightFont = *-regular, 
      BoldFont = *-bold, 
    ItalicFont = *-italic, 
BoldItalicFont = *-bolditalic, 
]{xits} 
∖setmathfont 
[    Extension = .otf, 
      BoldFont = *bold, 
]{xits-math}    

To facilitate the input of specific characters, so-called mnemonics can be enabled in some editors (e.g., in atom by the character-table package). For example, the 2-character mnemonics ‘:u’ gives ‘ü’ (diaeresis), or ‘D*’ the Greek Δ. The possible character mnemonics and character sets are listed in RFC 1345: http://www.faqs.org/rfcs/rfc1345.html (Simonsen, 1992).

Formulas

Formulas are written in LATEX mode using the delimiters $. For example, the formula for calculating the standard deviation s of a random sampling would be written as:

 
 
$s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N(x_i-\overline{x})^{2}}$    

and gives: ${\displaystyle s=\sqrt{\frac{1}{N-1}\sum _{i=1}^N{\left(x_i-\overline{x}\right)}^2}}$ with x_i the individual observations, $\overline{x}$ the sample mean and N the total number of samples.

Pandoc parses formulas into internal structures and allows conversion into formats other than LATEX. This allows for format-specific formula representation and enables computational analysis of the formulas (Corbí& Burgos, 2015).

Code listings

Verbatim code blocks are indicated by three tilde symbols:

 
 
~~~ 
verbatim code 
~~~    

Typesetting inline code is possible by enclosing text between back ticks.

 
 
‘inline code‘    

Other document elements

These examples are only a short demonstration of the capacities of Pandoc concerning scientific documents. For more detailed information, we refer to the official manual (http://pandoc.org/MANUAL.html).

Reference databases

Pandoc is able to process all mainstream literature database formats, such as RIS, BIB, etc. However, for maintaining compatibility with LATEX/ BIBTEX, the use of BIB databases is recommended. The used database either can be defined in the YAML metablock of the MD file (see below) or it can be passed as parameter when calling Pandoc.

Inserting citations

For inserting a reference, the database key is given within square brackets, and indicated by an ‘@’. It is also possible to add information, such as page:

 
 
[@suber_open_2012; @benkler_wealth_2006, 57 ff.]    

gives (Suber, 2012; Benkler, 2006, p. 57 ff.).

Styles

The Citation Style Language (CSL) (http://citationstyles.org/) is used for the citations and bibliographies. This file format is supported, for example, by the reference management programs Mendeley (https://www.mendeley.com/), Papers (http://papersapp.com/) and Zotero (https://www.zotero.org/). CSL styles for particular journals can be found from the Zotero style repository (https://www.zotero.org/styles). The bibliography style that Pandoc should use for the target document can be chosen in the YAML block of the Markdown document or can be passed in as an command line option. The latter is more recommendable, because distinct bibliography style may be used for different documents.

Creation of LATEX natbib citations

For citations in scientific manuscripts written in LATEX, the natbib package is widely used. To create a LATEX output file with natbib citations, Pandoc simply has to be run with the –natbib option, but without the –filter pandoc-citeproc parameter.

Database of cited references

To share the bibliography for a certain manuscript with co-authors or the publisher’s production team, it is often desirable to generate a subset of a larger database, which only contains the cited references. If LATEX output was generated with the –natbib option, the compilation of the file with LATEX gives an AUX file (in the example named md-article.aux), which subsequently can be extracted using BibTool (https://github.com/ge-ne/bibtool):

 
 
~~~ 
bibtool -x md-article.aux -o bibshort.bib 
~~~    

In this example, the article database will be called bibshort.bib.

For the direct creation of an article specific BIB database without using LATEX, we wrote a simple Perl script called mdbibexport (https://github.com/robert-winkler/mdbibexport).

Flexible metadata authoring

We developed a method to allow writers the flexible specification of authors and their respective affiliations. Author names can be given as a string, via the key of a single-element object, or explicitly as a name attribute of an object. Affiliations can be specified directly as properties of the author object, or separately in the institute object.

Additional information, for example, email addresses or identifiers like ORCID (Haak et al., 2012), can be added as additional values:

 
 
author: 
  - John Doe: 
      institute: fs 
      email: john.doe@example.com 
      orcid: 0000-0000-0000-0000 
institute: 
  fs: Science Formatting Working Group    

JATS support

The journal article tag suite (JATS) was developed by the National Library of Medicine (NLM) and standardized by ANSI/NISO as an archiving and exchange format of journal articles and the associated metadata (National Information Standards Organization, 2012), including data of the type shown above. The pandoc-jats writer by Martin Fenner is a plugin usable with Pandoc to produce JATS-formatted output. The writer was adapted to be compatible with our metadata authoring method, allowing for simple generation of files which contain the relevant metadata.

Citation types

Writers can add information about the reason a citation is given. This might help reviewers and readers, and can simplify the search for relevant literature. We developed an extended citation syntax that integrates seamlessly into Markdown and can be used to add complementary information to citations. Our method is based on CiTO, the Citation Typing Ontology (Shotton, 2010), which specifies a vocabulary for the motivation when citing a resource. The type of a citations can be added to a Markdown citation using @CITO_PROPERTY:KEY, where CITO_PROPERTY is a supported CiTO property, and KEY is the usual citation key. Our tool extracts that information and includes it in the generated linked data output. A general CiTO property (cites) is used, if no CiTO property is found in a citation key.

The work at hand will always be the subject of the generated semantic subject–predicate–object triples. Some CiTO predicates cannot be used in a sensical way under this condition. Focusing on author convenience, we use this fact to allow shortening of properties when sensible. For example, if authors of a biological paper include a reference to the paper describing a method which was used in their work, this relation can be described by the uses_method_in property of the CiTO ontology. The inverse property, provides_method_for, would always be nonsensical in this context as implied by causality. It is therefore not supported by our tool. This allows us to introduce an abbreviation (method) for the latter property, as any ambiguity has been eliminated. Users of Western blotting might hence write @method_in:towbin_1979 or even just @method:towbin_1979, where towbin_1979 is the citation identifier of the describing paper by Towbin, Staehelin & Gordon (1979).

Development of a DOCX/ ODT template

A first DOCX document with bibliography in PLoS format is created with Pandoc DOCX output:

 
 
pandoc -S -s --csl=plos.csl --filter pandoc-citeproc 
  -o pandoc-manuscript.docx agile-editing-pandoc.md    

The parameters -S -s generate a typographically correct (dashes, non-breaking spaces etc.) stand-alone document. A bibliography with the PLoS style is created by the citeproc filter setting –csl=plos.csl –filter pandoc-citeproc.

The document settings and styles of the resulting file pandoc-manuscript.docx can be optimized and be used again as document template (–reference-docx=pandoc-manuscript.docx): instead of .

 
 
pandoc -S -s --reference-docx=pandoc-manuscript.docx --csl=plos.csl 
  --filter pandoc-citeproc -o outfile.docx agile-editing-pandoc.md    

It is also possible to directly re-use a previous output file as template (i.e., template and output file have the same file name):

 
 
pandoc -S -s --columns=10 --reference-docx=pandoc-manuscript.docx 
  --csl=plos.csl --filter=pandoc-citeproc 
  -o pandoc-manuscript.docx agile-editing-pandoc.md    

In this way, the template can be incrementally adjusted to the desired document formatting. The final document may be employed later as Pandoc template for other manuscripts with the same specifications. In this case, running Pandoc the first time with the template, the contents of the new manuscript would be filled into the provided DOCX template. A page with DOCX manuscript formatting of this article is shown in Fig. 8.

The same procedure can be applied with an ODT formatted document.

Development of a TEX/PDF template

The default Pandoc LATEX template can be written into a separate file by:

 
 
pandoc -D latex > template-peerj.latex    

This template can be adjusted; for example, by defining Unicode encoding (see above), by including particular packages or setting document options (line numbering, font size). The template can then be used with the Pandoc parameter –template=pandoc-peerj.latex.

The templates used for this document are included as Supplemental Material (see section ‘Software and Code Availability’ below).

Styles for HTML and EPUB

The style for HTML and EPUB formats can be defined in .css stylesheets. The Supplemental Material (see section ‘Software and Code Availability’ below) contains a simple example .css file for modifying the HTML output, which can be used with the Pandoc parameter -c pandoc.css.

Cross-platform compatibility

The make process was tested on Windows 10 and Linux 64 bit. All documents—DOCX, ODT, JATS, LATEX, PDF, EPUB and HTML—were generated successfully, which demonstrates the cross-platform compatibility of the workflow.