40Ants Doc Manual

About this fork

This system is a fork of MGL-PAX.

There are a few reasons, why I've created the fork.

The main goal is to extract a core features into the 40ants-doc system with as little dependencies as possible. This is important, because with MGL-PAX's style, you define documentation sections in your library's code, which makes it dependent on the documentation system. However, heavy weight dependencies like IRONCLAD, 3BMD or SWANK should not be required.

The seconds goal was to refactor a 3.5k lines of pax.lisp file into a smaller modules to make navigation easier. This will help any person who will decide to learn how the documentation builder works. Also, granular design will make it possible loading subsystems like SLIME or SLY integration.

The third goal was to make documentation processing more sequential and hackable. To introduce hooks for adding new markup languages, and HTML themes.

Why this fork is different

Here are features already implemented in this fork:

• Core system 40ants-doc now has only two dependencies on NAMED-READTABLES and PYTHONIC-STRING-READER. If you want to compile a documentation, load 40ants-doc-full system which will download such dependencies as markdown parser and more.

• Now you don't have to import any locative symbols into your package. Import only a defsection macro and it will be enough to define documentation for your library!

• Added a warning mechanism, which will issue such warnings on words which looks like a symbol, but when real symbol or reference is absent:

WARNING: Unable to find target for reference #<XREF "FIND-SOURCE" GENERIC-FUNCTION> mentioned at 40Ants Doc Manual / Extension API / Reference Based Extensions

• Documentation processing now uses CommonDoc as intermediate format, and markup languages other than Markdown can be supported.

• Added a JS search index which will work when you are hosting pages on a static website like GitHub pages.

• It is possible to render pages in multiple formats and having cross references between them. See Multiple Formats.

I'm planning to extend this fork even more. Read todo section to learn about proposed features or start a new discussion on the GitHub to suggest a new feature.

See full list of changes in the ChangeLog section.

40ANTS-DOC ASDF System Details

• Version: 0.12.0

• Description: Allows to put documentation inside lisp files and cross-reference between different entities. Based on MGL-PAX.

• Licence: MIT

• Author: Alexander Artemenko

• Mailto: svetlyak.40wt@gmail.com

• Homepage: https://40ants.com/doc

• Bug tracker: https://github.com/40ants/doc/issues

• Source control: GIT

• Depends on: asdf, named-readtables, pythonic-string-reader, uiop

40ANTS-DOC-FULL ASDF System Details

• Version: 0.1.0

• Description: Documentation generator. You will need to load this system, to build documentation for a library which uses 40ants-doc system.

• Licence: MIT

• Author: Alexander Artemenko

• Mailto: svetlyak.40wt@gmail.com

• Homepage: https://40ants.com/doc

• Bug tracker: https://github.com/40ants/doc/issues

• Source control: GIT

• Depends on: 40ants-doc, alexandria, babel, cl-fad, cl-ppcre, closer-mop, common-doc, common-html, commondoc-markdown, dexador, docs-config, fare-utils, jonathan, lass, local-time, log4cl, named-readtables, pythonic-string-reader, slynk, spinneret, stem, str, swank, tmpdir, which, xml-emitter

Links

Here is the official repository and the HTML documentation for the latest version.

This system is a fork of the MGL-PAX. Because of massive refactoring, it is incompatible with original repository.

Background

Here is the story behind the MGL-PAX, precursor of 40ants-doc, written by Gábor Melis.

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with SLIME's M-.. As a library author, I spend a great deal of time polishing code, but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programming weenie turned inside out. The original prototype which did almost everything I wanted was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with defsection, I soon found myself organizing code following the flow of user level documentation and relegated comments to implementational details entirely. However, some portions of defsection docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was effectively repeated in the DEFPACKAGE form complete with little comments that were like section names. A clear violation of OAOO, one of them had to go, so defsection got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class are named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type which was later generalized into the concept of locatives:

(defsection @introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol and everything was good for a while.

Then I realized that sections could refer to other sections if there were a section locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was manifest in the defsection forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able find out what's being referred to.

I settled on Markdown as a reasonably non-intrusive format, and a few thousand lines later MGL-PAX was born.

Tutorial

40ants-doc provides an extremely poor man's Explorable Programming environment. Narrative primarily lives in so called sections that mix markdown docstrings with references to functions, variables, etc, all of which should probably have their own docstrings.

The primary focus is on making code easily explorable by using SLIME's M-. (slime-edit-definition). See how to enable some fanciness in Emacs Integration. Generating documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

With the simplistic tools provided, one may accomplish similar effects as with Literate Programming, but documentation is generated from code, not vice versa and there is no support for chunking yet. Code is first, code must look pretty, documentation is code.

When the code is loaded into the lisp, pressing M-. in SLIME on the name of the section will take you there. Sections can also refer to other sections, packages, functions, etc and you can keep exploring.

Here is an example of how it all works together:

(uiop:define-package #:foo-random
  (:nicknames #:40ants-doc-full/tutorial)
  (:documentation "This package provides various utilities for
                   random. See @FOO-RANDOM-MANUAL.")
  (:use #:common-lisp
        #:40ants-doc)
  (:import-from #:40ants-doc/ignored-words
                #:ignore-words-in-package)
  (:export #:foo-random-state
           #:state
           #:*foo-state*
           #:gaussian-random
           #:uniform-random))

(in-package foo-random)

(defsection @foo-random-manual (:title "Foo Random manual"
                                :ignore-words ("FOO"))
  "Here you describe what's common to all the referenced (and
   exported) functions that follow. They work with *FOO-STATE*,
   and have a :RANDOM-STATE keyword arg. Also explain when to
   choose which."
  (foo-random-state class)
  (state (reader foo-random-state))
  
  "Hey we can also print states!"
  
  (print-object (method () (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; this is a subsection
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state
          :documentation "Returns random foo's state.")))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the between 0 and LIMIT (exclusive)
   uniform distribution."
  (declare (ignore limit random-state))
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return not a random number from a zero mean normal distribution with
   STDDEV."
  (declare (ignore stddev random-state))
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
   with FOO:

   ```cl-transcript
   (values (princ :hello) (list 1 2))
   .. HELLO
   => :HELLO
   => (1 2)

   (make-instance 'foo-random-state)
   ==> #<FOO-RANDOM-STATE >
   ```")

Generating documentation in a very stripped down markdown format is easy:

(40ants-doc-full/builder:render-to-string
  @foo-random-manual
  :format :markdown)

For this example, the generated markdown would look like this:

<a id="x-28FOO-RANDOM-3A-3A-40FOO-RANDOM-MANUAL-2040ANTS-DOC-2FLOCATIVES-3ASECTION-29"></a>

# Foo Random manual

Here you describe what's common to all the referenced (and
exported) functions that follow. They work with [`*foo-state*`][2133],
and have a `:RANDOM-STATE` keyword arg. Also explain when to
choose which.

<a id="x-28FOO-RANDOM-3AFOO-RANDOM-STATE-20CLASS-29"></a>

## [class](2671) `foo-random:foo-random-state` ()

<a id="x-28FOO-RANDOM-3ASTATE-20-2840ANTS-DOC-2FLOCATIVES-3AREADER-20FOO-RANDOM-3AFOO-RANDOM-STATE-29-29"></a>

## [reader](1595) `foo-random:state` (foo-random-state) ()

Returns random foo's state.

Hey we can also print states!

<a id="x-28PRINT-OBJECT-20-28METHOD-20NIL-20-28FOO-RANDOM-3AFOO-RANDOM-STATE-20T-29-29-29"></a>

## [method](9473) `common-lisp:print-object` (object foo-random-state) stream

<a id="x-28FOO-RANDOM-3A-2AFOO-STATE-2A-20-28VARIABLE-29-29"></a>

## [variable](4a88) `foo-random:*foo-state*` #<foo-random-state >

Much like `*RANDOM-STATE*` but uses the `FOO` algorithm.

<a id="x-28FOO-RANDOM-3AGAUSSIAN-RANDOM-20FUNCTION-29"></a>

## [function](0da3) `foo-random:gaussian-random` stddev &key (random-state \*foo-state\*)

Return not a random number from a zero mean normal distribution with
`STDDEV`.

<a id="x-28FOO-RANDOM-3AUNIFORM-RANDOM-20FUNCTION-29"></a>

## [function](6a76) `foo-random:uniform-random` limit &key (random-state \*foo-state\*)

Return a random number from the between 0 and `LIMIT` (exclusive)
uniform distribution.

<a id="x-28FOO-RANDOM-3A-3A-40FOO-RANDOM-EXAMPLES-2040ANTS-DOC-2FLOCATIVES-3ASECTION-29"></a>

## Examples

Let's see the transcript of a real session of someone working
with `FOO`:

```cl-transcript
(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

(make-instance 'foo-random-state)
==> #<FOO-RANDOM-STATE >
```

[2133]: #x-28FOO-RANDOM-3A-2AFOO-STATE-2A-20-28VARIABLE-29-29
[2671]: https://github.com/40ants/doc/blob/8518b7169bfae0384d58d869d0662bfce91a1813/full/tutorial.lisp#L35
[1595]: https://github.com/40ants/doc/blob/8518b7169bfae0384d58d869d0662bfce91a1813/full/tutorial.lisp#L36
[9473]: https://github.com/40ants/doc/blob/8518b7169bfae0384d58d869d0662bfce91a1813/full/tutorial.lisp#L39
[4a88]: https://github.com/40ants/doc/blob/8518b7169bfae0384d58d869d0662bfce91a1813/full/tutorial.lisp#L42
[6a76]: https://github.com/40ants/doc/blob/8518b7169bfae0384d58d869d0662bfce91a1813/full/tutorial.lisp#L45
[0da3]: https://github.com/40ants/doc/blob/8518b7169bfae0384d58d869d0662bfce91a1813/full/tutorial.lisp#L51

MGL-PAX supported the plain text format which was more readble when viewed from a simple text editor, but I've dropped support for plain text in this fork because most time documentation are read in the browser these days.

To render into the files, use 40ants-doc-full/builder:render-to-files and 40ants-doc-full/builder:update-asdf-system-docs functions.

Last one can even generate documentation for different, but related libraries at the same time with the output going to different files, but with cross-page links being automatically added for symbols mentioned in docstrings. See Generating Documentation for some convenience functions to cover the most common cases.

Note how (*FOO-STATE* VARIABLE) in the defsection form includes its documentation in @FOO-RANDOM-MANUAL. The symbols variable and function are just two instances of 'locatives' which are used in defsection to refer to definitions tied to symbols. See Locative Types.

The transcript in the code block tagged with cl-transcript is automatically checked for up-to-dateness. See Transcripts.

Emacs Integration

Integration into SLIME's M-. (slime-edit-definition) allows one to visit the source location of the thing that's identified by a symbol and the locative before or after the symbol in a buffer. With this extension, if a locative is the previous or the next expression around the symbol of interest, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

Note that the this feature is implemented in terms of SWANK-BACKEND:FIND-SOURCE-LOCATION and SWANK-BACKEND:FIND-DEFINITIONS whose support varies across the Lisp implementations. Sadly, but this integration does not with SLY because it does not support hooks on finding definition.

In the following examples, pressing M-. when the cursor is on one of the characters of FOO or just after FOO, will visit the definition of function FOO:

function foo
foo function
(function foo)
(foo function)

In particular, references in a defsection form are in (SYMBOL locative) format so M-. will work just fine there.

Just like vanilla M-., this works in comments and docstrings. In this example pressing M-. on FOO will visit FOO's default method:

;;;; See FOO `(method () (t t t))` for how this all works.
;;;; But if the locative has semicolons inside: FOO `(method
;;;; () (t t t))`, then it won't, so be wary of line breaks
;;;; in comments.

With a prefix argument (C-u M-.), one can enter a symbol plus a locative separated by whitespace to preselect one of the possibilities.

The M-. extensions can be enabled by adding this to your Emacs initialization file (or loading src/pax.el):

;;; M-. integration

(defun 40ants-doc-edit-locative-definition (name &optional where)
  (or (40ants-doc-locate-definition name (40ants-doc-locative-before))
      (40ants-doc-locate-definition name (40ants-doc-locative-after))
      (40ants-doc-locate-definition name (40ants-doc-locative-after-in-brackets))
      ;; support "foo function" and "function foo" syntax in
      ;; interactive use
      (let ((pos (cl-position ?\s name)))
        (when pos
          (or (40ants-doc-locate-definition (cl-subseq name 0 pos)
                                            (cl-subseq name (1+ pos)))
              (40ants-doc-locate-definition (cl-subseq name (1+ pos))
                                            (cl-subseq name 0 pos)))))))

(defun 40ants-doc-locative-before ()
  (ignore-errors (save-excursion
                   (slime-beginning-of-symbol)
                   (slime-last-expression))))

(defun 40ants-doc-locative-after ()
  (ignore-errors (save-excursion
                   (slime-end-of-symbol)
                   (slime-forward-sexp)
                   (slime-last-expression))))

(defun 40ants-doc-locative-after-in-brackets ()
  (ignore-errors (save-excursion
                   (slime-end-of-symbol)
                   (skip-chars-forward "`" (+ (point) 1))
                   (when (and (= 1 (skip-chars-forward "\\]" (+ (point) 1)))
                              (= 1 (skip-chars-forward "\\[" (+ (point) 1))))
                     (buffer-substring-no-properties
                      (point)
                      (progn (search-forward "]" nil (+ (point) 1000))
                             (1- (point))))))))

(defun 40ants-doc-locate-definition (name locative)
  (when locative
    (let ((location
           (slime-eval
            ;; Silently fail if mgl-pax is not loaded.
            `(cl:when (cl:find-package :mgl-pax)
                      (cl:funcall
                       (cl:find-symbol
                        (cl:symbol-name :locate-definition-for-emacs) :mgl-pax)
                       ,name ,locative)))))
      (when (and (consp location)
                 (not (eq (car location) :error)))
        (slime-edit-definition-cont
         (list (make-slime-xref :dspec `(,name)
                                :location location))
         "dummy name"
         where)))))

(when (boundp 'slime-edit-definition-hooks)
  (add-hook 'slime-edit-definition-hooks '40ants-doc-edit-locative-definition))

Note, there is also another part of Emacs code, related to transcription blocks. It is described in Transcripts section.

Basics

Now let's examine the most important pieces in detail.

Defining Sections

(symbol locative) or ((symbol1 symbol2 ... symboln) locative)

(defparameter @readme (40ants-doc:copy-section @index))

(eval-when (:compile-toplevel :load-toplevel :execute)
  (pushnew 'reblocks-auth/models:profile-user
           40ants-doc:*symbols-with-ignored-missing-locations*))

Cross-referencing

You can cross-reference entries from different documentation parts be it content of the defsection or a documentation string of some lisp entity.

The simples form of cross reference is uppercased name of the entity, like: 40ants-doc/reference:make-reference. But if there are more than one locative bound to the name, then all these links will be rendered in a parenthesis. For example, docstring:

See 40ANTS-DOC/SOURCE-API:FIND-SOURCE.

will be rendered as "See 40ants-doc/source-api:find-source (1 2)." because there is a generic-function and a method called find-source (1 2).

But you can mention a locative type in a docstring before or after a symbol name:

See 40ANTS-DOC/SOURCE-API:FIND-SOURCE generic-function.

and it will be rendered as: See 40ants-doc/source-api:find-source generic-function.

In case if you don't want locative type to appear in the resulting documentation or if locative type is complex, then you can use in a docstring markdown reference:

See [40ANTS-DOC/SOURCE-API:FIND-SOURCE][(method () (40ants-doc/reference:reference))].

and link will lead to the specified method: See 40ants-doc/source-api:find-source.

Autodocumentation

40ants-doc system provides an additional subsystem and package 40ANTS-DOC/AUTODOC. This subsystem contains a macro defautodoc, which is similar to defsection, but generates a section filled with content of the given ASDF system.

This subsystem is not loaded by default because it brings a multiple additional dependencies:

• alexandria

• cl-change-case

• cl-ppcre

• cl-unicode

• closer-mop

but I'm trying to keep dependencies of the core 40ants-doc system is minimal.

Use it if your don't care or your have docs in a separate ASDF sybsystem.

(40ants/defautodoc @api (:system :cl-telegram-bot))

(defsection @index (:title "cl-telegram-bot - Telegram Bot API")
  (@installation section)
  (@quickstart section)
  (@api section))

When SHOW-SYSTEM-DESCRIPTION-P argument is not NIL, section will be started from the
description of the given ASDF system.

Generating Documentation

To make documentation builder work, you need to load 40ants-doc-full asdf system.

There are two core functions which render documentation to a string or files:

Besides render-to-string and render-to-files a convenience function is provided to serve the common case of having an ASDF system with a readme and a directory for the HTML documentation.

(40ants-doc-full/builder:update-asdf-system-docs 40ants-doc-full/doc:@index
                                                 :40ants-doc
                                                 :readme-sections 40ants-doc-full/doc:@readme)

Multiple Formats

With 40ants-doc you can render HTML and Markdown documentation simultaneously. This way, you can cross-reference entities from the README.md or ChangeLog.md to HTML docs.

To render documents in multiple formats, you have to pass to function render-to-files not 40ants-doc:section objects, but PAGE objects. Page object consists of one or more sections and additional information such as document format. A section can belong to a multiple pages usually having different formats. This allows you to include "tutorial" section into both HTML docs and README.

Here is an example of rendering the full documentation and a README with only introduction and tutorial:

(defsection @full-manual (:title "Manual")
  (@introduction)
  (@tutorial)
  (@api)
  (@changelog))

(render-to-files
 (list @full-manual
       (40ants-doc-full/page:make-page (list @introduction
                                        @tutorial)
                                       :format :markdown
                                       :base-filename "README")
       (40ants-doc-full/page:make-page @changelog
                                       :format :markdown
                                       :base-filename "ChangeLog")))

The same approach works with the update-asdf-system-docs function.

Changelog Generation

(defchangelog ()
  (0.2.0
   "- Feature B implemented.
    - Bug was fixed in function FOO.")
  
  (0.1.0
   "- Project forked from [MGL-PAX](https://github.com/melisgl/mgl-pax).
    - Feature A implemented."))

Github Workflow

It is generally recommended to commit generated readmes (see 40ants-doc-full/builder:update-asdf-system-docs) so that users have something to read without reading the code and sites like github can display them.

HTML documentation can also be committed, but there is an issue with that: when linking to the sources (see make-github-source-uri-fn), the commit id is in the link. This means that code changes need to be committed first, then HTML documentation regenerated and committed in a followup commit.

To serve static documentation, use gh-pages. You can use a separate branch gh-pages, or point GitHub Pages to a docs folder inside the main branch. Good description of this process is http://sangsoonam.github.io/2019/02/08/using-git-worktree-to-deploy-github-pages.html. Two commits needed still, but it is somewhat less painful.

This way the HTML documentation will be available at http://<username>.github.io/<repo-name>. It is probably a good idea to add section like the Links section to allow jumping between the repository and the gh-pages site.

https://github.com/melisgl/mgl-pax/blob/master/src/pax-early.lisp#L12

(defsystem 40ants-doc
  ...
  :source-control (:git "https://github.com/40ants/doc")
  ...))))

PAX World

MGL-PAX supported a "World" which was a registry of documents, which can generate cross-linked HTML documentation pages for all the registered documents.

But I decided to drop this feature for now, because usually build libraries documentation separately as part of their CI pipline.

If somebody want's cross referencing between different libraries, then instead of building their docs simultaneously, I'd suggest to create an index of entities, provided by libraries and to store them as a JSON file along with a library documentation.

This way it will be possible to enumerate such sources of cross references as usual URLs.

Such feature is not implemented in the 40ants-doc system yet, but probably it will be useful for libraries built around the Weblocks. If you want to help and implement the feature, please, let me know.

Markdown Support

The Markdown in docstrings is processed with the 3BMD library.

Indentation

Docstrings can be indented in any of the usual styles. 40ants-doc normalizes indentation by converting:

(defun foo ()
  "This is
   indented
   differently")

to

(defun foo ()
  "This is
indented
differently")

Docstrings in sources are indented in various ways which can easily mess up markdown. To handle the most common cases leave the first line alone, but from the rest of the lines strip the longest run of leading spaces that is common to all non-blank lines."

Syntax highlighting

For syntax highlighting, github's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. The language tag, elisp in this example, is optional and defaults to commonlisp.

Originally MGL-PAX used colorize for the syntax highlighting, but 40ants-doc uses Highlight.js which is able to guess code block language if it is not specified. To minimize HTML document's static size, Hightlight.js is configured to support only these languages:

• bash

• css

• json

• yaml

• plain-text

• html

• markdown

• lisp

There is a separate README where you will find instructions on how to support other languages.

Besides an automatic language detection, the other cool feature of Highlight.js is it's support for different color themes. Here you can view all available themes: https://highlightjs.org/static/demo/. There is no easy way to choose color theme yet, but probably this will be a nice feature for 40ants-doc.

MathJax

Displaying pretty mathematics in TeX format is supported via MathJax. It can be done inline with $ like this:

$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$

which is diplayed as $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$, or it can be delimited by $$ like this:

$$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

to get: $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

MathJax will leave code blocks (including those inline with backticks) alone. Outside code blocks, escape $ by prefixing it with a backslash to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be a pain. Pythonic String Reader can help with that.

Documentation Printer Variables

Docstrings are assumed to be in markdown format and they are pretty much copied verbatim to the documentation subject to a few knobs described below.

Note, some of these variables might be not supported yet in this fork.

"`FOO` and FOO."

"`FOO` and `FOO`."

(defsection @foo
  (foo function)
  (bar function))

(defun foo (x)
  "Calls `BAR` on `X`."
  (bar x))

- [function] FOO X

    Calls [`BAR`][1] on `X`.

(defsection @foo
  (foo function)
  (bar function)
  (bar type))

(defun foo (x)
  "Calls `BAR` on `X`."
  (bar x))

- [function] FOO X

    Calls `BAR`([`1`][link-id-1] [`2`][link-id-2]) on `X`.

"Calls [BAR][function] on X."

"Calls function `BAR` on X."

Locative Types

These are the locatives type supported out of the box. As all locative types, they are symbols and their names should make it obvious what kind of things they refer to. Unless otherwise noted, locatives take no arguments.

(foo (method () (t t t)))

(foo-slot (accessor foo))

(foo-slot (reader foo))

(foo-slot (writer foo))

`FOO`

[`FOO`][dislocated]

"See the FORMAT in DOCUMENT."

"See the FORMAT argument of DOCUMENT."

(defsection example-section ()
  (pax.el (include #.(asdf:system-relative-pathname :40ants-doc "elisp/pax.el")
                   :lang "elisp"))
  (foo-example (include (:start (foo function)
                         :end (end-of-foo-example variable))
                        :lang "commonlisp")))

(defun foo (x)
  (1+ x))

;;; Since file regions are copied verbatim, comments survive.
(defmacro bar ())

;;; This comment is the last thing in FOO-EXAMPLE's
;;; documentation since we use the dummy END-OF-FOO-EXAMPLE
;;; variable to mark the end location.
(defvar end-of-foo-example)

;;; More irrelevant code follows.

(defsection @example ()
 (describe-output (stdout-of (format t "Hello World!"))))

```markdown
Hello World!
```

(define-restart my-ignore-error ()
  "Available when MY-ERROR is signalled, MY-IGNORE-ERROR unsafely continues.")

There is also a helper function to compare locatives:

Extension API

Defining a Custom Theme

Out of the box, 40ants-doc system supports three color themes:

• 40ants-doc-full/themes/default:default-theme

• 40ants-doc-full/themes/light:light-theme

• 40ants-doc-full/themes/dark:dark-theme

You can pass these names as THEME argument to the 40ants-doc-full/builder:render-to-files function. Or you can pass a contructed theme object instead of a symbolic class name.

And of cause, you can define your own theme!

Theme allows to control HTML page rendering, colors and code highlighting.

Changing Colors

The simplest way to customize theme is to redefine some colors using CSS. Here is how to set orange page background:

(defclass my-theme (default-theme)
  ())

(defmethod 40ants-doc-full/themes/api:render-css ((theme my-theme))
  (concatenate
   'string
   (call-next-method)
  
   (lass:compile-and-write
    `(body
      :background orange))))

Also you might want to redefine a color theme for code highlighter:

(defmethod 40ants-doc-full/themes/api:highlight-theme ((theme my-theme))
  "atom-one-light")

Talking about code highlighting, you can also redefine a list of languages to highlight:

(defmethod 40ants-doc-full/themes/api:highlight-languages ((theme my-theme))
  (list "lisp"
        "python"
        "bash"))

Changing Page Layout

The main entry-point for page rendering is render-page generic-function. It calls all other rendering functions.

If you are inheriting your theme class from 40ants-doc-full/themes/default:default-theme, then rendering functions will be called in the following order:

On this page stripes on the right demonstrate order in which different rendering functions will be called:

• green is render-page;

• blue is render-page-header;

• violet is render-sidebar

• red is render-sidebar-header

• yellow is render-sidebar-content;

• orange is render-sidebar-footer;

• salad green is render-content;

• pink is render-page-footer.

Some of these methods might call render-toc and render-search-form to display a table of content and a table form. Also, you might want to redefine render-html-head generic-function to change html page metadata such as included stylesheets and js files, page title, etc.

If you want to introduce changes, it is better to inherit from existing theme class and to define a few methods to change only needed properties. For example, here is a theme I've made for all 40Ants projects. I've added header, footer and made colors match the main site.

Available Themes

Theme Definition Protocol

Plugins API

40ants-doc themes support plugins. Plugins are small objects holding some configuration parameters and able to inject additional content into documentation pages or able to copy some static files to the results directory.

By default, only highlightjs plugin is enabled, but you can pass a custom list of plugins when creating a theme object. For example, here is how to can enable both Hightlight.js and MathJax plugins:

(defsection @index (:title "Example")
  "MathJax example:

   1. $a ne 0$
   2. $x = {-b pm sqrt{b^2-4ac} over 2a}.$")

(40ants-doc-full/builder:render-to-files
 @index
 :base-dir "/tmp/output/"
 :theme (make-instance '40ants-doc-full/themes/light:light-theme
                       :plugins (list
                                 (highlightjs)
                                 (mathjax))))

Built-in Plugins

(make-instance '40ants-doc-full/themes/light:light-theme
               :plugins (list
                         (highlightjs :theme "magula"
                                      :languages '("lisp" "python"))))

Locatives and References

While Common Lisp has rather good introspective abilities, not everything is first class. For example, there is no object representing the variable defined with (DEFVAR FOO). (40ANTS-DOC/REFERENCE:MAKE-REFERENCE 'FOO 'VARIABLE) constructs a 40ants-doc/reference:reference that captures the path to take from an object (the symbol FOO) to an entity of interest (for example, the documentation of the variable). The path is called the locative. A locative can be applied to an object like this:

(locate 'foo 'variable)

which will return the same reference as (40ANTS-DOC/REFERENCE:MAKE-REFERENCE 'FOO 'VARIABLE). Operations need to know how to deal with references which we will see in 40ants-doc/locatives/base:locate-and-find-source (1 2).

Naturally, (40ANTS-DOC/LOCATIVES/BASE:LOCATE 'FOO 'FUNCTION) will simply return #'FOO, no need to muck with references when there is a perfectly good object.

Adding New Object Types

If you wish to make it possible to render documentation for a new object type, then you have to define a method for the 40ants-doc-full/commondoc/builder:to-commondoc generic function. And to make M-. navigation work with new object types, a methods of 40ants-doc/locatives/base:locate-object generic-function and 40ants-doc/source-api:find-source generic-function are to be defined. Also, additional method for 40ants-doc/reference-api:canonical-reference generic-function need to be defined to make an opposite to 40ants-doc/locatives/base:locate-object's action.

Finally, 40ants-doc:exportable-locative-type-p generic-function may be overridden if exporting does not makes sense. Here is a stripped down example of how all this is done for asdf:system:

(define-locative-type asdf:system ()
  "Refers to an asdf system. The generated documentation will include
  meta information extracted from the system definition. This also
  serves as an example of a symbol that's not accessible in the
  current package and consequently is not exported.

  A title of the documentation section can be modified if you'll
  define a method for 40ANTS-DOC/LOCATIVES/ASDF-SYSTEM:ASDF-SYSTEM-DOCUMENTATION-TITLE generic-function.
  Use EQL specifier for the method.")


(defun find-system (name)
  "ASDF:FIND-SYSTEM is 1000 times slower than ASDF:REGISTERED-SYSTEM,
   but REGISTERED-SYSTEM sometimes unable to find a system (for example
   when this is a primary ASDF system, but it's defpackage defines
   package with the name of primary system and a nickname equal to the
   subsystem name. See log4cl-extras/core as example).

   This we first try to use fast method and fallback to the slow one."
  (or (asdf:registered-system name)
      (asdf:find-system name)))


(defmethod locate-object (symbol (locative-type (eql 'asdf:system))
                          locative-args)
  (assert (endp locative-args))
  ;; FIXME: This is slow as hell.
  ;; TODO: check if replacement of find-system with registered-system helped
  (or (find-system symbol)
      (locate-error)))

(defmethod canonical-reference ((system asdf:system))
  (40ants-doc/reference:make-reference (asdf:primary-system-name system)
                                       'asdf:system))

(defmethod find-source ((system asdf:system))
  `(:location
    (:file ,(namestring (asdf/system:system-source-file system)))
    (:position 1)
    (:snippet "")))

(defmethod to-commondoc ((system asdf:system))
  (let ((title (asdf-system-documentation-title system)))
    (flet ((item (name getter &key type)
             (let* ((value (funcall getter system))
                    (href nil))
               (when value
                 (case type
                   (:link (setf href value))
                   (:mailto (setf href (format nil "mailto:~A"
                                               value)))
                   (:source-control (psetf value (format nil "~A"
                                                         (first value))
                                           href (second value))))
                 (make-list-item
                  (make-paragraph
                   (cond
                     ((eql type :asdf-systems)
                      (make-content
                       (list*
                        (make-text
                         (format nil "~A: "
                                 name))
                        (loop with first = t
                              for system-name in value
                              if first
                                do (setf first nil)
                              else
                                collect (make-text ", ")
                              collect (make-web-link (format nil "https://quickdocs.org/~A"
                                                             system-name)
                                                     (make-text system-name))))))
                     (href
                      (make-content
                       (list (make-text
                              (format nil "~A: "
                                      name))
                             (make-web-link href
                                            (make-text value)))))
                     (t
                      (make-text
                       (format nil "~A: ~A"
                               name
                               value))))))))))
      
      (let* ((items (list (item "Version" 'asdf/component:component-version)
                          (item "Description" 'asdf/system:system-description)
                          (item "Licence" 'asdf/system:system-licence)
                          (item "Author" 'asdf/system:system-author)
                          (item "Maintainer" 'asdf/system:system-maintainer)
                          (item "Mailto" 'asdf/system:system-mailto
                                :type :mailto)
                          (item "Homepage" 'asdf/system:system-homepage
                                :type :link)
                          (item "Bug tracker" 'asdf/system:system-bug-tracker
                                :type :link)
                          (item "Source control" 'asdf/system:system-source-control
                                :type :source-control)
                          (item "Depends on" 'asdf-system-dependencies
                                :type :asdf-systems)))
             (children (make-unordered-list
                        (remove nil items)))
             (reference (40ants-doc/reference-api:canonical-reference system)))
        (make-section-with-reference title
                                     children
                                     reference)))))

(define-locative-type variable (&optional initform)
  "Dummy docstring.")

(:location (:file "/home/mega/own/mgl/pax/test/test.lisp")
           (:position 24) nil)

(:error "Unknown source location for SOMETHING")

Reference Based Extensions

Let's see how to extend 40ants-doc-full/builder:render-to-files and M-. navigation if there is no first class object to represent the thing of interest. Recall that 40ants-doc/locatives/base:locate returns a 40ants-doc/reference:reference object in this case:

(40ants-doc/locatives/base:locate
   '40ants-doc:*discard-documentation-p*
   'variable)
==> #<40ANTS-DOC/REFERENCE:REFERENCE 40ANTS-DOC:*DISCARD-DOCUMENTATION-P* (VARIABLE)>

Some methods of 40ants-doc/source-api:find-source generic-function defer to 40ants-doc/locatives/base:locate-and-find-source generic-function, which have LOCATIVE-TYPE in their argument list for EQL specializing pleasure.

Here is a stripped down example of how the variable locative is defined. Pay attention how it defines a method of 40ants-doc-full/commondoc/builder:reference-to-commondoc generic-function instead of 40ants-doc-full/commondoc/builder:to-commondoc. This is because we have no a lisp object to represent a variable and have to specialize method on LOCATIVE-TYPE argument:

(define-locative-type variable (&optional initform)
  "Refers to a global special variable. INITFORM, or if not specified,
  the global value of the variable is included in the documentation.")

(defmethod locate-object (symbol (locative-type (eql 'variable)) locative-args)
  (assert (<= (length locative-args) 1))
  (40ants-doc/reference:make-reference symbol (cons locative-type locative-args)))


(defmethod 40ants-doc-full/commondoc/builder:reference-to-commondoc ((symbol symbol) (locative-type (eql 'variable)) locative-args)
  (destructuring-bind (&optional (initform nil initformp)) locative-args
    (let* ((reference (canonical-reference
                       (40ants-doc/reference:make-reference symbol
                                                            (cons locative-type
                                                                  locative-args))))
           (docstring (40ants-doc/docstring:get-docstring symbol 'variable))
           (arglist (multiple-value-bind (value unboundp) (40ants-doc-full/utils::symbol-global-value symbol)
                      (cond (initformp
                             (prin1-to-string initform))
                            (unboundp "-unbound-")
                            (t
                             (prin1-to-string value)))))
           (children (when docstring
                       (parse-markdown docstring))))

      (40ants-doc-full/commondoc/bullet:make-bullet reference
                                                    :arglist arglist
                                                    :children children
                                                    :dislocated-symbols symbol))))

(defmethod locate-and-find-source (symbol (locative-type (eql 'variable))
                                   locative-args)
  (declare (ignore locative-args))
  (40ants-doc-full/locatives/utils::find-one-location (swank-backend:find-definitions symbol)
                                                      '("variable" "defvar" "defparameter"
                                                        "special-declaration")))

(40ants-doc/source-api:find-source (method () (40ants-doc/reference:reference)))

We have covered the basic building blocks of reference based extensions. Now let's see how the obscure define-symbol-locative-type and define-definer-for-symbol-locative-type macros work together to simplify the common task of associating definition and documentation with symbols in a certain context.

(define-symbol-locative-type direction ()
  "A direction is a symbol. (After this `M-.` on `DIRECTION LOCATIVE`
                                   works and it can also be included in DEFSECTION forms.)")

(define-definer-for-symbol-locative-type define-direction direction
  "With DEFINE-DIRECTION one can document what a symbol means when
interpreted as a direction.")

(define-direction up ()
  "UP is equivalent to a coordinate delta of (0, -1).")

Sections

40ants-doc:section objects rarely need to be dissected since 40ants-doc:defsection and 40ants-doc-full/builder:render-to-files cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a repl session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for lisp forms. The PAX transcripts may include output and return values of all forms, or only selected ones. In either case the transcript itself can be easily generated from the source code.

The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts. In fact, this is exactly what happens during documentation generation with PAX. Code sections tagged cl-transcript are retranscribed and checked for inconsistency (that is, any difference in output or return values). If the consistency check fails, an error is signalled that includes a reference to the object being documented.

Going beyond documentation, transcript consistency checks can be used for writing simple tests in a very readable form. For example:

(+ 1 2)
=> 3

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)

All in all, transcripts are a handy tool especially when combined with the Emacs support to regenerate them and with PYTHONIC-STRING-READER and its triple-quoted strings that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

Transcribing with Emacs

Typical transcript usage from within Emacs is simple: add a lisp form to a docstring or comment at any indentation level. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by #\^:

This is part of a docstring.

```cl-transcript
(values (princ :hello) (list 1 2))^
```

Note that the use of fenced code blocks with the language tag cl-transcript is only to tell PAX to perform consistency checks at documentation generation time.

Now invoke the emacs command mgl-pax-transcribe-last-expression where the cursor is and the fenced code block from the docstring becomes:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then you change the printed message to :HELLO-WORLD and add a comment to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)

When generating the documentation you get a a warning:

WARNING:
   Transcription error. Inconsistent output found.

Source:
   "HELLO"

Output:
   "HELLO-WORLD"

Form:
   "(values (princ :hello-world) (list 1 2))"

because the printed output and the first return value changed so you regenerate the documentation by marking the region of bounded by | and the cursor at ^ in the example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> (1
    ;; This value is arbitrary.
    2)
^

then invoke the emacs command 40ants-doc-retranscribe-region to get:

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and the comment of (1 2) was left alone but the output and the first return value got updated.

Alternatively, C-u 1 40ants-doc-transcribe-last-expression will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

This can be useful for producing results outside of the docstrings.

C-u 0 40ants-doc-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in 40ants-doc-full/transcribe:*syntaxes*. Without a prefix argument 40ants-doc-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level, but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

Transcription support in emacs can be enabled by adding this to your Emacs initialization file (or loading elisp/transcribe.el):

;;; Code transcription

(defun 40ants-doc-lisp-eval (form)
  (cond
   ((and (fboundp 'sly-connected-p)
         (sly-connected-p))
    (sly-eval form))
   ((and (fboundp 'slime-connected-p)
         (slime-connected-p))
    (slime-eval form))
   (t
    (error "Nor SLY, nor SLIME is connected to the Lisp."))))


(defun 40ants-doc-transcribe-last-expression ()
  "A bit like C-u C-x C-e (slime-eval-last-expression) that
inserts the output and values of the sexp before the point, this
does the same but with 40ANTS-DOC-FULL/TRANSCRIBE:TRANSCRIBE. Use a numeric prefix
argument as in index to select one of the Common Lisp
40ANTS-DOC-FULL/TRANSCRIBE:*SYNTAXES* as the SYNTAX argument to 40ANTS-DOC-FULL/TRANSCRIBE:TRANSCRIBE.
Without a prefix argument, the first syntax is used."
  (interactive)
  (insert
   (save-excursion
     (let* ((end (point))
            (start (progn (backward-sexp)
                          (move-beginning-of-line nil)
                          (point))))
       (40ants-doc-transcribe start end (40ants-doc-transcribe-syntax-arg)
                           nil nil nil)))))

(defun 40ants-doc-retranscribe-region (start end)
  "Updates the transcription in the current region (as in calling
40ANTS-DOC-FULL/TRANSCRIBE:TRANSCRIBE with :UPDATE-ONLY T). Use a numeric prefix
argument as in index to select one of the Common Lisp
40ANTS-DOC-FULL/TRANSCRIBE:*SYNTAXES* as the SYNTAX argument to 40ANTS-DOC-FULL/TRANSCRIBE:TRANSCRIBE.
Without a prefix argument, the syntax of the input will not be
changed."
  (interactive "r")
  (let ((point-at-start-p (= (point) start)))
    ;; We need to extend selection to the
    ;; beginning of line because otherwise
    ;; block's indentation might be wrong and
    ;; transcription parsing will fail
    (goto-char start)
    (move-beginning-of-line nil)
    (setf start
          (point))
    
    (let ((transcript (40ants-doc-transcribe start end
                                          (40ants-doc-transcribe-syntax-arg)
                                          t t nil)))
      (if point-at-start-p
          (save-excursion
            (goto-char start)
            (delete-region start end)
            (insert transcript))
        (save-excursion
          (goto-char start)
          (delete-region start end))
        (insert transcript)))))

(defun 40ants-doc-transcribe-syntax-arg ()
  (if current-prefix-arg
      (prefix-numeric-value current-prefix-arg)
    nil))

(defun 40ants-doc-transcribe (start end syntax update-only echo
                                    first-line-special-p)
  (let ((transcription
         (40ants-doc-lisp-eval
          `(cl:if (cl:find-package :40ants-doc-full/transcribe)
                  (uiop:symbol-call :40ants-doc-full/transcribe :transcribe-for-emacs
                                    ,(buffer-substring-no-properties start end)
                                    ',syntax ',update-only ',echo ',first-line-special-p)
                  t))))
    (if (eq transcription t)
        (error "40ANTS-DOC is not loaded.")
      transcription)))

Transcript API

(transcribe "(princ 42) " nil)
=> "(princ 42)
.. 42
=> 42
"

(values (princ 42) (list 1 2))

(values (princ 42) (list 1 2))
.. 42
=> 42
=> (1 2)

(values (princ 42) (list 1 2))
=>

(values (princ 42) (list 1 2))
=> 42
=> (1 2)

(values)
..
=>

(values)

(values)
..
=> ; No value

(list 1 2)
=> (1
      2)

(defclass some-class () ())

(defmethod print-object ((obj some-class) stream)
  (print-unreadable-object (obj stream :type t)
    (format stream \"~%~%end\")))

(make-instance 'some-class)
==> #<SOME-CLASS 
-->
--> end>

(make-instance 'some-class)
;==> #<SOME-CLASS
;-->
;--> end>

(list 1 2)
;=> (1
;->    2)

(:commented-1
 (:output ";..")
 (:no-value ";=>  No value")
 (:readable ";=>")
 (:readable-continuation ";->")
 (:unreadable ";==>")
 (:unreadable-continuation ";-->"))

TODO

• Refactor code and make a core package with only a few dependencies.

• Add warnings on UPPERCASED symbols in docstrings which aren't found in the package and can't be cross referenced.

• Support SLY and make both SLIME and SLY integrations optional.

• Add a search facility which will build an index for static file like Sphinx does.

• Separate markup parsing and result rendering code to support markups other than Markdown and HTML.

• Add a new section type to render ChangeLog.

• Support custom HTML themes.

• Generate RSS or Atom feed out of changelog items, defined with 40ants-doc/changelog:defchangelog macro.

• Make some warnings compile-time for defsection and show them in the Emacs, if possible.