1 Introduction

1.1 What is it?

MkTechDocs is an open framework for collating and transforming markdown. It can be used for managing large complex technical documentation projects with dependencies, producing light documentation, or easily and quickly combining and transforming existing Markdown files into a styled webpage or PDF document.

1.2 Features

1.3 Can I see an example?

You're looking at one! Now, check out the source in the docsbuild directory to see what a MkTechDocs project looks like. Start with master.md. That's the master document. Every MkTechDocs project has one.

1.4 Supported architectures

MkTechDocs currently supports Arch and Ubuntu (16.04+) Linux and macOS. It may work in Windows under Cygwin, MinGW, or Ubuntu Bash for Windows.

1.5 Downloading and setting up your environment

See the Download and Setting up your environment sections at the end of this document for more information.

2 The basics

There are a couple of different use cases for MkTechDocs:

  1. You want to build and maintain a complex documentation project.
  2. You want to use MkTechDoc tools to process one-off documents for previewing or ingesting into some other documentation scheme.

In either case, the best way to start is to create a new MkTechDocs project and import existing files. This way, you automatically gain access to all the complex command-lines, CSS styling, PDF templates, Pandoc filters, and so on.

2.1 The executive summary

The docsbuild project is a good example of how to use MkTechDocs to manage your documentation projects effectively. Poke around in the directory, starting with the master.md master document.

If you're impatient, follow the dependency installation instructions and then build the $MKTECHDOCSHOME/bin/docsbuild project (this website):

cd $MKTECHDOCSHOME/docsbuild
mktechdocs

2.2 Markdown

MkTechDocs uses Markdown exclusively, although support for additional syntaxes may be added in the future. More specifically, it uses standard Markdown syntax with Pandoc extensions. In this documentation, assume that any file with a .md extension is a file that contains Markdown. Markdown is easy to learn, read, and edit and is easily transformable into a myriad of output formats. To use MkTechDocs effectively, take a few minutes to familiarize yourself with Markdown syntax.

2.3 Creating a new project

To create a new MkTechDocs project, cd inside the directory where you want to create it and run mktechdocs. For example:

mkdir mynewproject
cd mynewproject
mktechdocs init

In this case, mktechdocs will create the following files in mynewproject:

File Name Purpose
footer.html* Custom HTML content MkTechDocs should place in the footer area of HTML output.
header.html* Custom HTML content MkTechDocs should place in the header area of HTML output.
landing.html* Custom HTML landing-page content MkTechDocs uses in index.html when creating multi-html-page output.
master.md A stub master document.
subdocument.md An example subdocument that's included inside the master.
mktechdocs.conf Project configuration.

* NOTE 1: Optional. If you don't need this HTML content, for example, if you're producing PDF content only, you can safely delete these files.

* NOTE 2: If you need more dynamic content in your header, footer, or main content, dates, database information, etc., please read the footer.html, header.html, and landing.html as templates section

2.4 MkTechDocs projects

2.4.1 Master document

MkTechDocs projects consist of a "master" document — a single markdown document that defines your project, always named master.md— and any number of sub documents. Building a master document is simple:

```comment
Define one or more include blocks to include all the major subdocuments that
make up your project.
```
```{.include heading-level=0}
introduction.md
getting-started.md
more-things.md
the-end.md
```

Note the heading-level=0 attribute. This tells MkTechDocs that the current heading level at the point of the include block is '0'. This will result in each included document starting with heading level of 1, or <H1> in HTML parlance. If you want to keep all your heading-level ones in the master document, that's okay, too. Simply use heading-level=1 when including subdocuments. More information on this follows.

2.4.2 Sub documents

Every sub document you include should start with a heading level of 1 (i.e. <H1>). MkTechDocs automatically increases the heading level of included documents (even multi-level recursive includes, where a document includes another document that includes another document, and so on). The heading-level adjustments will work provided that you give MkTechDocs a hint about the current heading level at the point of inclusion, as we have already seen:

### A level three header

```{.include heading-level=3}
myincludeddoc.md
```

Now, regardless of how many included documents appear in myincludedoc.md and in documents included in those documents, the heading levels should be consistent, provided you remember that all sub documents should start with a heading level of 1, as if they were a standalone document.

Tip: Creating every sub document such that it can exist on its own has the added benefit of allowing you to assemble different combinations of documents for different audiences. For example, you might want to build a small section of a much larger document as a standalone PDF.

2.4.2.1 Images

If you want to include images in your project document, include them inline using standard Markdown:

E.g.

```
Please click to [the following link](images/smiley.png) to see a smiley.
    
Or let's include it inline with a title underneath:
 
![Smiley](images/smiley.png)
```

And here's what the code above looks like after rendering:


Please refer to the following link to see a smiley.

Or let's include it inline with a title underneath:

Smiley

Smiley


Then, tell MkTechDocs where your images directory is in the configuration section of your project makefile.

3 Building your document

To build your new project from scratch with the default configuration (a single CSS-styled HTML page with a navigation sidebar pinned to the left-hand side), simply do:

mktechdocs

If no errors occur, your new index.html HTML page and CSS file should appear in ./myproject/myproject_pages.

Here is the inline help:

$ mktechdocs help
Usage: mtechdocs [help|clean|init]
  help : Display this help message
  clean: Remove temporary build files
  init : Create a new MkTechDocs project in the current directory

Usage: mtechdocs
  Builds the MkTechDocs project in the current directory. Assumes
  that a mktechdocs.conf file and master.md file are present.

Usage: mtechdocs <config> <directory>
  Builds the MkTechDocs project in the given directory using the
  given configuration.
$

3.1 Multiple configurations

Note that a project can have as many configurations as necessary. For example, if you need to produce a PDF and styled webpage:

mktechdocs config1.conf /Users/mylogin/myproject
mktechdocs config2.cong /Users/mylogin/myproject

3.2 Pre- and post-build activities

Suppose you have a set of static images as part of your documentation. You'll need to copy these to both the build and output directories in order for the images to appear inline.

Use the BUILD_SCRIPT configuration variable to provide the name of a script to run pre-build and post-build. The script should accept 3 arguments:

Argument # Description
1 The activity being performed. Possible values given by MkTechDocs are pre and post. These indicate that any pre- or post-build activies should take place.
2 The full path to the build directory.
3 The full path to the output directory.

Here's an example build script that copies a directory of images into the build and output directories when appropriate:

#!/bin/bash

ACTIVITY=$1
OUT_DIR=$2

if [[ "$1" == "pre" ]] ; then
    echo "Creating some new stuff in ./mystuff..."
elif [[ "$1" == "post" ]] ; then
    echo "Copying ./mystuff to $OUT_DIR..."
    cp -r ./mystuff $OUT_DIR/.
fi

4 Configuration

You configure MkTechDocs projects with variables contained in a configuration file in your project directory. By default, the name of this file is mktechdocs.conf, and if you choose to use this name, you won't have to provide the configuration file and directory when you run MkTechDocs.

4.1 Simple configuration

For the majority of relatively simple documents and websites, the simple configuration options should suffice.

Variable Name Possible Values Effects
TITLE Any The natural-language title of your project.
OUTPUT_FILE_NAME Any. No spaces or punctuation. Certain outputs will produce a file of this name.
FORMAT pdf, pdffloat, html, htmlsimple, cssframes, htmlmulti, markdown, docx Output format

pdf: A PDF file with section numbers and a title page. Diagrams are positioned near where they are included.
pdffloat: A PDF file with section numbers and a title page. Diagrams are "floated" within the document.
html: A single CSS-styled HTML page with a hideable table of contents at the top.
htmlsimple: A single unstyled HTML page with no table of contents.
cssframes: A single CSS-styled HTML page with a CSS-locked navigation sidebar on the left-hand side and locked content on the right-hand side. A customizable header and footer (in header.html and footer.html) are used to frame the page.
htmlmulti: Multi-page CSS-styled HTML with a CSS-locked navigation bar on the left-hand side and content on the right-hand side. A customizable header and footer (in header.html and footer.html) are used to frame the page. In addition, landing.html is used as a landing page with the CSS-locked navigation sidebar on the left-hand side. Best for large projects.
markdown: A single markdown document.
docx: A single Microsoft Word document.
HTML_STYLE archwiki, github, custom CSS styles that loosely mimic the Arch Linux Wiki and GitHub documentation. See CUSTOM_CSS for more information about applying custom CSS styles.
PDF_MAIN_FONT Any installed font name. MkTechDocs will apply this font to the main "default" text of output if FORMAT is pdf or pdffloat.
PDF_MONO_FONT Any installed font name. MkTechDocs will apply this font to any sections of the PDF document that require a fixed-width font, such as code sections. Applies only to output if FORMAT is pdf or pdffloat.
TABLE_OF_CONTENTS_MAIN_DEPTH 1-6 The maximum number heading-level depth to count as a section number in the "main" document (not sub documents in htmlmulti format). For example, if TABLE_OF_CONTENTS_MAIN_DEPTH is 2, all level-three headings and greater will not appear in the table of contents.
TABLE_OF_CONTENTS_SUB_DEPTH 1-6 The maximum number heading-level depth to count as a section number in sub documents for the htmlmulti format.
SECTION_NUMBERS true,false Include section numbers in headings and tables of content.
TITLE_PAGE Any. No spaces or puncutation. See the title pages section for more information. Applies only to pdf and pdffloat output formats.
IMAGES Path to directory relative to project. See the Images section for more information. Applies to all formats.

4.2 Advanced configuration

More complex documents may require finer-grain control over how documents are rendered and stylized.

Variable Name Possible Values Effects
BUILD_SCRIPT Any See the Pre- and post-build activities section for for information.
CUSTOM_CSS Relative path to CSS file If you provide a file path here to a CSS file, MkTechDocs will copy that file into your *_pages output directory. To understand MkTechDocs CSS, see the $MKTECHDOCSHOME/lib/*.css files.
CUSTOM_TEMPLATE Relative path to pandoc template If you provide a file path here to a template file, MkTechDocs will use that template instead of one of the defaults. To understand MkTechDocs template files, see the $MKTECHDOCSHOME/*_template.html files. In addition, please see man pandoc -> "Variables set by pandoc" for a listing of pandoc variables available to template files. For more complex use of variables, see the Python Templates section.
KEEP_TEMP_FILES true,false If true, mktechdocs will delete all temporary build files after completing a build.

5 Title pages

There are three ways to create a title page for your pdf or pdffloat document. In all three cases, you must create a standalone file and assign that filename to the TITLE_PAGE variable in your project's configuration.

5.1 Title metadata block

% My Document Title
% John Doe; Jane Doe
% January 1, 2000

For more information about this syntax: man pandoc and search for "Metadata block."

5.2 YAML metadata block

---
title: My Document Title
author:
  - John Doe
  - Jane Doe
date: January 1, 2001
---

5.3 Templates

If your title page requires dynamic content (e.g. non-static date, version number) you can create a Python template to output information as in the above two exmaples. Follow along with the example in Python templates section.

6 Useful Blocks

MkTechDocs contains built-in blocks that are useful when creating documentation.

6.1 Comment

Comment blocks are ignored by MkTechDocs.

E.g.

```comment
This text will be ignored.
```

6.2 Include

The include block was introduced in the basics section.

Parameter Description
heading-level Provides a hint to MkTechDocs about the current heading level. In order for MkTechDocs to produce nested headings, this parameter should always be used.

E.g.

# My header

```{.include heading-level=1}
myinclude.md
mysecondinclude.md
```

include blocks support infinte recursion, so included documents can contain included documents and so on.

6.3 Include-code

The include-code block is used to include source-code files with syntax highlighting.

Parameter Description
language Optional parameter that tells MkTechDocs what language the source code is. If no language is provided, MkTechDocs will use the file's extension. E.g. file.c -> language=c

E.g.

```{.include-code language=c}
/home/mylogin/MkTechDocs/docs/file1.h
/home/mylogin/MkTechDocs/docs/file1.c
```

Produces:

#ifndef SOMETHING
typedef struct foo {
	int a;
} SOMETHING;
#endif
#include <stdio.h>
#include "file1.h"

int main(int argc, char **argv)
{
	printf("Size of SOMETHING is: %lu\n", sizeof(SOMETHING));
	return 0;
}

Note: Paths are an issue here. Because MkTechDocs projects are built inside your project directory, for files to be included correctly a full path or a path relative to the project directory must be given. MkTechDocs does its best to copy image files and directories to the output directory, but you may need to create a post-build script for more complex operations.

6.4 Note

note blocks are used to create documentation "notes" that appear highlighted in documentation text for HTML output formats. For PDF output, the content of the note block is prepended with "Note:".

E.g.

```note
This text will appear inside its own highlighted block and be prepended with "Note:".
```

Produces:

Note: This text will appear inside its own highlighted block and be prepended with "Note:".

6.5 Plantuml

plantuml blocks let you include PlantUML diagramming UML directly in your documentation.

Parameter Description
title The title of the diagram. Used for a caption that appears beneath the diagram.

E.g.

# My Diagram

```{.plantuml title="My Diagram"}
A->B    
```

Produces:

My Diagram

6.6 Tip

tip blocks are used to create documentation "tips" that appear highlighted in documentation text for HTML output formats. For PDF output, the content of the tip block is prepended with "Tip:".

E.g.

```tip
This text will appear inside its own highlighted block and be prepended with "Tip:".
```

Produces:

Tip: This text will appear inside its own highlighted block and be prepended with "Tip:".

7 Document links

Pandoc automatically creates anchor names for every heading within your documents. Therefore, generating internal links is theoretically easy thanks to Pandoc's naming convention. For example, suppose you create the following header:

# My New Header

Here's how you'd normally create a link to it using Pandoc naming conventions:

Some markdown text that contains a [link](#my-new-header)

Because MkTechDocs can generate documents that require internal links that cross pages as well as documents that contain only internal relative links (as illustrated above), some thought has to be given to how links are created. When you generate internal links in your document, assume that your top-level included documents exist as HTML documents. For example, consider the following master document:

```{.include heading-level=0}
introduction.md
middle-section.md
end.md
```

To create links to other parts of your document, point your links to headings in introduction.html, middle-section.html, and end.html. For example, suppose introduction.md contained:

# Some heading

Some text

# Some other heading

Some more interesting text.

To refer to a section in this file in another file:

# Middle section

Here is a [link](introduction.html#some-heading) that points to another file.

If you follow this convention, MkTechDocs will produce correct links for every available format.

8 Python templates

MkTechDocs contains built-in support for Jinja2 templates. Using jinja2 templates is simply a matter of creating templates (plain text files containing Jinja2 markup code) and adding a .pyt extension, or, in the case of your header, landing page, and footer, a .htmlt extension, so MkTechDocs knows how to build them.

To use templates in MkTechDocs, you'll need to create both a renderer and a template to render.

8.1 The template

A template is nothing more than plain text with some special Jinja2 markup thrown in. For example, let's create a dynamic title page using templates. First, we'll create a template that represents a dynamic YAML title block in our project directory:

title-page.pyt:

---
title: My Title {{ docVersion }}
author:
    - John Doe
    - Jane Doe
date: Built on {{ currentDateTime }}
---

The Jinja2 templating engine will replace the variables in the double curly brackets with values you provide in the renderer.

8.2 The renderer

Now, in your project directory, create a renderer with the same name as the template, but replace .pyt with .renderer. MkTechDocs will automatically use the renderer associated with the template name. Renderers are built with pure Python code. Continuing with our title-page example, here is a renderer that will properly render our title page template.

title-page.renderer:

#!/usr/bin/env python

import sys
import datetime
from mktechdocslib import render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
  # Here, we hard code a document version, but in a real-world example, we'd
  # probably want to grab this from a configuration file or even a database.
  docVersion = "0.1a"

  # Now set up the date and time however you see fit
  currentDateTime = datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y")

  varDictionary = {"docVersion":docVersion, "currentDateTime":currentDateTime}

  if not os.path.isfile(sys.argv[1]):
    sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
    sys.exit(1)

  print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
  render()

You can add as many variables to your varDictionary as necessary. You can even add entire modules and functions, if you need more logic in your templates.

Next, set the TITLE_PAGE variable in your project makefile to title-page.md. Why the .md extension? MkTeckDocs automatically converts anything with a .pyt extension to markdown before building your documents. Here's a diagram that illustrates the process:

Template Rendering

Since title pages are only used in PDF documents (currently), you should also set FORMAT to pdf or pdffloat.

8.3 Referencing templates in include blocks

MkTechDocs automatically detects *.pyt templates and converts them into markdown before processing include blocks. So, if you need to include the contents of a template's output in another markdown file, simply treat the template as if it were markown. For example, if your template's name is mytemplate.pyt reference the output in include blocks like:

```include
mytemplate.md
```

8.4 footer.html, header.html, and landing.html as templates

MkTechDocs will recognize footer.htmlt, header.htmlt, and landing.htmlt as templates and process into corresponding .html files, using similarly named renderers. E.g. footer.renderer.

8.5 For more information

For real-world examples of how to use Jinja2 templates with MkTechDocs, see the following files in the MkTechDocs directory:

docsbuild/runningfilters.pyt
docsbuild/runningfilters.renderer
docsbuild/scripts.pyt
docsbuild/scripts.renderer

These templates and renderers incorporate some pythondoc-like markup in the filters and scripts that come with MkTechDocs.

Jinja2 templates can be powerful documentation tools. Please reference the Jinja2 template documentation for more information.

9 Managing document versions

For some documentation projects, managing document versions is vital. It is helpful if you define your document version in one place, so that any sub-documents that reference the version number get it from the same place.

The following is one reasonable way of doing this.

9.1 global_vars

Whether you're using Python or Groovy, it's probably a good idea to create a module (file) called global_vars.py to hold any static variables or functions you need to build your project, such as a version string. Here are the steps:

  1. Add your project's directory to PYTHONPATH.
  2. For any documents that need access to information stored in global_vars, you simply convert the document into a template (by renaming it from .md to .pyt, for example).
  3. import global_vars as a module in your renderer and include it in your variable dictionary.
  4. Now, you can access global variables and functions in glob.py in your templates that have imported it: e.g. version=global_vars.VERS.

10 Incorporating Groovy code and templates

Groovy integration is an optional step and only necessary if you want to access existing Java libraries.

10.1 Installation of groovy components

First, start by examining and then running whichever install_deps_*_groovy.sh script in $MKTECHDOCSHOME/bin/groovy is appropriate for your architecture.

They install two packages, groovy and gradle. Then, they download a Groovy library called groovy-pandoc from GitHub and install the resulting jar file in $MKTECHDOCSHOME/lib.

10.1.1 Environment

Next, add $MKTECHDOCSHOME/bin/groovy to your PATH environment variable. Here's an example in Bash:

export PATH=$PATH:$MKTECHDOCSHOME/bin/groovy

10.1.2 Testing the integration

Finally, run $MKTECHDOCSHOME/test/testgroovyintegration.sh. Your output should look something like this:

[~/MkTechDocs/test]$ testgroovyintegration.sh
Generating 'svg' plantuml diagram named 'Diagram'
Test dependency installation
============================

First, we'll try an include:

Include test file
-----------------

This text is coming from inside test\_install\_include.md.

Now, we'll try some plantUML:

![Diagram](./8410b9ad92ce7f3e2d564faff2b834b1.plantuml.svg){#myid}
Pandoc seems ok.


Checking groovy...

If this prints with no error, you are good to go
Groovy seems ok.
[~/MkTechDocs/test]$

10.2 Groovy templates

You can use Groovy templates (*.gt) as you would Jinja2 Python templates, except that Groovy templates do not require a renderer, which makes them somewhat more convenient. However, this comes at a price. Groovy templates are far slower than Jinja2 to generate, even with GroovyServ installed.

Groovy templates work like PHP or JSP files. You can include plain text markdown and Groovy clode blocks. Here's an example:

# An example Groovy template

Let's count to 10:

<%
(1..10).each { n ->
    out.println(n)
}
%>

Here's the output:

$ gtp testtemplate.gt
# An example Groovy template

Let's count to 10:

1
2
3
4
5
6
7
8
9
10


$

Notice that gtp is a standalone template renderer that MkTechDocs uses to render *.gt files.

10.3 Groovy classes

You might need standalone classes for your Groovy templates. To do this, create *.groovy files in your documentation directory. MkTechDocs will automatically detect and compile these into *.class files.

For example, MkTechDocs includes a class called MarkdownUtils, a utility class. Even though you wouldn't likely use it in a real project, the following demonstrates the concept.

test.md:

# This is a level-one header

## This is a level-two header

mytempl.gt:

# Some heading

<%
import MarkdownUtils

def f = new File("./test.md")

out << MarkdownUtils.getMarkdownFileAndChangeHeader(f, 2)
%>

And here is the output:

[~/MkTechDocs/docs]$ gtp mytempl.gt
# Some heading

### This is a level-one header

#### This is a level-two header


[~/MkTechDocs/docs]$

Note how the heading levels were increased by two levels, which is exactly what the getMarkdownFileAndChangeHeader function does.

10.4 CLASSPATH

MkTechDocs creates a CLASSPATH automatically in order to process templates. However, if you need to add jars or directories to your CLASSPATH outside of MkTechDocs, simply create an environment variable. Here is an example in Bash:

export CLASSPATH=/path/to/my.jar

Now, when MkTechDocs runs, it will build a CLASSPATH that includes any existing CLASSPATH.

11 Team documentation

MkTechDocs is ideal for incorporating team-contributed documentation. Because MkTechDocs documentation is simple Markdown, team members can contribute or edit documents via Git (for example) and let a single technical writer manage the pull requests to keep things as simple and as clear as possible.

This makes scaling documentation easier because the technical writer is no longer limited to his or her own domain knowledge or learned knowledge.

That's a simple case. The following describes a different scenario that lets agile developers contribute to documentation in a more specific and controlled way.

11.1 Case study

In this study, a number of different agile development teams are responsible for individual components of a larger system. A configuration repository maintained by the larger system consists of a number of YAML files that describe the general deployment of each component.

Here, components could be Docker containers, VMs, micro-services, or whatever. Each component could have a different but similar set of deployment instructions for each category below, or it could simply conform to some generic set of instructions and require no extra documentation.

  1. Overview
  2. Environmental-setup
  3. Pre-deployment
  4. Deployment
  5. Undeployment
  6. Upgrading
  7. Jenkins integration

Agile team members in this scenario are encouraged to contribute documentation in support of their respective components with those steps in mind. If they feel that specialized instructions for any of the above areas applies to their component, they should create a file in a provided directory within the documention tree that matches their component name in the configuration YAML. This file should contain a standalone Markdown document containing instructions specific for their component for that particular step. E.g. overview/mycomponent.md, upgrading/mycomponent.md.

For example, suppose a VM named vm-inventory exists in some YAML configuration file:

location:
  description: >
    This is a sample description of a particular location
  vm-deployments:
         .
         .
         .
    vm-inventory:
      vm-config: vmi.cfg
      release: 1.234
      notes: Some notes about this particular VM
        .
        .
        .

Now, we can create a template that loads the YAML containing the list of components, and looks for files with the same name in the directories listed above. If we find specialized instructions, we output them for a particular stage (e.g. Overview, Undeployment). If we find no specialized instructions, we output some generic ones.

11.2 Managing git commit messages

Often, it is useful for your document's audience to understand what has changed from version to version. Rather than keeping track of this manually, in a "change log," for example, you can leverage Git's commit system.

The git log command shows a list of all commits to a respository. Each commit has a unique ID associated with it (SHA1 collisions not withstanding). This means that you can create "versions" by recording specific commit messages that represent the beginning and ending of a document version, pulling out all relevant commit messages dynamically when the document is built.

MkTechDocs includes this functionality in the mktechdocslib.py module. Here's an example of how to use it.

11.2.1 GitCommitDB

What we'd like to do is create a simple database of Git messages that exist between two arbitrary Git commit IDs. First, we examine the output of git log to determine the first and last commit IDs that frame the version of the document we're interested in creating. Note that the output of git log is ordered by commit date, so the most recent dates appear first.

Here is a sample Git log:

commit 20de5c8bf2a53efe54a71ae06e5b0b2a813d5176
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Tue Mar 7 08:04:25 2017 -0500

    Fixed several formatting issues

commit 3523b7f36d928519bba9568852babd793e783a7c
Merge: cd6c4bc f9b3edf
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Tue Mar 7 06:50:53 2017 -0600

    Merge pull request #71 in FOO_BAR/foo.bar.documentation from feature/FOOBAR/FOOBAR to master

    * commit 'f9b3edfb5194a2be0456d26d742043677948d5ea':
      Added some notes about documentation formats

commit f9b3edfb5194a2be0456d26d742043677948d5ea
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Mon Mar 6 16:21:17 2017 -0500

    Added some notes about documentation formats

From this, we determine that the first commit ID is f9b3edfb5194a2be0456d26d742043677948d5ea and the last commit ID is 20de5c8bf2a53efe54a71ae06e5b0b2a813d5176.

Now, in a template.renderer, we can do something like this:

#!/usr/bin/env python

import os
import sys
import datetime
import global_vars
from mktechdocslib import GitCommitDB, render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
    messages = GitCommitDB("f9b3edfb5194a2be0456d26d742043677948d5ea", "20de5c8bf2a53efe54a71ae06e5b0b2a813d5176").messages
    varDictionary = {"gitMessages":messages,
                     "numCommits":len(messages)}
    
    if not os.path.isfile(sys.argv[1]):
        sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
        sys.exit(1)

    print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
    render()

Here, we import GitCommitDB from mktechdocslib and add the GitCommitDB.messages list object to our variable dictionary. messages contains a list of GitCommitMessage objects, which have the following properties:

Property Type Description
id String Commit ID
mergeID String Merge commit ID
author String Author of the commit
date String Commit date
message String Commit message
mergeCommit boolean True if the commit is a merge

Finally, here's our template that displays the information we're interested in:

# Document change log

{% if numCommits == 0 %}
Looks like no changes have been committed.
{% endif %}

{% for m in gitMessages %}

{% if not m.mergeCommit %}
**Date**: {{ m.date }}<br />
**Author**: {{ m.author }}<br />
**Message**: {{ m.message }}<br />
{% endif %}

{% endfor %}

The version in the example above was hard coded. In reality, it would be better to keep track of version information in a list of tuples. For example:

versions.py:

gVersions = [("1.0", "2017-04-06", "f9b3edfb5194a2be0456d26d742043677948d5ea", "20de5c8bf2a53efe54a71ae06e5b0b2a813d5176"),
             ("0.9", "2017-03-25", "d453519c024a49806fbd3a6740a9f0f586aaafbc", "57831131f186a60b864af27e53e06d2772a6d1ef")]

In this scheme, the latest version tuple would always be gVersions[0]. Or, if you prefer to store the versions in reverse order: gVersions[len(gVersions)-1]. Then, our renderer would look like this:

#!/usr/bin/env python

import os
import sys
import datetime
import global_vars
from versions import gVersions
from mktechdocslib import GitCommitDB, render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
    (versString, date, fcid, lcid) = gVersions[0]
    
    messages = GitCommitDB(fcid, lcid).messages
    
    varDictionary = {"gitMessages":messages,
                     "numCommits":len(messages),
                     "versString":versString,
                     "date":date}
    
    if not os.path.isfile(sys.argv[1]):
        sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
        sys.exit(1)

    print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
    render()

12 Download

To download MkTechDocs, clone the repo:

git clone https://github.com/att/MkTechDocs

13 Setting up your environment

Please export the following variables. Here is an example in Bash:

# MkTechDocs libraries and scripts require the following variable:
export MKTECHDOCSHOME=/path/to/cloned/repo/MkTechDocs

# Make sure the MkTechDocs bin directory is in your path:
export PATH=$PATH:$MKTECHDOCSHOME/bin

# Let Python know where to find MkTechDocs libraries:
export PYTHONPATH=$MKTECHDOCSHOME/bin

13.1 Running the dependencies installation script

Install MkTechDocs dependencies using the (admittedly imperfect) installation script. Consider the script more of a guide.

Examine the contents of the $MKTECHDOCSHOME/bin/install_deps*sh script appropriate for your architecture (Ubuntu, Arch, Fedora, and macOS are currently supported). For Groovy support, please also see the install_deps*sh scripts in the bin/groovy directory. The install script simply tries to install the various dependencies using an architecture-appropriate package manager and in some cases checks version numbers of installed binaries. You can also install the dependencies manually.

13.1.1 Dependencies

MkTechDocs requires the following:

Package Version Description
Bash any recent A shell.
Git any recent A distributed version-control system.
Pandoc >= 1.18 Pandoc is a powerful document conversion tool and is at the heart of MkTechDocs.
Make any recent A tool to help manage document dependencies during the build process.
Graphviz any recent A collection of open-source tools for drawing graphs in the DOT language.
Plantuml any recent A tool for creating UML diagrams that uses Graphviz underneath.
XeTeX any recent A suite of tools for building PDF documents.
Python 2.7 MkTechDocs is currently compatible with Python 2.7.
Jinja2 2+ A Python templating library.
homebrew** any recent A package manager for macOS.

** homebew: macOS only

Groovy/Java*

Package Version Description
Java* >= 1.8 MkTechDocs is currently compatible with Java 1.8
Groovy* > 2.0 A Java-like scripting language that extends Java.
groovy-pandoc* >= 0.8 A Groovy library used to build Pandoc filters for document transformation.
Gradle* > 3.0 A build automation system that is friendly to the Groovy language.

*: Java/Groovy dependencies are optional but ideal for interfacing MkTechDocs with existing Java libraries. Note that Groovy pandoc filters are orders of magnitude slower than their Python counterparts. To shorten build time, you might want to try GroovyServ.

13.2 Testing your environment

After installing the dependencies, run this:

cd $MKTECHDOCSHOME/test
./testinstall.sh

You should see something like this:

[~/MkTechDocs/test]$ ./testinstall.sh
Created image plantuml-c6117aebb4ce8e59ba2b46950eca869f.svg
Test dependency installation
============================

First, we'll try an include:

Include test file
-----------------

This text is coming from inside test\_install\_include.md.

Now, we'll try some plantUML:

![](plantuml-c6117aebb4ce8e59ba2b46950eca869f.svg)
Pandoc seems ok.


Checking python...

If this prints with no error, you are good to go!
Python seems ok.
[~/MkTechDocs/test]$

If you get errors, examine the output of the installation script to see where something went wrong.

13.3 A note about titlesec

MkTechDocs uses a TeX package named titlesec to do various things with PDF section numbers (e.g. 6.1.3.2). Unfortunately, Ubuntu 16.04 LTS ships with a buggy version of this package, which causes section numbers to be empty in various situations, so replacing it will be necessary if you plan to use section numbering in PDF documents. To replace it, first download the latest titlesec from https://www.ctan.org/tex-archive/macros/latex/contrib/titlesec.

Now:

unzip titlesec.zip
sudo rm -rf /usr/share/texlive/texmf-dist/tex/latex/titlesec
sudo mkdir /usr/share/texlive/texmf-dist/tex/latex/titlesec
sudo cp -r titlesec/* /usr/share/texlive/texmf-dist/tex/latex/titlesec/.

Section numbering should work as expected.

14 Running filters

Running MkTechDocs Pandoc filters outside of the normal build environment is easy if you've set up your environment properly:

pandoc --filter <flt-filtername.py> -f markdown -t some-other-format inputfile.md >outputfile.fmt

In some cases, MkTechDocs filters produce artifacts on STDERR. In these cases use something like the following:

pandoc --filter <flt-filtername.py> -f markdown -t some-other-format inputfile.md >outputfile.fmt 2>artifacts.out

The following Pandoc python filters come with MkTechDocs. Groovy versions are available as well in $MKTECHDOCSHOME/bin/groovy. They are run in the same way (but have a .groovy extension).

14.1 flt-comment-block.py

Adapted from https://github.com/jgm/pandocfilters/blob/master/examples/comments.py

Pandoc filter that causes everything between '<!-- BEGIN COMMENT -->' and '<!-- END COMMENT -->' to be ignored. The comment lines must appear on lines by themselves, with blank lines surrounding them.

E.g.

 <!-- BEGIN COMMENT -->
 
 This text will be ignored.
 
 <!-- END COMMENT -->

14.2 flt-comment.py

Ignores all text contained in the comment block. E.g.

 ```comment
 This text will be ignored.
 ````

14.3 flt-decrement-header-1.py

Decrements all headers by 1 level

14.4 flt-decrement-header-2.py

Decrements all headers by 2 levels

14.5 flt-decrement-header-3.py

Decrements all headers by 3 levels

14.6 flt-decrement-header-4.py

Decrements all headers by 4 levels

14.7 flt-decrement-header-5.py

Decrements all headers by 5 levels

14.8 flt-get-includes.py

Outputs all files referenced in any include blocks in a file to stderr. Used internally by MkTechDocs to build dependency lists.

14.9 flt-include-code.py

This filter allows you to include source-code files in your markdown.

You can provide one file per line. If you include the "language" property:

 ```{.include-code language="java"}
 myfile.java
 myfile.sh
 ```

The filter will use whatever language you specify in the property regardless of what the file extensions are, which you may or may not want.

If you provide something like the following:

 ```include-code
 myfile.java
 myfile.bash
 myfile.c
 ```

The filter will produce 3 code blocks with the language property automatically set based on the extension of the files within.

14.10 flt-include-doc-map.py

This filter is identical to the include filter, except that while building a document, it outputs a document map on stderr so that a script can figure out where each part of the document came from. E.g.

     ```include
     includethisfile.md
     ```

This filter is recursive, so you markdown can include other markdown to any level.

14.11 flt-include.py

This filter allows you to include other markdown in your markdown. E.g.

     ```include
     includethisfile.md
     ```

This filter is recursive, so your markdown can include other markdown to any level.

Header levels are automatically adjusted to the correct heading level if you use the "heading-level" attribute:

 # Heading One

 Some text

 ## Heading Two

 ```{.include heading-level=2}
 somefiletoinclude.md
 ````

"heading-level" represents the heading level where the include happens. On subsequent inclusion, the heading levels in the included file will be increased by two. Of course, you can always leave out the heading-level if you don't want to alter the heading level of the included file.

Even multiple levels of include recursion will result in headers of the correct level. The reason this works is by virtue of the dependency lists generated by the makefile.

The lowest level pages (i.e. the last file included in a recursive string of includes), are processed and stored to disk first.

When the next outer level is processed, those headers are adjusted and the previously adjusted levels in the included file are in effect adjusted again. Then, the next level is adjusted, and so are all the inclusions.

This results in the innermost include's headers being adjusted upwards by however many levels of recursion there are.

14.12 flt-increment-header-1.py

Increments all headers by 1 level to a maximum of 6 levels.

14.13 flt-increment-header-2.py

Increments all headers by 2 levels to a maximum of 6 levels.

14.14 flt-increment-header-3.py

Increments all headers by 3 levels to a maximum of 6 levels.

14.15 flt-increment-header-4.py

Increments all headers by 4 levels to a maximum of 6 levels.

14.16 flt-increment-header-5.py

Increments all headers by 5 levels to a maximum of 6 levels.

14.17 flt-notetip.py

This filter replaces 'note' and 'tip' blocks with div tags of either a note or tip class

E.g.

 ```tip
 My tip
 ```

Becomes:

 <div class='tip'>**Tip**: My tip</div>

for markdown and html. For other output formats:

TIP: My tip

14.18 flt-plantuml.py

Adapted from: https://github.com/jgm/pandocfilters/blob/master/examples/plantuml.py

Turns PlantUML, e.g.:

 ```plantuml
 actor Foo1
 boundary Foo2
 control Foo3
 entity Foo4
 database Foo5
 Foo1 -> Foo2 : To boundary
 Foo1 -> Foo3 : To control
 Foo1 -> Foo4 : To entity
 Foo1 -> Foo5 : To database
 ````

Into an image for inclusion in HTML documents.

To affect the output format, provide a metadata variable on the command line:

 pandoc -M umlformat=[eps,svg] . . .

If no umlformat is indicated, the filter will default to svg.

Needs plantuml.jar from http://plantuml.com/.

14.19 flt-strip-pages-from-links.py

This filter turns links like this:

 [foo link](somepage.html#some-heading)

into:

 [foo link](#some-heading)

This is necessary to transform documents from multi-page to single page.

14.20 flt-template.py

A starting point for building new filters.

15 Scripts

Several potentially useful scripts come with MkTechDocs.

15.1 check_build_env.sh

This script does a simplistic check on your build environment to make sure that a MkTechDocs project can build.

Example usage check_build_env.sh
Arguments None
Special exit values 1 Problem with build environment
0 No problems with build environment

15.2 dec-headers.sh

A utility script that decrements the headers in a given markdown file by a given number of levels.

A copy of the original file is kept using the origial file name and a PID as the extension.

Example usage dec-headers.sh file.md 1
Arguments 1 A file to alter.
2 The number of heading levels to decrement.
Special exit values None

15.3 escape-jinja-brackets.sh

This script escapes opening Jinja2 template brackets, {# to avoid internal build problems. Generally, you won't need to run this script directly.

Example usage escape-jinja-brackets.sh file.md
Arguments 1 The file to alter
Special exit values None

15.4 inc-headers.sh

A utility script that increments the headers in a given markdown file by a given number of levels.

A copy of the original file is kept using the origial file name and a PID as the extension.

Example usage inc-headers.sh file.md 1
Arguments 1 A file to alter.
2 The number of heading levels to increment.
Special exit values None

15.5 install_deps_1604+_py.sh

This script will install all MkTechDocs dependencies in an Ubuntu 16.04 environment. It is not currently suited for using in a docker container or vagrantfile, although it would be easy enough to adapt it for such use.

Example usage install_deps_1604+_py.sh
Arguments None
Special exit values None

15.6 install_deps_arch_py.sh

This script will install all MkTechDocs dependencies in an Arch environment. It is not currently suited for using in a docker container or vagrantfile, although it would be easy enough to adapt it for such use.

Example usage install_deps_arch_py.sh
Arguments None
Special exit values None

15.7 install_deps_macos_py.sh

This script will install all MkTechDocs dependencies in a macOS environment. It is not currently suited for using in a docker container or vagrantfile, although it would be easy enough to adapt it for such use.

Example usage install_deps_macos_py.sh
Arguments None
Special exit values None

15.8 mktechdocs

16 Addendum

16.1 Understanding the build process

The MkTechDocs build process is complex. The following is a step-by-step guide towards understanding how it works.

  1. MkTechDocs consists of a BASH control script and a library of Python and Groovy filters.
  2. The control script is responsible for importing the project configuration and calling pandoc with the correct arguments to build reasonably formatted documents in a variety of formats.
  3. The "magic" behind it all is pandoc 's Abstract Syntax Tree (AST). Internally, pandoc converts all documents into AST, which can then be converted to JSON, which can then be manipulated by filters.
  4. MkTechDocs exploits this with a couple of libraries, GroovyPandoc, for Groovy template integration, and PandocFilters, for Python integration.
  5. Every time you run mktechdocs, the control script calls pandoc on your project's master document. Some Bash scripting then takes care of some little details, which you can see for yourself by examining the script.

16.2 Contact

MkTechDocs was built and is maintained by Spencer Seidel.

17 Document change log

Date: Tue Apr 24 15:58:05 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Updated version to 1.0 and reset document change log [doc]

Date: Tue Apr 24 15:13:25 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Changed build system to remove makefile and updated documentation [doc]