salesforce Metro Documentation

Metro is a platform migration tool which enables you to bring documentation from several formats into Atlassian Confluence.

Overview

Metro is a system for reducing friction in the process of publishing long-lived technical documentation at Salesforce. Confluence, our publishing environment, ships without any reasonable way to import content from other sources: as it is, you can import files directly in its cute XHTML pidgin tongue, or you can hamfistedly shove a PDF or a Word document into it. Great job onboarding your users, Confluence!

You’d think that third-party solutions would step up and solve this problem. Unfortunately, the third-party Confluence ecosystem seems a little moribund these days. Many products haven’t been touched in years, and those that are maintained often don’t seem like they’ve been thought through properly.

Metro solves this problem by choosing an intermediate representation for all of the docs we want to publish, and providing tools to

• convert our docs to that intermediate representation
• import that representation into Confluence directly and systematically

Metro can interpret information in a JSON manifest file to decide how a given page or set of pages is published.

How to Install Metro

This section describes how to set up metro. It’s pretty simple:

1. Clone the metro source<https://github.com/salesforce/metro> from GitHub.
2. Follow the instructions for your source content of choice (google docs, quip, etc) to identify the docs you want to port. For Google docs, you will need to set up a Google Script project. See Converting a Google Doc<https://metro.readthedocs.io/en/latest/using/convert-google-doc.html> for more information.

That’s it. When you are ready to do an import, just run the import_pages file.

System Requirements

Metro is implemented as a Python module (metro) and a few utility programs. To use it, you need Python 3 and the following modules:

• keyring - A portable keyring for storing passwords
• requests - A module for making HTTP/REST requests
• markdown - Support for converting Markdown to (X)HTML
• bleach - An HTML sanitizer module

If you have a linux or mac, you probably have python already. You can grab the extra modules using the pyton pip or pip3 installers. (This is all standard python functionality. If you are new to python, check out the Beginner’s Guide on the python wiki.

Installing Trusted SSL/TLS Certificates

Note

Currently, metro doesn’t validate certificates, so you can ignore this section. Just make sure you are connected via VPN when you use metro.

Because metro uses Confluence’s REST API to connect to our internal Confluence servers, you must install Salesforce’s internal Confluence TLS certificates on your machine before you use it. This enables us to verify that connections to the Confluence server aren’t intercepted.

Installing these certificates is so tricky under some circumstances that one of our own has written a Python module that appends the certificates to those shipped with Python. Arguably not the most elegant approach, but it’s pretty foolproof.

If that’s not your style, you can follow the official instructions for installing the Salesforce Confluence TLS root certificates to your machine.

Supported Document Types

Metro can convert and import the following document types to Confluence:

• Markdown (our intermediate representation)
• Google Docs
• Quip
• Jekyll (in work)

Want to help provide support for other platforms? Start forking!

Components

Importing a document into Confluence involves two Metro components:

1. A converter for getting the document into the intermediate representation.
2. An importer for getting the intermediate representation into Confluence.

Metro has a converter for each type of document supported. type of document you want to convert. All conversion tools produce a zip file with a JSON manifest, manifest.json, at the top level of the zipped folder tree. This manifest contains information about any pages, images, and attachments that have been converted, and how they relate to each other.

There’s only one importer: import_pages. This Python script takes a manifest and a set of files (or a zip file containing the manifest and those files) and imports the content into Confluence. The manifest is a JSON file that tells import_pages where to place the imported content, what images and attachments to upload, and so on.

Preparing to Import a Markdown Document

1. Look over your document and make sure it’s clean. If you see oddities in converted content, you can help out metro by using some unambiguous standards for Markdown in your input:

   • Use _ instead of * for italic text.
   • Use + instead of * for unordered lists.

2. Place your document file into a folder.

3. In the same directory as that folder, create a manifest.json file with contents like the following:

   manifest.json

   {
     "pages": [
       {
         "folder": "doc_folder",
         "file": "doc.md",
         "parent_id": 75199145,
         "operation": "create",
         "overwrite": true,
         "images": [],
         "attachments": []
       }
     ]
   }
   

   Here, we’ve assumed you created a file called doc.md, and that you want to place it under the Confluence page with the ID 7519945.

4. Metro tries to guess the title of your document, using the first heading it finds. This title becomes the title of the imported Confluence page. If you want more control over the Confluence page title, you can include a title field in the page entry. This field overrides any title metro finds.

5. If the file has images and/or contents you’d like to attach to your Confluence page, place each image and attachment into your folder (doc_folder in this case) and add their filenames to the images and attachments fields.

6. If you want to add child pages to any page, add a field called children to that page’s entry. This field is a JSON list of page entries just like the top-level list in the manifest.

7. For longer documents, you may want to add a table of contents at the top. You can do this by adding a table_of_contents field to your page’s entry in manifest.json. This field should be a JSON object with any parameters you want to set in the Confluence table of contents macro. See this page for details about the macro. Use lower-case text with underscores instead of camel case for the parameter and values (e.g. max_level instead of maxLevel).

8. If you want to maintain the page content in Google Docs, Quip, or somewhere else outside of Confluence, you can add a link to a working draft of the doc in a auto_gen field in your page’s entry. This adds a badge to the top of the converted Confluence page with a link to the working draft so you can collect questions and comments in one place.

Now your document is ready to be imported.

Metro-Specific Extensions to Markdown

Because Markdown is so spartan in terms of markup features, and because Confluence has some nifty macros we like to use, we have added some metro-specific stuff to Markdown.

• Info

  • Use > Info: text to place text into a grey Confluence Info block.
  • Alternatively, use ~?text?~.

• Notes

  • Use > Note: text to place text into a yellow Confluence Note block.
  • Alternatively, use ~!text!~.

• Warnings

  • Use > Warning: text to place text into a red Confluence Warning block.
  • Alternatively, use ~%text%~.

• Panels

  • Use > Panel: title to create a Confluence Panel with the given title. Use > text on subsequent lines to put text into the body of the Panel. The Panel is closed with the first line not beginning with >.

• Collapsible content

  • Use > Expand: title to create a Confluence Expand

    macro with the given title. Use > *text* on subsequent lines to put text into the collapsible portion. The Expand environment is closed with the first line not beginning with >.

  • The title must be text and not a link. You click the title to expand the collapsed content, not to go somewhere.

    Note

    Currently, Metro doesn’t support the use of nested Expand macros.

  • If you want a button that expands all Expand macros on your page, you can add one with > ExpandAll.

• Other HTMLisms (as allowed by standard Markdown):

  • HTML tables
  • HTML anchors. Use <a id=”anchorName”></a> to make one, and make sure you use the Confluence Anchor syntax to link to it (it’s more complicated than you might think).
  • HTML images, links, etc

Let us know if you have problems with anything that you’d expect to work.

Converting a Google Doc

The Google Doc converter is a spreadsheet-based user interface for selecting Google documents with some simple metadata. To use it:

1. Create a google sheet that has the format depicted in this figure:

   

2. Create a google script project and attach it to the google sheet.

3. Include the content from the code.gs file (in this repo root) as the code for your script project.

4. Add to the script project a new html source file, download.html. For its content, add the html contained in the download.html file (in this repo root). Save the project and sheet.

5. In each row of the spreadsheet:

• In the Resource column, fill in the Google Doc name of a document or a folder containing documents that you want to publish.
• In the File or Folder column, select File or Folder, depending on the type of the resource you specified.
• In the Confluence Parent Page ID column, specify the [page ID](https://confluence.atlassian.com/confkb/how-to-get-confluence-page-id-648380445.html) of the Confluence page under which this resource is to be published.
• In the Working Draft column, check the box if you want a badge at the top of your published Confluence page that links to your document in Google Drive.
• Finally, in the Table of Contents column, select the appropriate option. The TOC will be at the top of the finished page. If you want no TOC, select None.

6. Delete any extra rows that contain only partial information.
7. Now check the box in the first column of each resource you’d like to publish.
8. Editing the spreadsheet should reveal the Metro menu at the top of the sheet. Click it and select Show Selected Files to bring up a dialog box of the documents you’re publishing.
9. In the Metro menu, select Convert to Markdown to convert the selected resources to Markdown. The conversion process can take a while, depending on the number and complexity of the documents you’re converting. The process produces a .zip file that you can download by clicking the link in the resulting dialogue.
10. After downloading the zip file, you can unzip it to inspect or modify its contents if you like. When you’re ready to publish it to confluence, you can use the import_pages script with the -z option. You don’t need to supply a parent page ID, since you’ve already specified one for each page in the spreadsheet.

Using zip files (e.g. with Google Docs)

Another easy way to import stuff into Confluence is to hand it a zip file with converted documents and specify a parent page under which the documents will live. Each of these documents becomes a Confluence page under the parent page.

./import_pages -z converted_stuff.zip -p <parent_page_id> -u <username>

You only need to provide a parent page ID if your manifest includes pages that don’t have parent_id fields. If you specify a parent page with -p, the parent_id field in the manifest overrides the parent page you provide with this option.

To get a Confluence page ID, click … on that page’s menu bar and select Page Information. The page ID is the last portion of the URL for the resulting page.

You can do a lot more with the importer. For details, get usage information from import_pages with

./import_pages --help

Converting a Quip Document

Converting a Quip document to a Confluence page is easy, because you can export Quip text as Markdown.

1. Select Document->Export->Markdown. This copies Markdown text to your clipboard.
2. Paste the text to your favorite unicode-safe editor and save it to a file with an .md suffix.
3. If the Quip document has images and/or contents you’d like to attach to the page, download each image and attachment from Quip.
4. Now just follow the rules for Preparing to Import a Markdown Document.

Importing Converted Content

To import a set of files associated with a manifest, use the following command:

` ./import_pages -m /path/to/manifest.json -u <username> ` where <username> is the username for your Confluence account. If you’re importing a page to our internal Confluence production server, this is just your SSO identity (just your username, not your email address).

This finds the files associated with the manifest (assuming they are in the same folder) and imports them as child pages of their respective parents.

Frequently Asked Questions

1. Does `import_pages` update pages with no changes?

   Nope! If a page’s content hasn’t changed since the last import, no update is performed. The same goes for images and attachments: they are only updated if they’ve changed. Metro accomplishes this by computing MD5 checksums for new content and storing them as metadata on Confluence for comparison during updates.

2. What kinds of markup does Metro support?

   Anything in basic Markdown is fair game:

   • Headings (1-6)
   • Bold, _italic_, and monospace text
   • Ordered and unordered lists (try to use + instead of *)
   • Code blocks
   • Links
   • Inlined images (in HTML, for simplicity)
   • Tables (in HTML, to preserve image attributes)

3. If a page entry in the manifest contains a `parent_id` field and I specify a `parent_id` on the command line, which parent ID “wins”?

   A parent ID specified in a manifest always overrides a parent ID specified from the command line.

Google Docs

1. How do I make a “code block” in a Google Doc?

   Make a single-cell (1x1) table and use Courier New for the text inside. That table will be converted to a code block.

2. Some of the headings in the Confluence page for my Google Doc are… off. What’s going on?

   Check the type of headings you used. Metro doesn’t pay attention to font sizes, so if you sized up a smaller heading, that won’t translate.

Resources

• Doxygen to Markdown converter
• Google Sites to Markdown converter
• Markdown to Confluence converter

Salesforce Metro Open Source License

Copyright (c) 2019, Salesforce.com, Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of Salesforce.com nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

BSD 3-Clause License: https://opensource.org/licenses/BSD-3-Clause