Documentation

I love documentation. But also, documentation is the worst. Making documentation is hard. Making great documentation even harder. But it is instrumental to helping your code be useful to people that aren’t you, and that’s a worthwhile pursuit! Even if you’re making things just for yourself, the people that aren't you descriptor also includes you in a few years/months/weeks, who has no idea what you were even thinking.

These are the notes that I’m making while reworking the documentation for semantic.works.

Design principles

1. Don’t Repeat Yourself. If we’re copying documentation across places, we’re doing something wrong.
2. Good. The Divio documentation principles should apply.
3. Ease of use: writers. Minimising maintenance for the people who make the code is essential. Having them sign in to a whole different platform to write the documentation for their code would be way too much friction.
4. Ease of use: readers. We have a bunch of projects going on, written in Ruby, Lisp, Elixer, JavaScript, Python… The documentations should resemble eachother and be consistent.
5. Micro-first. Semantic.works is about microservices, so that means there’ll be a bunch of documentation which is small. And that’s okay! But it is important to not build the base structure to accommodate big projects. A documentation folder for something that can be explained in one README is a no-go. Start from a single file, and expand only if necessary.
6. Non-proprietary. It should be able to move, not be bloated. No editor lock-in, no host lock-in.

Sidequest: automated generation

There are a doc generators! JSDoc is my favourite just for its syntax alone.
But ruh roh, did you read the first two letters of that name? A lock-in to a programming language. Yes, we could find a documentation generator for every language we use, but then we will semi-break 3., but mostly full break 4.. Unless we morph every of those generators to the same output. Also breaking 4..

How about the general purpose ones?
Yes! There are two really cool options. First was Doxygen: a seemingly cool tool with a bunch of supported languages! But a few of our common ones are missing, so I feel like a fool.

Then Dexy. You can see the remains of my tests with it (including Dockerising it) in the files listed here. But to give you the gist: it is a super extendible and really cool amalgemation of text processing, bundled into a pretty easily configurable yaml. Jinja, pandoc and markdown are but a few of the impressive list of built-in tools. Sadly, while it can run a bunch of languages and grab the output, it seems to lack the ability to dissect the code you give it in any meaningful ways. This bundled with a website that is split across domains and a lack of updates in the last few years, make it sub-ideal for what I’m trying to do.

Dexy files

• dexy.yaml
• Dockerfile
• foo.md.example
• iterate.md.example

Updating existing documentation

Well first I had to know which repos to update! So of course I made an script for it.

1

python3 diviocheck.py

(You can run it locally! It has no dependencies except Python3 itself and its native libraries)

Returned the following:

Sidequest: documentation structuring

So. Cat from a bit in the future. I’ve started importing blog posts from mu.semte.ch, because (ignoring the bias that I work for them) they’re pretty darn good at explaining the concepts.

Challenges

I: bloat

Now look at the README in this commit and despair.
Something is awry. This is too big.

II: ordering

The README from above also includes another goofy problem: the order of information. Having a singular README is consise in terms of writing, but the references explanations on the bottom of the document might be good reading material before starting. Splitting up this README into multiple would perhaps be a bit asinine: its a project template, it shouldn’t be filled with docs/ folders. But it should also easily link to all this info, right?

#### III: order in the ordering
But we can’t just move the docs to wherever. Using readthedocs crossed
Now there’s also an interesting conundrum. What about (i forgot what i was writing about)

Solutions

This section will refer to

• Documentation that should be kept as close as possible) as (essential) documentation. This is the documentation that would be required to have even for people familiar with the project. How to start up the application, which license…
• Documentation that could be expected in a seperate file or link as extra documentation.

By seperating these two, we can more easily interlink between documentation. mu-semtech is built on [reactive-programming](/mu-semtech/reactive-programming), or this tutorial

Readthedocs

Pros:

• It seems to support markdown alongside .rst
• It’s a familiar layout for most developers

Cons:

• Reliance on an external service
• Requires permissions on the user/org
• Some external configuration needed (.yaml file, website)
• Not easily adaptable

Documentation in project repo

Documentation in a dedicated repo

We could also have a docs repo, where all extra documentation is kept.

Pros:

• Easily moveable
• Easily adaptable

Cons:

• More friction in upkeep

But, what if the friction could be removed?

Note: the rest of this section has been moved to its own repo oh no.

Rocketchat - Matrix bridge

blog announcement - docs

Moving platforms sucks. Because there’s no good way to do it. People are used to the workflows and UI’s of the previous thing, and may not like the new thing. But using two different but functionally similar platforms is how you get miscommunications. This bridge might allow a slow transition! I wanna test if this perhaps could even import previous conversations. If you’re reading this and it hasn’t been strikethroughed, I haven’t yet. But either way, it’s worth checking out.

Update: I have found things

See denperidge-redpencil/rockatrix to test it out for yourself!

Matterbridge

As of writing, I have succesfully been able to integrate this.
I’ve screen recorded the footage below, but I’ll summarise my findings here.

tl;dr: no message history, new messages & message edits get synced with barely any delay and only a few issues/quirks.

• It works by making a gateway between two specific channels/rooms, and will send new messages to the other platform near immmediately. It will show a note of the source protocol as well as the username of who sent it, and of course the message contents.
• Attachments get synced as well.
• Message edits in matrix get synced to rocketchat, but rocketchat not to matrix. See the docs for more information.
• Formatting generally gets copied over well. Some issues with # headings, but bold and italics seem to go well both ways.
• Besides making a bot user on both Rocket.chat & Matrix, there is no setup needed within the services in question.
• It needs to be actively run, and does not import the message history.
• Threads don’t work but also don’t not work. The messages still get sent, but not put in a thread on the other platform (they will just appear below).
• You can seemingly set up as many channel connections as wanted.
• The matrix & rocket.chat channel are allowed to have different names (cool), but need to be explicitly defined (less cool). There doesn’t seem to be a way to say “if something is sent in any channel, send it to the channel with the same name on the other platform”.

Link for if the above footage doesn’t embed - Uncompressed footage - First footage recording

Official bridge

As of writing, I have not succesfully integrated this yet. But I have tried. A lot. Automated setup’s Traefik kept getting permission denied, manual setup didn’t give errors but also no results.

Rocketchat supports matrix but with an asterisk

• Some federated functionality is locked behind enterprise editon and/or not encryptable
• The Rocket.chat settings say the following: No user should connect to the homeserver with third party clients, only Rocket.Chat. I don’t know if this refers to the people wanting to send messages through rocket.chat itself to Matrix users, or if they mean that all Matrix traffic should go through Rocket.chat.
  • The docs say We strongly recommend not connecting to this Matrix homeserver using other Matrix clients, if you want to do that, please configure another Matrix homeserver instance..
• When enabing federated functionality:

  1 2	From now on, you can invite federated users only to private rooms or discussions. Those channels are going to be replicated to the remote server, without the message history.

Things to check out

Score spec

repo - website

Score is essentially a container/workload configuration tool that translates into {Docker, Kubernetes, custom stuff}. Switching to Score would possibly be very time consuming and not worth the effort, but it is good to keep this jotted down somewhere; in case other container technologies are to be implemented or Docker begins being wack.

GitHub Actions

I love GitHub Actions. I don’t make a profit of my personal projects, and Actions allows me to build & deploy to Pages and Surge without having to worry about building everything locally and then ftp/ssh’ing it into another location. I bring that knowledge into my work, because streamlining & automating development is a very worthwhile pursuit. That being said, if GitHub decides to change Actions/Pages, that would not be ideal, so I feel apprehensive about recommendng/

Here’s a few things that might be interesting

Locally running Actions

This is more of a band-aid if anything in terms of safety around GitHub Actions (as it is merely a local implementation of GitHub Actions), but it’s still good to write down: https://github.com/nektos/act.

Jekyll vs CircleCI vs Drone vs Haskell vs Woodpecker

Okay first of all, lets get a few of the confusing things out of the way

• A bunch of these I have found on the gitea awesome list. This is because Gitea is self-hostable, and the (seemingly) most free hosted git service Codeberg uses Gitea and is even linked on the gitea page.
• Woodpecker is an open source fork from the last Drone commit before their license became proprietary.
• Drone =/= Harness, however, Harness did buy Drone and Drone is becoming Harness Community Edition.
• Gitea.com notes that it uses gitea to developer gitea which is great and it also notes that this practice is called dogfooding which brightens my day.

Semantic.works

Docker-Ember

Install

A way to install docker-ember without having to change your path, compliant with Linux directory standards (I think), and having it all neatly tucked away.

1
2
3
4

cd /usr/local/lib/
git clone https://github.com/madnificent/docker-ember.git
cd ../bin/
ln -s ../lib/docker-ember/bin/* .

Usually symbolic links are made using absolute paths, but in /usr/local/ it’s not unheard of to use relative (npm, for example, does this).

User namespace

If you don’t want every file generated to be owned by root, change the userns-remap! This might especially be beneficial in production environments where security is key. Check the Docker documentation for more info.

Now, the default way is mapped out here. Below are the changes I made to both steps, and why.

/etc/sub{u,g}id

• Skipping this step causes EACCES: permission denied, mkdir '...' when trying to mount/create files using Docker.
• Original step:

  •   echo "$( whoami ):$(id -u):65536" |  sudo tee -a /etc/subuid
      echo "$( whoami ):$(id -g):65536" |  sudo tee -a /etc/subgid
      
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
        - I think the provided command didn't work for me because /etc/{subuid,subgid} had existing values for my user (`...:100000:65536`). I tried ignoring the existing values and just adding the new values to the end of the file. But to no avail: the created files' ownership belonged to the UID/GUID of the pre-existing values, not (easily) editable for my own user.
    - New step:
        - Replacing the existing value for my user in the file. `...:100000:65536` --> `...:1000:65536` & `...:100:65536` in subuid subgid respectively.
    
    #### daemon.json
    Instead of editing the service startup, you can use the daemon config file!
    
    Edit `/etc/docker/daemon.json`, and add the following
    ```json
    {
      /* ... */
      "userns-remap": "username:group"
    }
    
    

According to the documentation, you can use "UID:GID", but I used "username:groupname".
Alternatively, you can set the value to default, which then uses a user generated by Docker.

Restart

Afterwards, restart the docker service using systemctl restart docker.service. If Docker fails to start, you can use dockerd --debug.

Uninstall

1
2
3
4

cd /usr/local/bin/
rm ed edi edl eds
cd ../lib/
rm -rf docker-ember/

Full-reset a crashed virtuoso container

Note: this will probably remove all data. So don’t do this if there’s important data. But if you’re like me and had a goofed container from minute 1, here’s how you fully remake it!

1
2

docker-compose rm -fs database
rm data/db/{virtuoso*,.backup_restored,.data_loaded,.dba_pwd_set}

mu-migrations-service

Make sure to:

• Use the http://mu.semte.ch/application graph
• define mu:uuid

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

PREFIX bands: <http://mu.semte.ch/vocabularies/ext/bands/>
PREFIX albums: <http://mu.semte.ch/vocabularies/ext/albums/>
PREFIX songs: <http://mu.semte.ch/vocabularies/ext/songs/>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>

INSERT DATA {
  GRAPH <http://mu.semte.ch/application> {
    bands:metallica a ext:Band;
      bands:name "Metallica";
      mu:uuid "ae0987ea98ea09870e980e9a0".
    bands:kayaproject a ext:Band;
      bands:name "Kaya Project";
      mu:uuid "be0987ea98ea09870e980e9a0".
    bands:ledzeppelin a ext:Band;
      bands:name "Led Zeppelin";
      mu:uuid "ce0987ea98ea09870e980e9a0".
    bands:kayaproject a ext:Band;
      bands:name "Pearl Jam";
      bands:description "Pearl Jam is an American rock band, formed in Seattle, Washington in 1990.";
      mu:uuid "de0987ea98ea09870e980e9a0".
    bands:radiohead a ext:Band;
      bands:name "Radiohead";
      mu:uuid "ee0987ea98ea09870e980e9a0".
    bands:redhotchilipeppers a ext:Band;
      bands:name "Red Hot Chili Peppers";
      mu:uuid "fe0987ea98ea09870e980e9a0".

      #ext:hasAlbum albums:masterOfPuppets, albums:garageInc;
      #mu:uuid "ae0987ea98ea09870e980e9a0".
    # albums:masterOfPuppets
    #   a ext:Album;
    #   ext:title "Master of Puppets";
    #   ext:hasSong songs:Battery, songs:MasterOfPuppets";
    #   mu:uuid "745ea47ea64587ea".
    # songs:Battery
    #     a ext:Song;
    #     ext:title "Battery";
    #     mu:uuid "5643aoe7645a74aoe654".
  }
}

Madnificent’s troubleshooting search path

1
2
3
4
5
6
7
8
9
10

graph TD
    one[do we hit mu-cl-resources?]
    two[is the type the type I expect it to be?]
    three[is there data in the triplestore of that type?]
    four["where is the data of that type (graphs)"]
    five[which properties are known for that type?]
    six[is mu-authorization involved]
    seven[does mu-authorization fetch data from where it's at?]
    
    one-->two-->three-->four-->five-->six-->seven

Kaleidos diagram recreation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

graph TD
    Frontend===Identifier
    subgraph Identify & dispatch
        Identifier===Dispatcher
    end
    subgraph Core and shared
        Migrations
        subgraph Login
            Dispatcher===acm[ACM/IDM login]
            sink---acm
        end
        subgraph Resource
            Dispatcher---Cache
            Cache---cl-resources
        end
        Dispatcher---File
        subgraph Search
            Dispatcher---mu-search
            mu-search---el[(Elastic Search)]
        end
    end

    subgraph Custom services
        Dispatcher---range-file
        subgraph Access distribution
            meeting[Meeting distribution]
            userinfo[User info distribution]
        end
    end

    subgraph SEAS

        File o--o filesystem[(filesystem)]
        mu-search o--o filesystem
        range-file o--o filesystem

        acm===auth
        File---auth
        mu-search---auth
        cl-resources---auth
        meeting===auth
        userinfo===auth
        range-file---auth


        meeting-.-delta-notifier
        userinfo-.-delta-notifier
        cl-resources-.-delta-notifier
        mu-search-.-delta-notifier

        auth[mu-authorization]---Virtuoso[(Virtuoso)]
        Migrations===Virtuoso
        
        auth---delta-notifier
    end

Note

This is an import from my original notes repository original draft of my notes to quicky help someone get started with the diataxis/divio documentation structure! They’ve been integrated into Denperidge-Redpencil/project.

Divio Docs Quickstart

An attempt at boiling down the amazing 30min talk/documentation on how to write better documentation into something unobtrusive enough for people familiar with it to use as a checklist, but instructive enough for people unfamiliar with it to somewhat do the thing.

Does your documentation include…

Tutorials?

For each tutorial: write as if you’re teaching a child how to cook.

Checklist:

• Holds the readers hand from start to finish
• Gives a sense of achievement
• Works for everyone, with minimal explanation
• Also teaches what you take for granted

How-To guides?

For each how-to: write down how to achieve 1 specific thing as if it were a cooking recipe.

Checklist:

• Only 1 practical goal per how-to
• Minimal explanation
• Flexible: works for different but similar uses

A reference?

For each reference: write as if you are writing an encyclopedia article.

Checklist:

• Structured like the codebase
• Doesn’t explain common tasks (how-to guides)
• Doesn’t explain basic concepts (tutorials)
• Fully describes the machinery

Explanations: discussions/background material?

For each background material: write as if you’re writing about the history and context of a subject.

Checklist:

• Explains design decisions
• Considers alternatives
• Helps the reader make sense of things