Alan's Q3 Update

By Alan Freitas on
Oct 27, 2023

Summary

MrDocs

Over the last quarter, we have been working intensely on MrDocs, a documentation generator for C++ projects. I’ve been overseeing and reviewing all the work done by the other contributors in the project. I’ve also been responsible for:

  • setting up and maintaining CI for the project;
  • MrDocs and LLVM release binaries;
  • build scripts;
  • setting up and integrating dependencies;
  • setting up and deploying the Antora toolchains and documentation to the project website;
  • working on supporting libraries; and
  • fixing bugs.

These are some of the highlights of the work done in the last quarter:

  • Refactor library layout and scripts so all the implementation files are in src
  • Automated deployment of demo documentation to the project website. The website is updated on every commit to the develop or master branches. The action uses the actual release package to generate the documentation. This PR involved fixing all scripts and directory layouts for the release package to be properly usable. The new project layout uses the FHS standard. CMake modules to create a target to generate the documentation for projects were included in the installation. The MrDocs executable and libraries are installed as separate CMake package components. Any usage of FetchContent has been removed from the CMake scripts. The documentation was refactored to reflect the changes. Included complete instructions in the documentation pages for installation and usage, describing all commands and options.
  • Deployed new LLVM binaries to the website and updated CI. The binaries were regenerated for all platforms in 4 modes: Debug, Release, RelWithDebInfo, and DebWithOpt. This involved fixing long-standing bugs related to conflicts in LLVM versions used by the project and providing new pre-built binaries on the website. In particular, the previous pre-built binaries used a special ReleaseWithDebInfo LLVM configuration that caused conflicts with MrDocs on MSVC, being used with a Debug CMake configuration variant by developers. This eliminated the need for the ad-hoc GitHub LLVM binaries release and for the special docker container we had been using so far.
  • Added support for CMakePresets. This allowed us to simplify the build process for MrDocs and its dependencies, which was previously counting on long command line commands and CMakeUserPresets.json files without a corresponding CMakePresets.json file. It was also a step towards the new installation instructions in the documentation. An example file for CMakeUserPresets.json including all compilers was provided. The base CMakePresets.json file included a special vendor configuration to hide base configurations from Visual Studio. CMakeSettings.json was deprecated.
  • Added complete installation instructions for the project. The instructions were adapted so that all commands are relative to an enclosing directory containing all the dependencies and MrDocs. Included instructions for all steps considering package managers, installed dependencies, or binaries provided on the project website. The CMake scripts were adapted to make it easier to build the project in a variety of environments according to the instructions.
  • Included a polyfill implementation of std::expected as mrdocs::Expected. This implementation is currently being used by MrDocs and support libraries.
  • Refactored MrDocs generators to use our custom C++ implementation of the Handlebars template engine. Javascript helpers are loaded with duktape with our Javascript Bindings and compiled into functions in the Javascript engine.
  • Refactored the project name from MrDox to MrDocs

Handlebars

MrDocs includes a support library that reimplements the Handlebars template engine in C++. This module is used to generate documentation from templates.

Over the last quarter, this is the MrDocs support library in which I have been investing most of my time. All the development started in this same quarter in July. It already supports all features from the original Handlebars test suite, including all mustache features. The library is already integrated with MrDocs to generate the documentation for the project website.

  • Initial proposal of the C++ Handlebars library.
  • Fixed and refactored code that relied on references to temporaries generated by the Dom.
  • Included support for inverse blocks without helpers
  • All features, specs, and tests from the original Handlebars test suite were then ported to C++ and are now passing: basic specs; partials and automatic indentation; whitespace control; block helpers, mustache blocks, and chained blocks; subexpressions; builtin helpers; private data; helper formats; track-ids mode; strict mode; util; mustache. As many handlebars features were undocumented in the original implementation, adjusting our handlebars implementation, which was only designed to handle basic templates, to pass all tests from the original Handlebars test suite involved multiple significant refactors and improvements to the entire codebase. The tests are a superset of the mustache specs, which are also passing. The previously available SafeString type because a regular dom::Value with the implementation of the specs.
  • Support for dom::Function in all components of the Handlebars engine instead of custom callback types. This also allows the engine context to contain functions that can work similarly to helpers. The engine callback object is passed as the last argument in a helper, similar to the Javascript implementation. Because most built-in helpers support variable arguments, a new dom::Function implementation type was provided to support this use case.
  • Support for error handling via Expected. All functions that might fail have a variant that throws exceptions and another that returns an Expected value. Helpers functions are also allowed to propagate errors via mrdocs::Error.
  • Fixed a bug that caused MrDocs to emit [object Object] for \n after the transition to C++ handlebars.
  • The “find” helper was adjusted to support strings, arrays, and objects. This fixes a problem with the variants of similar but conflicting helpers that were categorized as container and string helpers.

The DOM

MrDocs also includes a support library called “DOM” that provides a C++ interface to type-erased property trees, such as JSON and Javascript Objects. This module is used by MrDocs to create opaque representations of property trees that can be used by the Handlebars engine and other library functions. Such representations can come from a variety of sources, including JSON files, Javascript objects, and internal C++ objects with information parsed by MrDocs.

After completing the Handlebars implementation, I also included complete unit tests for the DOM. Tests were included for all DOM types and many bugs have been fixed.

All classes have been documented to reflect their intended behavior, which is loosely modeled after JavaScript types and data structures. The APIs have also been adjusted to be safer and more consistent with the model for reference types. Objects and Value types received functions for nested object lookup and objects were generalized to support non-enumerable properties and Javascript bindings.

Javascript Bindings

MrDocs includes a support library that wraps duktape to provide a C++ interface to the Javascript engine. This module is used by the MrDocs executable to evaluate user-defined helpers.

I also included unit tests for Javascript wrapper and bindings.

Throughout the process,

  • the implementation was completed for classes that were placeholders and existing bugs have been fixed
  • the API was documented
  • The javascript Scope object was extended to support all types of alternative syntax to evaluate expressions.
  • Value types: support for integers and floating point numbers.
  • Value types: support for all dom::Value operations using the native duktape functions
  • Provided classes to wrap javascript Objects, Arrays, and Functions as DOM values
  • Scope functions that might fail were adjusted to return Expected values

Unit Tests

MrDocs includes a support library for unit tests. The library was initially adapted from the Boost.URL unit tests and extended to support the needs of MrDocs.

I had previously implemented a smaller system for the Handlebars unit tests which was then integrated with the boost.url test suite library. Features from the handlebars test suite library were ported to the boost.url test suite library, including the expression decomposer and the diff algorithm for golden master tests.

With this integration, Handlebars tests were then listed among any other tests in the library. These tests were later complemented with regular MrDocs tests.

The decomposer has later been improved for integral comparison operators.

Boost Website

In this last quarter, the Boost website went beta on https://www.preview.boost.org/. Among the many support projects for the website, I’ve been helping the most on the cppalliance/site-docs, which includes the Boost website documentation as an Antora project. Its components represent the “User Guide”, “Contributor Guide”, and “Formal Review” sections of the website.

Since the inception of the project, I’ve been overseeing and reviewing all the work done by the other contributors to the project. I’ve also been responsible for:

  • setting up and maintaining CI for the project;
  • coordinating with cppalliance/temp-site on content uploaded to AWS buckets;
  • build scripts to be reused by the release tools and previews;
  • writing sections of the documentation that require technical knowledge;
  • developing custom Boost/Antora extensions, such as the Boost Macro extension;
  • maintaining the Antora toolchain and templates; and
  • adjusted Boost libraries to match formats expected by the website.

The Antora playbooks were recently adjusted to initially contain no content sources, now that the Antora-enabled build process also implemented by me was deployed in the official Boost release process.

Boost Libraries

As in other quarters, the Boost Library in which I have been investing most of my time is Boost.URL, since it’s our most recently accepted library. The library is in maintenance mode since our focus shifted to MrDocs, but considering how recent it is, there is a constant demand for work fixing bugs and improving the documentation. In Boost.URL, I’ve been responsible for:

  • upgrading CI, mostly coordinating with the C++ Github Actions;
  • maintaining, simplifying, and updating build scripts;
  • integrating more spec tests, such as the Ada tests included more recently;
  • including more examples, such as the more recent sanitize-URL example;
  • fixing documentation content that is out of date; and
  • fixing bugs.

Besides bugs, the library was recently refactored to remove what was previously called a “header-only” mode and deprecated several aliases, which caused some small friction in this last quarter. These are some of the highlights of the work done in the last quarter:

  • Extended fuzz testing. Fuzz was included as a new clang factor in CI. The process was adjusted so that the corpus is properly reused from and stored in GitHub action archives. CMake scripts were refactored to include CMake options that control the fuzzer parameters. Fuzzers were included for each of the grammar rules for URLs.
  • Support IP-literal as IPv6addrz. This is an issue where a valid IPv6addrz wasn’t being considered an IP-literal. IPv6addrz includes a ZoneID at the end, delimited by an encoded "%25". The ipv6_address class is unmodified, as the mapping from the ZoneID to a std::uint32_t is dependent on the application context. The original ZoneID can be obtained from the url_view but the library is agnostic about it.
  • Included GDB pretty printers and documentation. All available URL components are now pretty printed in GDB. A developer mode was also included which prints the URL components in a format corresponding to the internal URL string offsets.
  • Updated the content of both the documentation and README.adoc so that they match current best practices. The documentation in README.md contained dated and incorrect information, while the quickbook documentation was missing important information and contained bad practices.

Some relevant bug fixes were:

Besides Boost.URL, as usual, I’ve been overseeing and fixing smaller issues with other boost libraries, such as Boost.Docca, Boost.StaticString, and helping with libraries by other contributors when asked for assistance, as in a more recent case with Boost.Outcome.

In particular, we had to fix a smaller issue in Boost.Docca that was also affecting Boost.URL. The issue involved Boost.Docca’s dependence on a deprecated version of Doxygen that is no longer supported by the Boost toolchain.

Boost Release Tools

Over the last quarter, I’ve been working on the integration of toolchains I developed into the Boost Release Tools to add support for features desired for the new website.

Some of the highlights of the work done in the last quarter:

  • Introduced support for libraries with Antora documentation into the official Boost release process. Deployed new docker containers that include NodeJS, Gulp, and the Antora toolchain. With this enhancement, each library now can function as an Antora component within an Antora master project hosted in a separate repository (https://github.com/cppalliance/site-docs). This master project repository also contains additional components, such as the user guide, contributor guide, and a dedicated component for the review process. In a subsequent phase of the release process, this Antora documentation is seamlessly merged with the pre-existing in-source documentation, which has been generated using various other tools. When a library is “Antora-enabled”, the release process will automatically generate the Antora documentation and publish it with the documentation of other libraries. No b2 scripts are required to generate the documentation for the library. All Antora-enabled libraries use the same master Antora UI template that matches the design of the boost website. Antora playbooks were adjusted to initially contain no content sources, now that the Antora-enabled build process in deployed in the official Boost release process.
  • Deployed new container to boostorg/boost to support the new release process.
  • New archive variants for boost. Add extra archive variants such as boost-docs and boost-source. These variants can reduce expenses with JFrog download bandwidth, provide users with archives that are simpler to use, and provide docs-only archives for the website. The new MakeBoostDistro.py script includes parameters to determine what types of files should be included in the distribution. All other functions are adapted to handle these requirements accordingly. Switching to source-only downloads would save Boost $1000 per month.

C++ Github Actions

C++ Github Actions is a project I created and have been maintaining since the second quarter of the year. It is a collection of reusable Github Actions for any C++ project that needs to be tested on a variety of compilers. Both MrDocs are Boost.URL are currently using these actions in their CI.

The project includes actions to:

  • Generate a Github Actions Matrix for C++ projects;
  • Setup C++ compilers;
  • Install and setup packages;
  • Clone Boost modules;
  • Run complete CMake and b2 workflows;
  • Generate changelogs from conventional commits;
  • Generate summaries; and
  • Generate time-trace reports and flame graphs

These actions include a myriad of options and features.

  • The “setup-*” actions include logic to detect, install, and cache dependencies, which can be used by the CMake and b2 actions.
  • Individual options and actions attempt to set up a wide variety of compilers on different platforms, including MSVC, GCC, Clang, MinGW, AppleClang, and Clang-CL.
  • Actions that generate reports include a multitude of tools and options to analyze changes, time traces, and coverage.

Since then, these actions have been adapted as needed to support the needs of MrDocs and Boost.URL, which have also been using conventional commits. Here’s a recent summary report generated for Boost.URL by the CI workflow: https://github.com/boostorg/url/actions/runs/6512424067

The project documentation also uses the Antora UI template we have been maintaining for all other projects: https://alandefreitas.github.io/cpp-actions

On 13 September 2023, the following paper I co-authored was published:

Lopes, R.A., Freitas, A.R.R. 
Gray-box local search with groups of step sizes. 
Soft Computing. 
p. 1-14
2023
https://doi.org/10.1007/s00500-023-09129-1

The paper was accepted on 18 August 2023 and published on 13 September 2023.

While the paper is more aligned with my educational background than my daily C++ Alliance tasks, it is one more paper that carries the C++ Alliance in the affiliation, contributing to its reputation.

All Posts by This Author