Building dds from Source

While prebuilt dds executables are available on the GitHub page, one may wish to build dds from source.

The dds build process is designed to be as turn-key simple as possible.

Platform Support

dds aims to be as cross-platform as possible. It currently build and executes on Windows, macOS, Linux, and FreeBSD. Support for additional platforms is possible but will require modifications to bootstrap.py that will allow it to be built on such platforms.

Build Requirements

Building dds has a simple set of requirements:

  • Python 3.6 or newer to run the bootstrap/CI scripts.

  • A C++ compiler that has rudimentary support for C++20 concepts. Newer releases of Visual C++ that ship with VS 2019 will be sufficient on Windows, as will GCC 9 with -fconcepts on other platforms.

Note

On Windows, you will need to execute the build from within a Visual C++ enabled environment. This will involve launching the build from a Visual Studio Command Prompt.

Note

At the time of writing, C++20 Concepts has not yet been released in Clang, but should be available in LLVM/Clang 11 and newer.

Build Scripts and the CI Process

The main CI process is driven by Python. The root CI script is tools/ci.py, and it accepts several command-line parameters. Only a few of are immediate interest:

--bootstrap-with=<method> or -B <method>

Tell ci.py how to obtain the previous dds executable that can build the current dds source tree. This accepts one of three values: skip, download, or build. Refer to Bootstrapping dds.

--build-only

A flag that tells ci.py to exit after it has successfully built the current source tree, and to not execute the phase-2 build nor the automated tests.

--toolchain=<path> or -T <path>

Tell ci.py what toolchain to give to the prior dds to build the current dds.

The ci.py script performs the following actions, in order:

  1. Prepare the build output directory

  2. Bootstrap the prior version of dds that will build the current version.

  3. Import the embedded catalog.json into a catalog database stored within _build/. This will be used to resolve the third-party packages that dds itself uses.

  4. Invoke the build of dds using the prebuilt dds from the prior bootstrap phase. If --build-only was specified, the CI script stops here.

  5. Use the new dds executable to rebuild itself again (phase-2 self-build test). A bit of a “sanity test.”

  6. Execute the test suite using pytest.

Bootstrapping dds

In the beginning, dds was built by a Python script that globbed the sources and invoked the compiler+linker on those sources. Once dds was able to build and link itself, this Python script was replaced instead with dds building itself. dds has never used another build system.

The ci.py script accepts one of three methods for the --bootstrap-with flag: skip, download, or build.

Once bootstrapping is complete, a dds executable will be written to _prebuilt/dds. This executable refers to a previous version of dds that is able to build the newer dds source tree.

Note

For all development work on dds, the _prebuilt/dds executable should always be used. This means that newer dds features are not available for use within the dds repository.

Bootstrap: skip

If given skip, ci.py will not perform any bootstrapping steps. It will assume that there is an existing _prebuilt/dds executable. This option should be used once bootstrapping has been performed at least once with another method, as this is much faster than rebuilding/redownloading every time.

Bootstrap: download

The ci.py script has a reference to a download URL of the prior version of dds that has been designated for the bootstrap. These executables originate from the GitHub releases page.

If given download, then ci.py will download a predetermined dds executable and use it to perform the remainder of the build.

Bootstrap: build

Another script, tools/bootstrap.py is able to build dds from the ground up. It works by progressively cloning previous versions of the dds repository and using them to build the next commit in the chain.

While this is a neat trick, it isn’t necessary for most development, as the resulting executable will be derived from the same commit as the executable that would be obtained using the download method. This is also more fragile as the past commits may make certain assumptions about the system that might not be true outside of the CI environment. The build process may be tweaked in the future to correct these assumptions.

Selecting a Build Toolchain

dds includes three toolchains that it uses to build itself in its CI environment: tools/gcc-9.jsonc for Linux and macOS, tools/freebsd-gcc-9.jsonc for FreeBSD, and tools/msvc.jsonc for Windows.

While these toolchains will work perfectly well in CI, you may need to tweak these for your build setup. For example: gcc-9.jsonc assumes that the GCC 9 executables are named gcc-9 and g++-9, which is incorrect on some Linux distributions.

It is recommended to tweak these files as necessary to get the build working on your system. However, do not include those tweaks in a commit unless they are necessary to get the build running in CI.

Giving a Toolchain to ci.py

Just like passing a toolchain to dds, ci.py also requires a toolchain. Simply pass the path to your desired toolchain using the --toolchain/ -T argument:

$ python3 tools/ci.py [...] -T tools/gcc-9.jsonc

Building for Development

While ci.py is rigorous in maintaining a clean and reproducible environment, we often don’t need such rigor for a rapid development iteration cycle. Instead we can invoke the build command directly in the same way that ci.py does it:

$ _prebuilt/dds build -t [toolchain] \
    --catalog _build/catalog.db \
    --repo-dir _build/ci-repo

The --catalog and --repo-dir arguments are not strictly necessary, but help to isolate the dds dev environment from the user-local dds environment. This is important if modifications are made to the catalog database schema that would conflict with the one of an external dds version.

Note

You’ll likely want to run ci.py at least once for it to prepare the necessary catalog.db.

Note

As mentioned previously, if using MSVC, the above command must execute with the appropriate VS development environment enabled.

Running the Test Suite

The --build-only flag for ci.py will disable test execution. When this flag is omitted, ci.py will execute a self-build sanity test and then execute the main test suite, which is itself written as a set of pytest tests in the tests/ subdirectory.

Unit Tests

Various pieces of dds contain unit tests. These are stored within the src/ directory itself in *.test.cpp files. They are built and executed by the bootstrapped dds executable unconditionally. These tests execute in milliseconds and do not burden the development iteration cycle.