DEVELOPMENT(7)

NAME
development - operating system development instructions

SYNOPSIS
/src

DESCRIPTION
Releases come with the system source code in /src as a git(1) repository. It can be modified, compiled and installed on the current system. The source code is built with a make(1) build system. The source code can be located in any location, if so, simply substitute /src with the real location. These instructions only apply to building the operating system from within itself, those building it from another operating system needs to follow cross-development(7) instead.

If you are building a new version of the operation system where build tools have been added or changed, you first need to install the new tools. This is not needed when building the matching release. To do so, run as root:

        cd /src
        make distclean            # fully clean build directory
        make install-build-tools  # install new tools
        make clean-build-tools    # clean for real build below

To build the base operating system and upgrade the current system, run as root:

        cd /src
        make            # build new operating system in /src/sysroot
        make sysmerge   # upgrade current operating system with /src/sysroot

The build system creates a minimal root filesystem structure in the /src/sysroot and builds each operating system component in turn, installing them into the sysroot. The ports in the /src/ports are built automatically by downloading the upstream releases and applying the appropriate patches. The result is a system image that can be turned into working system by adding important configuration such as passwd(5).

The sysmerge make target ensures a system is built in /src/sysroot and then runs the sysmerge(8) program which installs the new system files onto the existing system. It updates the system manifest as well all ports installed in the sysroot. The initrd(7) is automatically regenerated using update-initrd(8). The bootloader, if enabled in upgrade.conf(5), is reinstalled and configured as necessary. The new user-space is running on completion, though existing processes will be running the old programs. A reboot is needed to run the new kernel. If the ABI changed and the current kernel isn't able to run the new programs, then the upgrade is delayed and will be automatically completed on the next boot. The sysmerge-wait make target forces waiting until the next boot.

  Root Makefile
The /src/Makefile handles the high level build of the operating system. The important targets are:

all (default) Build each component in turn and install them into the sysroot.

available-ports
    Search for newer available versions of ports.

build-tools
    Make all build tools.

clean
    Clean the component directories and the port source code. (clean-core, clean-ports)

clean-build-tools
    Clean the directories of all build tools.

clean-cross-compiler
    Clean the directories for the cross-compiler.

clean-cross-grub
    Clean the directories for the GRUB bootloader.

clean-cross-toolchain
    Clean the directories for the build tools and cross-compiler. (clean-build-tools, clean-cross-compiler)

distclean
    Run every clean target such that the source code is ready for distribution. (clean-builds, clean-core, clean-mirror, clean-release, clean-repository, clean-sysroot, distclean-ports, clean-cross-compiler)

distclean-ports
    Remove the port source code extractions.

extract-ports
    Extract the upstream release for each port(5) in /src/ports and apply the appropriate patches.

install-build-tools
    Install all build tools in PREFIX after making them.

install-cross-compiler
    Install the cross-compiler in PREFIX after making it.

install-cross-grub
    Install the GRUB bootloader in PREFIX.

install-cross-toolchain
    Install the build tools and cross-compiler in PREFIX after making them. (install-build-tools, install-cross-compiler)

iso Create a release iso in the /src/builds directory after making all.

mirror
    Download the upstream release for each port(5) in /src/ports from SORTIX_PORTS_MIRROR into the /src/mirror directory. The operating system can be built without network connectivity once the mirror is populated. The local mirror has the same structure as the remote mirror and can be used as a remote mirror.

mostlyclean
    Clean everything except binary packages and the mirror of upstream releases. (clean-builds, clean-ports, clean-release, clean-sysroot, distclean-ports, clean-cross-compiler)

presubmit
    Verify the port configuration (verify-ports), the coding style is followed (verify-coding-style), the manual pages does not have lints (verify-manual), the build tools compile (verify-build-tools), that everything compiles without warnings on all architectures (verify-build), the sysroot-source target includes all the sources (verify-sysroot-source), and the system headers works in all supported configurations (verify-headers).

release
    Make a release and construct a release directory tree in /src/release suitable for online publishing. The release is stored in a subdirectory named per the RELEASE variable. The release is optionally signed with tix-release(8) per the SIGNING environment variables below.

sortix.iso
    Make iso and place it in the current directory as sortix.iso.

sysmerge
    Upgrade the current operating system using the sysroot after making the all target.

sysmerge-full
    Like sysmerge but do a full operating system upgrade that uninstalls ports not present in the sysroot using --full.

sysmerge-full-wait
    The combination of sysmerge-full and sysmerge-full-wait.

sysmerge-wait
    Like sysmerge but delay the upgrade until the next boot using --wait.

sysroot-base-headers
    Create the sysroot and install only the headers of the standard library and kernel into it. This is useful when bootstrapping the runtime libraries of the compiler that need to know about libc prior to building libc.

upgrade-ports
    Search for newer available versions of ports and update the VERSION variable in the port(5) and switch it into development mode.

The important environment variables influencing the Makefile are:

BUILD
    The platform of the current operating system. This defaults to the current machine and operating system.

BUILD_ID
    The build identifier for the build, which is stored in the release information, allowing installations to check if they are up to date, or whether a new build is available. This variable defaults to the commit hash if built from git, and otherwise to the RELEASE variable.

CHANNEL
    The name of the tix-release(8) channel the release will use. This variable affects the RELEASE_URL variable. This variable defaults to the value in build-aux/version.mak.

HOST
    Specifies platform on which the compiled code will run. This defaults to the current machine and operating system. This is used when cross-compiling the operating system. When cross-compiling the operating system, it must be set to one of i686-sortix and x86_64-sortix. This must be unset when building the build tools as they run on the current operating system. The compiler tools are prefixed with this variable if it does not match BUILD.

OPTLEVEL
    Specifies compiler optimization options that gets added to CFLAGS and CXXFLAGS.

PACKAGES
    Specifies which ports to build, or all ports if unset. Suffixing a port with a ‘!’ includes its mandatory dependencies, and suffixing it with ‘!!’ includes the optional dependencies as well. The sets of ports defined in /src/build-aux/ports.conf can be specified as well. For instance PACKAGES=minimal! builds all the minimal ports with no optional dependencies.

RELEASE
    The name of the tix-release(8) release to produce in the release makefile target. This variable defaults to the version number in the VERSION variable.

RELEASE_AUTHORITATIVE
    The URL to the official authoritative release site where channels and releases are published in the channel and release subdirectories per tix-release(8). This variable affects the RELEASE_URL variable. This variable defaults to the value in build-aux/version.mak.

RELEASE_URL
    The URL to the channel or release to receive upgrades from. The release will place the value in the RELEASE_URL variable in the collection.conf(5) configuration file upon installation or upgrade. This default value is to use the CHANNEL channel on the RELEASE_AUTHORITATIVE site, namely: $RELEASE_AUTHORITATIVE/channel/$CHANNEL/$VERSION

    The SIGNING_PUBLIC_KEY public key is used to verify downloads from this URL.

SIGNING_KEY
    The base path to the tix-release(8) signing key, where where the secret key is in this path with the .sec file extension and the public key is this path with the .pub file extension.

SIGNING_KEY_SEARCH
    The base path to the release signing key, subject to the tix-release(8) --key-search feature.

SIGNING_PUBLIC_KEY
    The path to the tix-release(8) public key to sign the release with. If unset, SIGNING_KEY or SIGNING_KEY_SEARCH are used instead, if set. This public key is placed in the release as /tix/release.pub and will be used to verify downloads from RELEASE_URL.

SIGNING_SECRET_KEY
    The path to the tix-release(8) public key to sign the release with. If unset, SIGNING_KEY or SIGNING_KEY_SEARCH are used instead, if set.

SORTIX_INCLUDE_SOURCE
    Specifies whether the source code is included in the sysroot. This must be one of no, yes or git and defaults to git if git(1) is installed and yes otherwise.

SORTIX_ISO_COMPRESSION
    Specifies the compression algorithm used in iso files. This must be one of none, gzip or xz and defaults to xz.

SORTIX_PORTS_MIRROR
    Upstream releases of ports are downloaded from this mirror, defaulting to the official mirror, and falling back to the upstream release site if the mirror failed.

SORTIX_RELEASE_ADDITIONAL
    Optionally can be set to a directory that contains addition files that are copied into the release directory.

VERSION
    The version number being built. This variable defaults to the value in build-aux/version.mak.

  Components
The operating systems components, such as libc and the kernel, each have their own directory by that name. It contains a makefile that can build and install that component. This allows building and installing only that component onto the current operating system.

For instance, to build and install libc, run as root:

        cd /src/libc
        make
        make install

Note the individual makefiles only install the new system files and leak any files that don't exist anymore; and they also don't run any upgrade hooks to migrate the current system. This mechanism isn't supported unless you are building the same source code as the current operating system. The global sysmerge makefile targets should be used instead as the supported mechanism for operating system upgrades.

System libraries are statically linked and you will have to relink programs with the new library for changes to take effect. Building the whole operating system from the root makefile ensures components are built in the right order such that all programs use fresh libraries. The root makefile invokes component makefiles with SYSROOT set to /src/sysroot to force the compiler to locate files there. Likewise when installing, it sets DESTDIR to /src/sysroot to make it install files there.

  Directories
In addition to the directories for each operating system component, there are these special directories:

/src/builds
    The build artifacts produced when building the operating system.

/src/mirror
    The upstream releases of ports are downloaded on the first use and cached inside this mirror directory.

/src/ports
    Each subdirectory contains a port(5) which is automatically built along with the operating system per the PACKAGES environment variable.

/src/release
    The release root makefile target creates this directory and populates it with a directory structure suitable for online publishing of a release.

/src/repository
    This directory stores the binary packages and is created when they are built. This works as a cache so ports don't have to be rebuilt every time the operating system is. Packages are also copied from here rather than the sysroot when making releases.

/src/sysroot
    This directory is made when building the operating system and the freshly made files are installed here. The build system uses this as the system root which forces the compiler to look here for headers and libraries. This ensures a clean bootstrap where files from the current operating system do not leak into the new system.

/src/sysroot-overlay
    If this directory exists, it is added to the initrd of the produced iso and can contain additional system files.

  Build Tools
Some components are used to build the source code and must match the versions in the source code being built. These are currently:

•   carray
•   kblayout-compiler
•   sf
•   tix

If the currently installed versions of those tools are older than the ones in the source code, you must update them. The clean-build-tools root makefile target cleans the applicable directories, the build-tools root makefile target builds them from the source code, and the install-build-tools root makefile target installs the new version. You must clean the compiled files from the source code afterwards because the compiled tools are intended to run on the current system, and have not been built properly using /src/sysroot.

  Ports
Each port(5) in the /src/ports directory will get built automatically using tix-metabuild(8) and installed into the sysroot when building the whole operating system. Installable binary packages are created in the /src/repository/$HOST directory using tix-port(8) and can be installed with tix-install(8). If an existing binary package with the right version exists in the repository, it is used instead of the building the port again.

The ports system workflow is described in porting(7).

  Patches
The source code is managed as a git(1) repository and you can make your own changes and commit them. A good approach is to set up your own local development branch and work there:

        git checkout -b local
        git add utils/hello.c
        git commit -m 'Add hello(1).'

The presubmit makefile target can be used to verify your work needs some of the development conventions.

Prepare a set of patches suitable for upstream submission and submit a merge request to the upstream project.

If your installation does not have network connectivity, you will need to submit the changes from another system. If you are dual booting and have another operating system with network connectivity, you can boot into the other operating system and mount the appropriate filesystem from there. If you have a serial line, you can produce a set of .patch files containing your changes with

        git format-patch master..local

and then transfer them over the serial connection as described in serial-transfer(7).

  Releases
CD-ROM release of the operating system can be built with the iso root makefile target. This will build the whole operating system, if not done already, and produce a bootable iso for the current architecture in the /src/builds directory. The sortix.iso root makefile target will do the above and place a sortix.iso file in the current directory.

The release root makefile target will run the iso target and prepare a /src/release directory with a directory structure and miscellaneous files suitable for a formal online release. The release is optionally signed with tix-release(8) per the SIGNING environment variables above, or it can be signed at a later time. tix-release(8) is used to publish the signed releases on a signed channel, allowing installations to upgrade to the latest release.

  Following Development
The following-development(7) manual page documents what needs to be done to stay updated with the latest developments. You will need to read the new version of that document whenever you update the source code.

SEE ALSO
git(1), make(1), port(5), cross-development(7), following-development(7), installation(7), porting(7), serial-transfer(7), upgrade(7), sysinstall(8), sysmerge(8), update-initrd(8)