This change adds all the kernel parts of a network stack. The network stack is partial but implements many of the important parts. Add if(4) network interface abstraction. Network interfaces are registered in a global list that can be iterated and each assigned an unique integer identifier. Add reference counted packets with a cache that recycles recent packets. Add support for lo(4) loopback and ether(4) ethernet network interfaces. The /dev/lo0 loopback device is created automatically on boot. Add arp(4) address resolution protocol driver for translation of inet(4) network layer addresses into ether(4) link layer addresses. arp(4) entries are cached and evicted from the cache when needed or when the entry has not been used for a while. The cache is limited to 256 entries for now. Add ip(4) internet protocol version 4 support. IP fragmentation and options are not implemented yet. Add tcp(4) transmission control protocol sockets for a reliable transport layer protocol that provides a reliable byte stream connection between two hosts. The implementation is incomplete and does not yet implement out of band data, options, and high performance extensions. Add udp(4) user datagram protocol sockets for a connectionless transport layer that provides best-effort delivery of datagrams. Add ping(4) sockets for a best-effort delivery of echo datagrams. Change type of sa_family_t from unsigned short to uint16_t. Add --disable-network-drivers to the kernel(7) options and expose it with a bootloader menu. tix-iso-bootconfig can set this option by default. Import CRC32 code from libz for the Ethernet checksum. This is a compatible ABI change that adds features to socket(2) (AF_INET, IPPROTO_TCP, IPPROTO_UDP, IPPROTO_PING), the ioctls for if(4), socket options, and the lo0 loopback interface. This commit is based on work by Meisaka Yukara contributed as the commit bbf7f1e8a5238a2bd1fe8eb1d2cc5c9c2421e2c4. Almost no lines of this work remains in this final commit as it has been rewritten or refactored away over the years, see the individual file headers for which files contain remnants of this work. Co-authored-by: Meisaka Yukara <Meisaka.Yukara@gmail.com>
459 lines
14 KiB
Groff
459 lines
14 KiB
Groff
.Dd December 29, 2015
|
|
.Dt DEVELOPMENT 7
|
|
.Os
|
|
.Sh NAME
|
|
.Nm development
|
|
.Nd operating system development instructions
|
|
.Sh SYNOPSIS
|
|
.Pa /src
|
|
.Sh DESCRIPTION
|
|
Releases come with the system source code in
|
|
.Pa /src
|
|
as a
|
|
.Xr git 1
|
|
repository.
|
|
It can be modified, compiled and installed on the current system.
|
|
The source code is built with a
|
|
.Xr make 1
|
|
build system.
|
|
The source code can be located in any location, if so, simply substitute
|
|
.Pa /src
|
|
with the real location.
|
|
These instructions only apply to building the operating system from within
|
|
itself, those building it from another operating system needs to follow
|
|
.Xr cross-development 7
|
|
instead.
|
|
.Pp
|
|
If you are building a new version of the operation system where build tools have
|
|
been added or changed, you first need to install the new tools.
|
|
This is not needed when building the matching release.
|
|
To do so, run as root:
|
|
.Bd -literal
|
|
cd /src
|
|
make distclean # fully clean build directory
|
|
make install-build-tools # install new tools
|
|
make clean-build-tools # clean for real build below
|
|
.Ed
|
|
.Pp
|
|
To build the base operating system and upgrade the current
|
|
system, run as root:
|
|
.Bd -literal
|
|
cd /src
|
|
make # build new operating system in /src/sysroot
|
|
make sysmerge # upgrade current operating system with /src/sysroot
|
|
.Ed
|
|
.Pp
|
|
The build system creates a minimal root filesystem structure in the
|
|
.Pa /src/sysroot
|
|
and builds each operating system component in turn, installing them into the
|
|
sysroot.
|
|
The ports in the
|
|
.Pa /src/ports
|
|
are built automatically by downloading the upstream releases and applying the
|
|
appropriate patches.
|
|
The result is a system image that can be turned into working system by adding
|
|
important configuration such as
|
|
.Xr passwd 5 .
|
|
.Pp
|
|
The
|
|
.Sy sysmerge
|
|
make target ensures a system is built in
|
|
.Pa /src/sysroot
|
|
and then runs the
|
|
.Xr sysmerge 8
|
|
program which installs the new system files onto the existing system.
|
|
It updates the system manifest as well all ports installed in the sysroot.
|
|
The
|
|
.Xr initrd 7
|
|
is automatically regenerated using
|
|
.Xr update-initrd 8 .
|
|
The bootloader, if enabled in
|
|
.Xr upgrade.conf 5 ,
|
|
is reinstalled and configured as necessary.
|
|
The new user-space is running on completion, though existing processes will be
|
|
running the old programs.
|
|
A reboot is needed to run the new kernel.
|
|
If the ABI changed and the current kernel isn't able to run the new programs,
|
|
then the upgrade is delayed and will be automatically completed on the next
|
|
boot.
|
|
The
|
|
.Sy sysmerge-wait
|
|
make target forces waiting until the next boot.
|
|
.Ss Root Makefile
|
|
The
|
|
.Pa /src/Makefile
|
|
handles the high level build of the operating system.
|
|
The important targets are:
|
|
.Bl -tag -width "12345678"
|
|
.It Sy all
|
|
(default) Build each component in turn and install them into the sysroot.
|
|
.It Sy available-ports
|
|
Search for newer available versions of ports.
|
|
.It Sy build-tools
|
|
Make all build tools.
|
|
.It Sy clean
|
|
Clean the component directories and the port source code.
|
|
.Sy ( clean-core , clean-ports )
|
|
.It Sy clean-build-tools
|
|
Clean the directories of all build tools.
|
|
.It Sy clean-cross-compiler
|
|
Clean the directories for the cross-compiler.
|
|
.It Sy clean-cross-toolchain
|
|
Clean the directories for the build tools and cross-compiler.
|
|
.Sy ( clean-build-tools , clean-cross-compiler )
|
|
.It Sy distclean
|
|
Run every clean target such that the source code is ready for distribution.
|
|
.Sy ( clean-builds , clean-core , clean-mirror , clean-release , clean-repository , clean-sysroot , distclean-ports, clean-cross-compiler )
|
|
.It Sy distclean-ports
|
|
Remove the port source code extractions.
|
|
.It Sy extract-ports
|
|
Extract the upstream release for each
|
|
.Xr port 5
|
|
in
|
|
.Pa /src/ports
|
|
and apply the appropriate patches.
|
|
.It Sy install-build-tools
|
|
Install all build tools after making them.
|
|
.It Sy install-cross-compiler
|
|
Install the cross-compiler after making it.
|
|
.It Sy install-cross-toolchain
|
|
Install the build tools and cross-compiler after making them.
|
|
.Sy ( install-build-tools , install-cross-compiler )
|
|
.It Sy iso
|
|
Create a release iso in the
|
|
.Pa /src/builds
|
|
directory after making
|
|
.Sy all .
|
|
.It Sy mirror
|
|
Download the upstream release for each
|
|
.Xr port 5
|
|
in
|
|
.Pa /src/ports
|
|
from
|
|
.Ev SORTIX_PORTS_MIRROR
|
|
into the
|
|
.Pa /src/mirror
|
|
directory.
|
|
The operating system can be built without network connectivity once the mirror
|
|
is populated.
|
|
The local mirror has the same structure as the remote mirror and can be used as
|
|
a remote mirror.
|
|
.It Sy mostlyclean
|
|
Clean everything except binary packages and the mirror of upstream releases.
|
|
.Sy ( clean-builds , clean-ports , clean-release , clean-sysroot , distclean-ports, clean-cross-compiler )
|
|
.It Sy presubmit
|
|
Verify the port configuration
|
|
.Sy ( verify-ports ) ,
|
|
the coding style is followed
|
|
.Sy ( verify-coding-style ) ,
|
|
the manual pages does not have lints
|
|
.Sy ( verify-manual ) ,
|
|
the build tools compile
|
|
.Sy ( verify-build-tools ) ,
|
|
that everything compiles without warnings on all architectures
|
|
.Sy ( verify-build ) ,
|
|
and the system headers works in all supported configurations
|
|
.Sy ( verify-headers ) .
|
|
.It Sy release
|
|
Make
|
|
.Sy iso
|
|
and construct release directory tree in
|
|
.Pa /src/release
|
|
suitable for online publishing.
|
|
.It Sy sortix.iso
|
|
Make
|
|
.Sy iso
|
|
and place it in the current directory as
|
|
.Pa sortix.iso .
|
|
.It Sy sysmerge
|
|
Upgrade the current operating system using the sysroot after making the
|
|
.Sy all
|
|
target.
|
|
.It Sy sysmerge-full
|
|
Like
|
|
.Sy sysmerge
|
|
but do a full operating system upgrade that uninstalls ports not present in the
|
|
sysroot using
|
|
.Fl \-full .
|
|
.It Sy sysmerge-full-wait
|
|
The combination of
|
|
.Sy sysmerge-full
|
|
and
|
|
.Sy sysmerge-full-wait .
|
|
.It Sy sysmerge-wait
|
|
Like
|
|
.Sy sysmerge
|
|
but delay the upgrade until the next boot using
|
|
.Fl \-wait .
|
|
.It Sy sysroot-base-headers
|
|
Create the sysroot and install only the headers of the standard library and
|
|
kernel into it.
|
|
This is useful when bootstrapping the runtime libraries of the compiler that
|
|
need to know about libc prior to building libc.
|
|
.It Sy upgrade-ports
|
|
Search for newer available versions of ports and update the
|
|
.Sy VERSION
|
|
variable in the
|
|
.Xr port 5
|
|
and switch it into development mode.
|
|
.El
|
|
.Pp
|
|
The important environment variables influencing the Makefile are:
|
|
.Bl -tag -width "12345678"
|
|
.It Ev BUILD
|
|
The platform of the current operating system.
|
|
This defaults to the current machine and operating system.
|
|
.It Ev HOST
|
|
Specifies platform on which the compiled code will run.
|
|
This defaults to the current machine and operating system.
|
|
This is used when cross-compiling the operating system.
|
|
When cross-compiling the operating system, it must be set to one of
|
|
.Sy i686-sortix
|
|
and
|
|
.Sy x86_64-sortix .
|
|
This must be unset when building the build tools as they run on the current
|
|
operating system.
|
|
The compiler tools are prefixed with this variable if it does not match
|
|
.Ev BUILD.
|
|
.It Ev OPTLEVEL
|
|
Specifies compiler optimization options that gets added to
|
|
.Ev CFLAGS
|
|
and
|
|
.Ev CXXFLAGS .
|
|
.It Ev PACKAGES
|
|
Specifies which ports to build, or all ports if unset.
|
|
Suffixing a port with a
|
|
.Sq "!"
|
|
includes its mandatory dependencies, and suffixing it with
|
|
.Sq !!
|
|
includes the optional dependencies as well.
|
|
The sets of ports defined in
|
|
.Pa /src/build-aux/ports.conf
|
|
can be specified as well.
|
|
For instance
|
|
.Sy PACKAGES=minimal!
|
|
builds all the minimal ports with no optional dependencies.
|
|
.It Ev SORTIX_INCLUDE_SOURCE
|
|
Specifies whether the source code is included in the sysroot.
|
|
This must be one of
|
|
.Sy no , yes
|
|
or
|
|
.Sy git
|
|
and defaults to
|
|
.Sy git
|
|
if
|
|
.Xr git 1
|
|
is installed and
|
|
.Sy yes
|
|
otherwise.
|
|
.It Ev SORTIX_ISO_COMPRESSION
|
|
Specifies the compression algorithm used in iso files.
|
|
This must be one of
|
|
.Sy none , gzip
|
|
or
|
|
.Sy xz
|
|
and defaults to
|
|
.Sy xz .
|
|
.It Ev SORTIX_PORTS_MIRROR
|
|
Upstream releases of ports are downloaded from this mirror, defaulting to the
|
|
official mirror, and falling back to the upstream release site if the mirror
|
|
failed.
|
|
.El
|
|
.Ss Components
|
|
The operating systems components, such as libc and the kernel, each have their
|
|
own directory by that name.
|
|
It contains a makefile that can build and install that component.
|
|
This allows building and installing only that component onto the current
|
|
operating system.
|
|
.Pp
|
|
For instance, to build and install libc, run as root:
|
|
.Bd -literal
|
|
cd /src/libc
|
|
make
|
|
make install
|
|
.Ed
|
|
.Pp
|
|
Note the individual makefiles only install the new system files and leak any
|
|
files that don't exist anymore; and they also don't run any upgrade hooks to
|
|
migrate the current system.
|
|
This mechanism isn't supported unless you are building the same source code as
|
|
the current operating system.
|
|
The global
|
|
.Sy sysmerge
|
|
makefile targets should be used instead as the supported mechanism for operating
|
|
system upgrades.
|
|
.Pp
|
|
System libraries are statically linked and you will have to relink programs with
|
|
the new library for changes to take effect.
|
|
Building the whole operating system from the root makefile ensures components
|
|
are built in the right order such that all programs use fresh libraries.
|
|
The root makefile invokes component makefiles with
|
|
.Ev SYSROOT
|
|
set to
|
|
.Pa /src/sysroot
|
|
to force the compiler to locate files there.
|
|
Likewise when installing, it sets
|
|
.Ev DESTDIR
|
|
to
|
|
.Pa /src/sysroot
|
|
to make it install files there.
|
|
.Ss Directories
|
|
In addition to the directories for each operating system component, there are
|
|
these special directories:
|
|
.Bl -tag -width "12345678"
|
|
.It Pa /src/builds
|
|
The build artifacts produced when building the operating system.
|
|
.It Pa /src/mirror
|
|
The upstream releases of ports are downloaded on the first use and cached inside
|
|
this mirror directory.
|
|
.It Pa /src/ports
|
|
Each subdirectory contains a
|
|
.Xr port 5
|
|
which is automatically built along with the operating system per the
|
|
.Ev PACKAGES
|
|
environment variable.
|
|
.It Pa /src/release
|
|
The
|
|
.Sy release
|
|
root makefile target creates this directory and populates it with a directory
|
|
structure suitable for online publishing of a release.
|
|
.It Pa /src/repository
|
|
This directory stores the binary packages and is created when they are built.
|
|
This works as a cache so ports don't have to be rebuilt every time the operating
|
|
system is.
|
|
Packages are also copied from here rather than the sysroot when making releases.
|
|
.It Pa /src/sysroot
|
|
This directory is made when building the operating system and the freshly made
|
|
files are installed here.
|
|
The build system uses this as the system root which forces the compiler to look
|
|
here for headers and libraries.
|
|
This ensures a clean bootstrap where files from the current operating system do
|
|
not leak into the new system.
|
|
.It Pa /src/sysroot-overlay
|
|
If this directory exists, it is added to the initrd of the produced iso and can
|
|
contain additional system files.
|
|
.El
|
|
.Ss Build Tools
|
|
Some components are used to build the source code and must match the versions in
|
|
the source code being built.
|
|
These are currently:
|
|
.Pp
|
|
.Bl -bullet -compact
|
|
.It
|
|
carray
|
|
.It
|
|
kblayout-compiler
|
|
.It
|
|
mkinitrd
|
|
.It
|
|
sf
|
|
.It
|
|
tix
|
|
.El
|
|
.Pp
|
|
If the currently installed versions of those tools are older than the ones in
|
|
the source code, you must update them.
|
|
The
|
|
.Sy clean-build-tools
|
|
root makefile target cleans the applicable directories, the
|
|
.Sy build-tools
|
|
root makefile target builds them from the source code, and the
|
|
.Sy install-build-tools
|
|
root makefile target installs the new version.
|
|
You must clean the compiled files from the source code afterwards because the
|
|
compiled tools are intended to run on the current system, and have not been
|
|
built properly using
|
|
.Pa /src/sysroot .
|
|
.Ss Ports
|
|
Each
|
|
.Xr port 5
|
|
in the
|
|
.Pa /src/ports
|
|
directory will get built automatically when and installed into the sysroot when
|
|
building the whole operating system.
|
|
Installable binary packages are created in the
|
|
.Pa /src/repository/$HOST
|
|
directory using
|
|
.Xr tix-port 8
|
|
and can be installed with
|
|
.Xr tix-install 8 .
|
|
If an existing binary package with the right version exists in the repository,
|
|
it is used instead of the building the port again.
|
|
.Pp
|
|
The ports system workflow is described in
|
|
.Xr porting 7 .
|
|
.Ss Patches
|
|
The source code is managed as a
|
|
.Xr git 1
|
|
repository and you can make your own changes and commit them.
|
|
A good approach is to set up your own local development branch and work there:
|
|
.Bd -literal
|
|
git checkout -b local
|
|
git add utils/hello.c
|
|
git commit -m 'Add hello(1).'
|
|
.Ed
|
|
.Pp
|
|
The
|
|
.Sy presubmit
|
|
makefile target can be used to verify your work needs some of the development
|
|
conventions.
|
|
.Pp
|
|
Prepare a set of patches suitable for upstream submission and submit a merge
|
|
request to the upstream project.
|
|
.Pp
|
|
If your installation does not have network connectivity, you will need to
|
|
submit the changes from another system.
|
|
If you are dual booting and have another operating system with network
|
|
connectivity, you can boot into the other operating system and mount the
|
|
appropriate filesystem from there.
|
|
If you have a serial line, you can produce a set of .patch files containing
|
|
your changes with
|
|
.Bd -literal
|
|
git format-patch master..local
|
|
.Ed
|
|
.Pp
|
|
and then transfer them over the serial connection as described in
|
|
.Xr serial-transfer 7 .
|
|
.Ss Releases
|
|
CD-ROM release of the operating system can be built with the
|
|
.Sy iso
|
|
root makefile target.
|
|
This will build the whole operating system, if not done already, and produce a
|
|
bootable iso for the current architecture in the
|
|
.Pa /src/builds
|
|
directory.
|
|
The
|
|
.Sy sortix.iso
|
|
root makefile target will do the above and place a
|
|
.Pa sortix.iso
|
|
file in the current directory.
|
|
.Pp
|
|
The
|
|
.Sy release
|
|
root makefile target will run the
|
|
.Sy iso
|
|
target and prepare a
|
|
.Pa /src/release
|
|
directory with a directory structure and miscellaneous files suitable for a
|
|
formal online release.
|
|
.Ss Following Development
|
|
The
|
|
.Xr following-development 7
|
|
manual page documents what needs to be done to stay updated with the latest
|
|
developments.
|
|
You will need to read the new version of that document whenever you update the
|
|
source code.
|
|
.Sh SEE ALSO
|
|
.Xr git 1 ,
|
|
.Xr make 1 ,
|
|
.Xr port 5 ,
|
|
.Xr cross-development 7 ,
|
|
.Xr following-development 7 ,
|
|
.Xr installation 7 ,
|
|
.Xr porting 7 ,
|
|
.Xr serial-transfer 7 ,
|
|
.Xr upgrade 7 ,
|
|
.Xr sysinstall 8 ,
|
|
.Xr sysmerge 8 ,
|
|
.Xr update-initrd 8
|