Bootstrapable Debian

Cyclic dependencies, staged builds and cross-compiling Debian is extremely difficult to bootstrap for a new architecture or a new ABI, and every time someone does the work it's incredibly painful. We have a lot of cyclic build-dependencies which means there is no valid build-order, and many packages don't cross-build properly, where at least some of the packages must be cross-built for a new port. We also don't have tools to manage such bootstrap-builds. In practice people tend to use some other system to get started, then try to turn it into Debian. We really ought to be able to bootstrap ourselves.

This talks explains the issues (with pretty graphs!), shows how to fix them, which packages and tools are affected, and covers the current state of this work. It also explains why people want/need this functionality, and what affected package maintainers need to do to keep things bootstrappable. Making all this work makes Debian useful in various places where currently it isn;t very useful, which is why engineering effort is available from various companies. Lets take the opportunity to fix it properly.

There is a real need to bootstrap Debian from sources when doing new ports or flavours. Every new architecture or optimisation flavour needs to do this at least once, and making it easier than the current 'really very hard' would be great. It is also very useful for cross-compiling to new or non-self-hosted architectures, and for a genuinely new arch at least part of the system (toolchains+build-essential) has to be cross-built until there is enough to become self-hosting. Recent new bootstraps have been done for sh4, armhf and avr32. More are coming down the line. Subarch flavoured rebuilds (e.g. to optimise for a particular CPU) are particularly useful on ARM and MIPS architectures. Bootstrapping is also helpful when bringing a lagged architecture up to date. Currently people tend to use non-debian tools (such as Yocto/gentoo/OpenEmbedded) to get a basic rootfs image of the target arch/ABI then do native building within that. This works but needs a great deal of manual loop-breaking and we really out to be able to bootstrap our own OS. Putting the necessary bootstrapping metadata and build rules into the packages themselves in an orderly fashion enables the info to be maintained easily. QA tests to report on breakage will help enormously here. It also makes for a repeatable and deterministic process. This work does need build-system and policy changes, to allow 'staged' or 'bootstrap' package builds which untangle cyclic dependencies. And then preferably some automated analysis, QA and buildd tools.