Alpine Linux - User contributions [en]

User:Darkfader/distcc

2026-04-01T00:36:15Z

Darkfader: /* ccache and memcached */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and distributed caches ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

This patch is described to be solving the interactions between ccache and distcc to work much better
https://patch-diff.githubusercontent.com/raw/ccache/ccache/pull/301.diff

patch status is, unclear

zephyrOS runs KeyDB backend for distcc at scale since a while.
They picked this software to not need a large ram redis server.
One could alternatively use one system if it has enough ram (rumor has it DDR3 ECC is cheap)

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
links for further review:

- https://repology.org/project/selinux-distcc/versions
- https://gitlab.com/liguros/liguros-repo/-/tree/stable/sec-policy/selinux-distcc

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
There are references to some 'forever free' tier, also, but I could not yet find the terms and conditions for having a 'forever free thing'. it also seems you need a windows-based 'coordinator' host.
opinion: generally if there's some professional context with high (business) pressure for faster builds I would check that out, and then without time pressure build a great distcc setup for the long-term.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-04-01T00:34:32Z

Darkfader: /* ccache and memcached */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

This patch is described to be solving the interactions between ccache and distcc to work much better
https://patch-diff.githubusercontent.com/raw/ccache/ccache/pull/301.diff

patch status is, unclear

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
links for further review:

- https://repology.org/project/selinux-distcc/versions
- https://gitlab.com/liguros/liguros-repo/-/tree/stable/sec-policy/selinux-distcc

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
There are references to some 'forever free' tier, also, but I could not yet find the terms and conditions for having a 'forever free thing'. it also seems you need a windows-based 'coordinator' host.
opinion: generally if there's some professional context with high (business) pressure for faster builds I would check that out, and then without time pressure build a great distcc setup for the long-term.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

Talk:Custom Kernel

2026-03-29T16:09:21Z

Darkfader: /* build_NAME Answer */

Needs to be adjusted for other archs

— Preceding [[Help:Signature|unsigned]] comment added by [[User:Orson Teodoro|Orson Teodoro]] ([[User talk:Orson Teodoro|{{int:talkpagelinktext}}]] • [[Special:Contributions/Orson Teodoro|{{int:contribslink}}]]) 03:36, 4 March 2018‎

== Locations/paths of build config files ==

My goal is to build the Virt (not Standard/LTS) kernel using the defaults for Virt, and I don't understand which files are needed in exactly which paths.

I had difficulty understanding the excerpt below. Is anyone willing to assist me in understanding it? For example, where/what is the "linux-4.15" folder?

:''When you are done with your edits either by editing directly the APKBUILD and copying the lts.ARCH.config as .config in the linux-4.15 folder. You will then move the .config back overriding the lts.ARCH.config generated by make menuconfig.''

— Preceding [[Help:Signature|unsigned]] comment added by [[User:-anthumchris-|-anthumchris-]] ([[User talk:-anthumchris-|{{int:talkpagelinktext}}]] • [[Special:Contributions/-anthumchris-|{{int:contribslink}}]]) 17:34, 23 November 2024‎

== build_NAME ==

''After you are done using the menu in the build-NAME folder by doing make menuconfig''

It is really not clear what is meant by "the build-NAME folder" here.

== Locations/paths of build config files Answer ==

Usually these are the directories you would look for in linux-lts apk package development (replace "x86_64" with different $ARCH if needed) (replace $YOUR_WORK_DIR to which directory you downloaded aports to),

Below should contain <code>APKBUILD</code>, <code>lts.x86_64.config</code>, <code>virt.x86_64.config</code> and <code>.patch</code> files (and more $ARCH's by default, but to keep it simple, I will just state the x86_64 files)
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts}}

ALL <code>.patch</code> files that are in $YOUR_WORK_DIR/aports/main/linux-lts are also in below
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src}}

Below is a tar compressed linux file that decompresses into linux-$VERSION
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION.tar.gz}}

{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION}}
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/build-lts.x86_64}}
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/build-virt.x86_64}}

In the "<code>APKBUILD</code>" file, change this "<code>source</code>" line to this:
{{cat|$YOUR_WORK_DIR/aports/main/linux-lts/APKBUILD|...
source{{=}}"https://cdn.kernel.org/pub/linux/kernel/v${pkgver%%.*}.x/linux-$_kernver.tar.xz
0001-powerpc-boot-wrapper-Add-z-notext-flag-for-ppc64le.patch
0002-x86-Compress-vmlinux-with-zstd-19-instead-of-22.patch
0003-kexec-add-kexec_load_disabled-boot-option.patch
0004-objtool-respect-AWK-setting.patch
0005-powerpc-config-defang-gcc-check-for-stack-protector-.patch

lts.x86_64.config

virt.x86_64.config
"
...}}

do <code>make menuconfig</code> within <code>$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION</code> and copy the .config file to <code>$YOUR_WORK_DIR/aports/main/linux-lts/lts.x86_64.config</code> and/or <code>$YOUR_WORK_DIR/aports/main/linux-lts/virt.x86_64.config</code>

— Preceding [[Help:Signature|unsigned]] comment added by [[User:Pursuable1652|Pursuable1652]] ([[User talk:Pursuable1652|{{int:talkpagelinktext}}]] • [[Special:Contributions/Pursuable1652|{{int:contribslink}}]]) 16:55, 25 December 2024‎

== build_NAME Answer ==

''After you are done using the menu in the build-NAME folder by doing make menuconfig''

the "build-NAME" folder should actually be <code>$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION</code>

— Preceding [[Help:Signature|unsigned]] comment added by [[User:Pursuable1652|Pursuable1652]] ([[User talk:Pursuable1652|{{int:talkpagelinktext}}]] • [[Special:Contributions/Pursuable1652|{{int:contribslink}}]]) 16:55, 25 December 2024‎

=====

I also need to run an abuild clean after doing the menuconfig.
those parts are still muddy.

[[User:Darkfader|Darkfader]] ([[User talk:Darkfader|talk]]) 16:09, 29 March 2026 (UTC)

== Page ready for regular release? ==
Perhaps this [[Custom Kernel]] page is now ready for regular use, and the "This material is work-in-progress" banner can be removed, unless there is a shortcoming found. This draft state has held since it was helpfully created in 2014.
Thanks for kind updates by [[User:Prabuanand|Prabuanand]], [[User:Pawciobiel|Pawciobiel]], [[User:Teopp7|Teopp7]], [[User:Fossdd|Fossdd]], [[User:Anthumchris|Anthumchris]], [[User:Gsora|Gsora]], [[User:Zcrayfish|Zcrayfish]], [[User:Omni|Omni]], [[User:Nabbi|Nabbi]], [[User:Psykose|Psykose]], [[User:Egberts|Egberts]] and thanks also for kind contributions by others that have not been active on the wiki for over five years. [[User:John3-16|John3-16]] ([[User talk:John3-16|talk]]) 19:22, 30 December 2025 (UTC)
:Further to one month without objections, having tagged recent contributors, the ''"Work-in-progress"'' flag is being removed. The contributor who kindly initiated this useful page duly placed the flag in the original [https://wiki.alpinelinux.org/w/index.php?title=Custom_Kernel&oldid=10201 draft version] in 2014 and last contributed to the wiki that year. [[User:John3-16|John3-16]] ([[User talk:John3-16|talk]]) 02:02, 1 February 2026 (UTC)

=== kexec ===

the kernel and tools now have better exploit-kernel protection and there's actually been some ransomware actors using kexec to load bad things[tm]

is there a way to sign kernels on alpine? If so, maybe we should document that and link to it here.
CONFIG_KECEC_SIG etc.

[[User:Darkfader|Darkfader]] ([[User talk:Darkfader|talk]]) 16:07, 29 March 2026 (UTC)

Talk:Custom Kernel

2026-03-29T16:07:42Z

Darkfader: sigs

Needs to be adjusted for other archs

— Preceding [[Help:Signature|unsigned]] comment added by [[User:Orson Teodoro|Orson Teodoro]] ([[User talk:Orson Teodoro|{{int:talkpagelinktext}}]] • [[Special:Contributions/Orson Teodoro|{{int:contribslink}}]]) 03:36, 4 March 2018‎

== Locations/paths of build config files ==

My goal is to build the Virt (not Standard/LTS) kernel using the defaults for Virt, and I don't understand which files are needed in exactly which paths.

I had difficulty understanding the excerpt below. Is anyone willing to assist me in understanding it? For example, where/what is the "linux-4.15" folder?

:''When you are done with your edits either by editing directly the APKBUILD and copying the lts.ARCH.config as .config in the linux-4.15 folder. You will then move the .config back overriding the lts.ARCH.config generated by make menuconfig.''

— Preceding [[Help:Signature|unsigned]] comment added by [[User:-anthumchris-|-anthumchris-]] ([[User talk:-anthumchris-|{{int:talkpagelinktext}}]] • [[Special:Contributions/-anthumchris-|{{int:contribslink}}]]) 17:34, 23 November 2024‎

== build_NAME ==

''After you are done using the menu in the build-NAME folder by doing make menuconfig''

It is really not clear what is meant by "the build-NAME folder" here.

== Locations/paths of build config files Answer ==

Usually these are the directories you would look for in linux-lts apk package development (replace "x86_64" with different $ARCH if needed) (replace $YOUR_WORK_DIR to which directory you downloaded aports to),

Below should contain <code>APKBUILD</code>, <code>lts.x86_64.config</code>, <code>virt.x86_64.config</code> and <code>.patch</code> files (and more $ARCH's by default, but to keep it simple, I will just state the x86_64 files)
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts}}

ALL <code>.patch</code> files that are in $YOUR_WORK_DIR/aports/main/linux-lts are also in below
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src}}

Below is a tar compressed linux file that decompresses into linux-$VERSION
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION.tar.gz}}

{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION}}
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/build-lts.x86_64}}
{{cmd|$YOUR_WORK_DIR/aports/main/linux-lts/src/build-virt.x86_64}}

In the "<code>APKBUILD</code>" file, change this "<code>source</code>" line to this:
{{cat|$YOUR_WORK_DIR/aports/main/linux-lts/APKBUILD|...
source{{=}}"https://cdn.kernel.org/pub/linux/kernel/v${pkgver%%.*}.x/linux-$_kernver.tar.xz
0001-powerpc-boot-wrapper-Add-z-notext-flag-for-ppc64le.patch
0002-x86-Compress-vmlinux-with-zstd-19-instead-of-22.patch
0003-kexec-add-kexec_load_disabled-boot-option.patch
0004-objtool-respect-AWK-setting.patch
0005-powerpc-config-defang-gcc-check-for-stack-protector-.patch

lts.x86_64.config

virt.x86_64.config
"
...}}

do <code>make menuconfig</code> within <code>$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION</code> and copy the .config file to <code>$YOUR_WORK_DIR/aports/main/linux-lts/lts.x86_64.config</code> and/or <code>$YOUR_WORK_DIR/aports/main/linux-lts/virt.x86_64.config</code>

— Preceding [[Help:Signature|unsigned]] comment added by [[User:Pursuable1652|Pursuable1652]] ([[User talk:Pursuable1652|{{int:talkpagelinktext}}]] • [[Special:Contributions/Pursuable1652|{{int:contribslink}}]]) 16:55, 25 December 2024‎

== build_NAME Answer ==

''After you are done using the menu in the build-NAME folder by doing make menuconfig''

the "build-NAME" folder should actually be <code>$YOUR_WORK_DIR/aports/main/linux-lts/src/linux-$VERSION</code>

— Preceding [[Help:Signature|unsigned]] comment added by [[User:Pursuable1652|Pursuable1652]] ([[User talk:Pursuable1652|{{int:talkpagelinktext}}]] • [[Special:Contributions/Pursuable1652|{{int:contribslink}}]]) 16:55, 25 December 2024‎

== Page ready for regular release? ==
Perhaps this [[Custom Kernel]] page is now ready for regular use, and the "This material is work-in-progress" banner can be removed, unless there is a shortcoming found. This draft state has held since it was helpfully created in 2014.
Thanks for kind updates by [[User:Prabuanand|Prabuanand]], [[User:Pawciobiel|Pawciobiel]], [[User:Teopp7|Teopp7]], [[User:Fossdd|Fossdd]], [[User:Anthumchris|Anthumchris]], [[User:Gsora|Gsora]], [[User:Zcrayfish|Zcrayfish]], [[User:Omni|Omni]], [[User:Nabbi|Nabbi]], [[User:Psykose|Psykose]], [[User:Egberts|Egberts]] and thanks also for kind contributions by others that have not been active on the wiki for over five years. [[User:John3-16|John3-16]] ([[User talk:John3-16|talk]]) 19:22, 30 December 2025 (UTC)
:Further to one month without objections, having tagged recent contributors, the ''"Work-in-progress"'' flag is being removed. The contributor who kindly initiated this useful page duly placed the flag in the original [https://wiki.alpinelinux.org/w/index.php?title=Custom_Kernel&oldid=10201 draft version] in 2014 and last contributed to the wiki that year. [[User:John3-16|John3-16]] ([[User talk:John3-16|talk]]) 02:02, 1 February 2026 (UTC)

=== kexec ===

the kernel and tools now have better exploit-kernel protection and there's actually been some ransomware actors using kexec to load bad things[tm]

is there a way to sign kernels on alpine? If so, maybe we should document that and link to it here.
CONFIG_KECEC_SIG etc.

[[User:Darkfader|Darkfader]] ([[User talk:Darkfader|talk]]) 16:07, 29 March 2026 (UTC)

Custom Kernel

2026-03-29T16:03:09Z

Darkfader: /* Configuring kernel */

This process of building a '''custom configured kernel''' assumes you are running on Alpine Linux utilizing abuild & aports.

== But why? ==

You want to build a custom kernel to enable experimental hardware or features or outdated hardware, to reduce bloat further, to tune the kernel to the hardware.

The lts kernel for most Alpine ARCHs uses defaults to balance throughput at the expense of some responsiveness, and support for many devices. You can tweak the kernel for desktop use and low latency and responsiveness.

You should disable modules to increase security. By default, Alpine will install modules but not disable most of them. Disabling modules will reduce an DMA attack but not eliminate it completely. If you have a newer processor with VT-d, you can mitigate as long as you:

Leave <code>CONFIG_INTEL_IOMMU_DEFAULT_ON=y</code> or pass <code>intel_iommu=on</code> as a kernel parameter and disable kernel logging so the attacker doesn't gain DMAR address information through dmesg.[https://blog.frizk.net/2016/11/disable-virtualization-based-security.html] Also remove references to the kernel version to calculate the IOMMU addresses.[https://link.springer.com/content/pdf/10.1186/s13173-017-0066-7.pdf]

You may also want to harden your kernel by adding at least some of the config changes recommended by <code>kernel-hardening-checker</code> [https://github.com/a13xp0p0v/kernel-hardening-checker/]

To increase the security of the boot process, if you have a TPM, you could set <code>CONFIG_INTEL_TXT=y</code> (Enable Intel(R) Trusted Execution Technology (Intel(R) TXT)) (which is not enabled in the hardened kernel by default), then you would need the SINIT module (provided only by Intel)[https://software.intel.com/en-us/articles/intel-trusted-execution-technology], a possibly compiled TrustedGrub2[https://github.com/Rohde-Schwarz-Cybersecurity/TrustedGRUB2], trousers[https://sourceforge.net/projects/trousers/?source=navbar], tboot[https://sourceforge.net/projects/tboot/]. These packages are not in aports and it is unknown if these tools work on musl. It's not recommended for Edge. Also, there would be trigger packages to generate hashes for the kernel and the mkinitfs updates.

== Setting up the Alpine Build System ==

First, you need to follow the steps in [[Creating_an_Alpine_package#Setup_your_system_and_account|Setup your system and account for building packages]]. You also need to configure your {{path|/etc/apk/repositories}} so that they search locally for your apks. See [[Creating_an_Alpine_package#Testing_the_package_locally|Testing the package locally]] for details.

After setting up accounts and repos, change your shell's current working directory to '''aports''' that you just cloned.

{{cmd|$ cd aports}}

== Working with aports ==

We will try using an existing lts kernel just tweaking the {{path|lts.ARCH.config}} file.

=== Switching to the proper release version ===

You need to switch to the proper branch that matches the release so that the kernel compiles against the dependencies properly.

{| cellpadding="5" border="1" class="wikitable"
|-
! Alpine version
! Remote branch
|-
| Edge
| master
|-
|{{#expr:{{AlpineLatest}}}}
|{{#expr:{{AlpineLatest}}}}-stable
|-
|}

The following is required to get access to the {{path|APKBUILD}} released for that version of Alpine and which you will create a commit for.

If you are on {{#expr:{{AlpineLatest}}}} do:

{{cmd|$ git checkout -b {{#expr:{{AlpineLatest}}}}-stable origin/{{#expr:{{AlpineLatest}}}}-stable}}

If you are on Edge do:

{{cmd|$ git checkout master}}

=== Creating your config ===

You can use {{pkg|linux-lts}} but what you should do is create a local branch by doing:

For <var>linux-lts</var> kernel on Alpine {{#expr:{{AlpineLatest}}}}:

{{cmd|$ git checkout -b my-custom-kernel origin/{{#expr:{{AlpineLatest}}}}-stable}}

For Alpine Edge:

{{cmd|$ git checkout -b my-custom-kernel}}

Doing it this way, you do less work in maintaining. All you need to do is keep ''master'' or ''{{#expr:{{AlpineLatest}}}}-stable'' in sync[https://help.github.com/articles/syncing-a-fork/][https://help.github.com/articles/configuring-a-remote-for-a-fork/] and merge any conflicts.

First switch to the branch by doing:{{cmd|$ git checkout my-custom-kernel}}
Then, you need to navigate to the {{path|main/linux-lts}} folder where you should see a APKBUILD and some config- files.

It is a good idea to use custom `FLAVOR` in order to change the name of your kernel and package (custom flavor or revision) so you newly build kernel will not override the existing linux-lts and it's modules. Flavor build will only use your config and omit files that are not for your cpu architectures or virt* configs.(Virt configs provide optimized kernel configurations for running Alpine as a guest in virtual machines.)
This will also speed up the process of applying kernel patches by applying it for your architecture only. (Obviously ARCH in the following example is whatever architecture x86, x86_64...you use.)

First make a copy of your ARCH config file:
<pre>
cp lts.x86_64.config lts-my_custom.x86_64.config
</pre>

Edit your custom config file and add your changes. When you are done with your edits you need to run checksum to update APKBUILD:

<code>FLAVOR=lts-my_custom abuild checksum</code>

Then commit your changes:

<code>git commit -a -m "Enabled these options ...."</code>

You do this so that git can keep your code separate from Alpine's and so your changes float forward between kernel updates.

== Adding custom patches ==

Custom patches should be added to ''sources=''.

After you added the URL, you need to produce a checksum by doing <code>abuild checksum</code>.

The custom patches may not be autopatched, due to being distributed as an archive or different patch level, so you need to define what to do with it in the prepare().

== Configuring kernel ==

There are multiple ways to prepare and change kernel config. The basic one is to just edit your custom config file. You can also attempt to create a config tailored for your currently running system and modules.

Extract your kernel and apply the standard patches:

<pre>
abuild fetch && abuild unpack && abuild prepare
</pre>

To create a kernel config based on your currently running PC, use:
<pre>
make localmodconfig
</pre>
This command reads your current kernel's configuration from /proc/config.gz (if available) or /boot/config-* and creates a .config file that includes only the modules currently loaded on your system. This results in a minimal configuration tailored to your hardware.
Alternatively, you can use:
<pre>make localyesconfig</pre>
This is similar to ''localmodconfig'', but instead of building features as modules (m), it builds them directly into the kernel (y), which can result in a larger kernel image but eliminates the need for separate module files.
Note: Before running either command, make sure you have all the hardware you want to support actively in use (USB devices plugged in, network cards active, etc.) so their modules are loaded and included in the configuration.

The alternative is to use the kernel configuration menu in the build-NAME folder, but before you do that you need to <code>sudo apk add {{pkg|ncurses-dev}}</code>

After you are done using the menu in the build-NAME folder by doing:

<pre>make menuconfig</pre>

You want to remove <code>ncurses-dev</code>. When you are done, it will be stored in ''.config'' which you need to again override the {{path|lts.ARCH.config}} file.

<pre>
cp .config ../../lts-my-kernel.x86_64.config
</pre>

When you are done updating the {{path|config-NAME.ARCH}}, you need to update sums:
<pre>
abuild checksum
</pre>

The options in the kernel config are typically defaults. If your device is old, it may be set to n by default.

=== Vanilla targets and tuning ===

{|cellpadding="5" border="1" class="wikitable"
!ARCH
!Processor Type / CPU Selection / System Type
!Code Generation / Instruction Extensions
!Timer Frequency
!Preemption Model
!Bitness
|-
|s390x
|IBM zEnterprise 114 and 196
|IBM zBC12 and zEC12 (<code>-march=zEC12 -mtune=zEC12</code>)
|1000 Hz
|No Forced Preemption (Server)
|64
|-
|ppc64le
|Server processors
|POWER8 (<code>-mcpu=power8</code>), AltiVec (<code>-Wa,-maltivec</code> to assembler or <code>-maltivec -mabi=altivec</code>), VSX
|1000 HZ
|No Forced Preemption (Server)
|64
|-
|ppc
|
512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx
* Apple PowerMac based machines
|AltiVec (<code>-Wa,-maltivec</code> to assembler or <code>-maltivec -mabi=altivec</code>) on >=74xx
|1000 HZ
|No Forced Preemption (Server)
|32
|-
|x86_64
|Generic-x86-64
|(-mtune=generic ; SIMD assembly modules enabled based on simple compile test and/or presence of CPU flag)
|1000 HZ
|Voluntary Kernel Preemption (Desktop)
|64
|-
|x86
|586/K5/5x86/6x86/6x86MX
|(-mtune=generic ; SIMD assembly modules enabled based on simple compile test and/or presence of CPU flag)
|1000 HZ
|Voluntary Kernel Preemption (Desktop)
|32
|-
|armv7
|
* ARMv7 based platforms (Cortex-A, PJ4, Scorpion, Krait)
* Freescale i.MX family -- Cortex A (i.MX51, i.MX53, i.MX6 Quad/DualLite, i.MX6 SoloLite, i.MX6 SoloX, i.MX6 UltraLite, i.MX7 Dual)
* Qualcomm -- (MSM8X60, MSM8960, MSM8974)
* Allwinner SoCs -- (A10 (sun4i), A10s / A13 (sun5i), A31 (sun6i), A20 (sun7i), sun8i Family, (sun9i))
* ARM Ldt Versatile Express family --
|Either <code>-march=armv7-a</code> or <code>-march=armv5t -Wa,-march=armv7-a</code> based on a compile test. <code>-mfpu=vfp</code>
|1000 Hz
|Voluntary Kernel Preemption (Desktop)
|32
|-
|aarch64
|
* Allwinner sunxi 64-bit SoC Family
* Broadcom BCM2835 family
* Marvell Berlin SoC Family
* ARMv8 based Samsung Exynos SoC family
* ARMv8 based Freescale Layerscape SoC family
* Hisilicon SoC Family
* Mediatek MT65xx & MT81xx ARMv8 SoC
* Marvell EBU SoC Family
* Qualcomm Platforms
* Rockchip Platforms
* AMD Seattle SoC Family
* Altera's Stratix 10 SoCFPGA Family
* NVIDIA Tegra SoC Family
* Spreadtrum SoC platform
* Cavium Inc. Thunder SoC Family
* ARMv8 software model (Versatile Express)
* AppliedMicro X-Gene SOC Family
* Xilinx ZynqMP Family
|
|1000 HZ
|Voluntary Kernel Preemption (Desktop)
|64
|-
|}
If you do desktop multitasking, you may want to switch to Voluntary Kernel Preemption (Desktop) or Preemptible Kernel (Low-Latency Desktop) and up the Timer Frequency. If you run a dedicated render farm node or a dedicated bitcoin miner use No Forced Preemption (Server) and decrease the Timer Frequency.

Optimized modules (most are already compiled as modules):
* raid6 -- altivec, avx512, ssse3, avx2, mmx, sse, sse2, neon
* some operations of raid5 -- mmx (32 bit), sse (64 bit), avx
For Kernel API:
* 32-bit memcpy -- 3dnow
* 32-bit memory page clearing and copying -- sse (Athlon/K7 only), mmx
From x86/crypto, arm/crypto, powerpc/crypto:
* CAMELLIA -- avx2, avx, aes-ni
* CHACHA20 -- avx2, neon
* CAST5 -- avx
* CAST6 -- avx
* TWOFISH -- avx
* SERPENT -- avx2, avx, sse2
* SHA1 -- avx2, ssse3, neon, spe
* SHA2 -- avx2
* SHA256 -- ssse3, neon, spe
* SHA512 -- avx2, ssse3, neon
* POLY1305 -- avx2
* GHASH -- pclmulqdq (part of aes-ni), vmx (power8)
* AES -- aes-ni, neon, vmx (power8), spe
* CRC32 -- pclmulqdq, sse, neon, vmx (power8)
* CRCT10DIF -- pclmulqdq, sse, neon, vmx (power8)

=== Fast reboots with kexec ===
{{main|kexec}}

If you want to reboot the kernel fast avoiding the POST test, you need {{ic|doas apk add {{pkg|kexec-tools}}}} and enable kexec in the kernel:

Processor type and features
[*] kexec system call

=== Hibernation to prevent data loss ===

Power management and ACPI options
[*] Hibernation (aka 'suspend to disk')

Hibernation should be used if you have a laptop. You don't want the laptop to suddenly shut off resulting in data loss, you want it to save your work based on a percentage of battery life (this requires special script). When hibernation resumes, should lock and ask for credentials. Depending on your needs, the hibernated image can be encrypted/decrypted which again requires additional customization to scripts.

Hibernation with an unsanitized swap file is generally insecure because data and unlocked memory pages are swapped out in plaintext. To increase the security either disable swap or use an encrypted swap. The swap file/partition is typically used as the hibernation resume image.

== Building ==

Before building, make sure you have ccache installed. This should reduce compile time on multiple builds.
You may also want to read up on compile flags - perhaps you want your build to be optimized for speed (or size which is the default).
Review <code>/etc/abuild.conf</code> and set:
<pre>
PACKAGER_PRIVKEY="/home/MY_USER_HERE/.abuild/my-email@mydomain.com-ID.rsa"
USE_CCACHE=1
</pre>
Please note that kernel build process may not use some of the settings you set in `abuild.conf` so in order to customize compiler or linker and/or it's flags you may need to edit APKBUILD.
If you have your config ready, first try building with:

<pre>
FLAVOR=lts-my_custom abuild -rK 2>&1 | tee build1.log
</pre>

This will install most of the dependencies and keep buildtime temp dirs and files (srcdir/pkgdir/deps).

If it complains about a dependency like {{pkg|elfutils-dev}} use <code>-rKd</code>.
Then, when it prompts for values for new found config options just hold enter till it starts compiling the kernel.
Unless you use <code>FLAVOR</code> or removed or commented out <code>virt.*.config</code> configs in APKBUILD there should be two sets one for -lts and the other for the -virt.
Just {{Key|Ctrl}}+{{Key|C}} out of the compilation process after
the second set so you can further customize the config.
Then you go into the {{path|src/linux-VER}} and edit the config file.
Copy the {{path|.config}} file overriding the {{path|lts.ARCH.config}} in the srcdir.

== Installing ==

If the build was successful the kernel packages are located in <code>~/packages/main/ARCH</code>
You probably already know how to install a package in your Alpine Linux...

== Testing ==

In case your new kernel may be missing a module and can't boot it is generally a good idea to keep the default <code>linux-lts</code>, so make sure you have it installed using the command: {{Cmd|# apk add {{pkg|linux-lts}}}}

Normally during the installation using with [[apk]], there are tools that will update initramfs and a boot loader automatically for you. For easier debugging you may want to change boot loader config and remove <code>quiet</code> from linux command line in {{Path|/etc/default/grub}} or {{Path|/etc/update-extlinux.conf}}.
Or perhaps you want to change other kernel options in {{Path|/etc/mkinitfs/mkinitfs.conf}}. Please remember to generate initramfs and update your bootloader manually, eg:
{{Cmd|<nowiki># mkinitfs
# grub-mkconfig -o /boot/grub/grub.cfg</nowiki>}}

In case something goes wrong with a boot process it is also a good idea to have a bootable rescue Alpine USB ready.

Once you have the default lts kernel and rescue USB <code>reboot</code> the computer.

If you are curious about correctness testing, some kernel modules or components do preform self tests at the beginning of the boot process. The tools may have test suites that you run with the make command.

== See Also ==
* [[Kernels]]
* [https://wiki.archlinux.org/title/Kernel Archwiki Kernels]
* [https://wiki.gentoo.org/wiki/Kernel Gentoo Wiki Kernel]
* [https://wiki.gentoo.org/wiki/Kernel/Configuration Gentoo Wiki Kernel Configuration]
* [[How to build the Alpine Linux kernel]]
* [[Kernel_live_patching|Kernel Live Patching (KLP)]]

[[Category:Kernel]]

Custom Kernel

2026-03-29T15:31:10Z

Darkfader: /* Configuring kernel */

This process of building a '''custom configured kernel''' assumes you are running on Alpine Linux utilizing abuild & aports.

== But why? ==

You want to build a custom kernel to enable experimental hardware or features or outdated hardware, to reduce bloat further, to tune the kernel to the hardware.

The lts kernel for most Alpine ARCHs uses defaults to balance throughput at the expense of some responsiveness, and support for many devices. You can tweak the kernel for desktop use and low latency and responsiveness.

You should disable modules to increase security. By default, Alpine will install modules but not disable most of them. Disabling modules will reduce an DMA attack but not eliminate it completely. If you have a newer processor with VT-d, you can mitigate as long as you:

Leave <code>CONFIG_INTEL_IOMMU_DEFAULT_ON=y</code> or pass <code>intel_iommu=on</code> as a kernel parameter and disable kernel logging so the attacker doesn't gain DMAR address information through dmesg.[https://blog.frizk.net/2016/11/disable-virtualization-based-security.html] Also remove references to the kernel version to calculate the IOMMU addresses.[https://link.springer.com/content/pdf/10.1186/s13173-017-0066-7.pdf]

You may also want to harden your kernel by adding at least some of the config changes recommended by <code>kernel-hardening-checker</code> [https://github.com/a13xp0p0v/kernel-hardening-checker/]

To increase the security of the boot process, if you have a TPM, you could set <code>CONFIG_INTEL_TXT=y</code> (Enable Intel(R) Trusted Execution Technology (Intel(R) TXT)) (which is not enabled in the hardened kernel by default), then you would need the SINIT module (provided only by Intel)[https://software.intel.com/en-us/articles/intel-trusted-execution-technology], a possibly compiled TrustedGrub2[https://github.com/Rohde-Schwarz-Cybersecurity/TrustedGRUB2], trousers[https://sourceforge.net/projects/trousers/?source=navbar], tboot[https://sourceforge.net/projects/tboot/]. These packages are not in aports and it is unknown if these tools work on musl. It's not recommended for Edge. Also, there would be trigger packages to generate hashes for the kernel and the mkinitfs updates.

== Setting up the Alpine Build System ==

First, you need to follow the steps in [[Creating_an_Alpine_package#Setup_your_system_and_account|Setup your system and account for building packages]]. You also need to configure your {{path|/etc/apk/repositories}} so that they search locally for your apks. See [[Creating_an_Alpine_package#Testing_the_package_locally|Testing the package locally]] for details.

After setting up accounts and repos, change your shell's current working directory to '''aports''' that you just cloned.

{{cmd|$ cd aports}}

== Working with aports ==

We will try using an existing lts kernel just tweaking the {{path|lts.ARCH.config}} file.

=== Switching to the proper release version ===

You need to switch to the proper branch that matches the release so that the kernel compiles against the dependencies properly.

{| cellpadding="5" border="1" class="wikitable"
|-
! Alpine version
! Remote branch
|-
| Edge
| master
|-
|{{#expr:{{AlpineLatest}}}}
|{{#expr:{{AlpineLatest}}}}-stable
|-
|}

The following is required to get access to the {{path|APKBUILD}} released for that version of Alpine and which you will create a commit for.

If you are on {{#expr:{{AlpineLatest}}}} do:

{{cmd|$ git checkout -b {{#expr:{{AlpineLatest}}}}-stable origin/{{#expr:{{AlpineLatest}}}}-stable}}

If you are on Edge do:

{{cmd|$ git checkout master}}

=== Creating your config ===

You can use {{pkg|linux-lts}} but what you should do is create a local branch by doing:

For <var>linux-lts</var> kernel on Alpine {{#expr:{{AlpineLatest}}}}:

{{cmd|$ git checkout -b my-custom-kernel origin/{{#expr:{{AlpineLatest}}}}-stable}}

For Alpine Edge:

{{cmd|$ git checkout -b my-custom-kernel}}

Doing it this way, you do less work in maintaining. All you need to do is keep ''master'' or ''{{#expr:{{AlpineLatest}}}}-stable'' in sync[https://help.github.com/articles/syncing-a-fork/][https://help.github.com/articles/configuring-a-remote-for-a-fork/] and merge any conflicts.

First switch to the branch by doing:{{cmd|$ git checkout my-custom-kernel}}
Then, you need to navigate to the {{path|main/linux-lts}} folder where you should see a APKBUILD and some config- files.

It is a good idea to use custom `FLAVOR` in order to change the name of your kernel and package (custom flavor or revision) so you newly build kernel will not override the existing linux-lts and it's modules. Flavor build will only use your config and omit files that are not for your cpu architectures or virt* configs.(Virt configs provide optimized kernel configurations for running Alpine as a guest in virtual machines.)
This will also speed up the process of applying kernel patches by applying it for your architecture only. (Obviously ARCH in the following example is whatever architecture x86, x86_64...you use.)

First make a copy of your ARCH config file:
<pre>
cp lts.x86_64.config lts-my_custom.x86_64.config
</pre>

Edit your custom config file and add your changes. When you are done with your edits you need to run checksum to update APKBUILD:

<code>FLAVOR=lts-my_custom abuild checksum</code>

Then commit your changes:

<code>git commit -a -m "Enabled these options ...."</code>

You do this so that git can keep your code separate from Alpine's and so your changes float forward between kernel updates.

== Adding custom patches ==

Custom patches should be added to ''sources=''.

After you added the URL, you need to produce a checksum by doing <code>abuild checksum</code>.

The custom patches may not be autopatched, due to being distributed as an archive or different patch level, so you need to define what to do with it in the prepare().

== Configuring kernel ==

There are multiple ways to prepare and change kernel config. The basic one is to just edit your custom config file. You can also attempt to create a config tailored for your currently running system and modules.

Extract your kernel and apply the standard patches:

<pre>
abuild fetch && abuild unpack && abuild prepare
</pre>

To create a kernel config based on your currently running PC, use:
<pre>
make localmodconfig
</pre>
This command reads your current kernel's configuration from /proc/config.gz (if available) or /boot/config-* and creates a .config file that includes only the modules currently loaded on your system. This results in a minimal configuration tailored to your hardware.
Alternatively, you can use:
<pre>make localyesconfig</pre>
This is similar to ''localmodconfig'', but instead of building features as modules (m), it builds them directly into the kernel (y), which can result in a larger kernel image but eliminates the need for separate module files.
Note: Before running either command, make sure you have all the hardware you want to support actively in use (USB devices plugged in, network cards active, etc.) so their modules are loaded and included in the configuration.

The alternative is to use the kernel configuration menu in the build-NAME folder, but before you do that you need to <code>sudo apk add {{pkg|ncurses-dev}}</code>

After you are done using the menu in the build-NAME folder by doing:

<pre>make menuconfig</pre>

You want to remove <code>ncurses-dev</code>. When you are done, it will be stored in ''.config'' which you need to again override the {{path|lts.ARCH.config}} file.

When you are done updating the {{path|config-NAME.ARCH}}, you need to update sums:
<pre>
abuild checksum
</pre>

The options in the kernel config are typically defaults. If your device is old, it may be set to n by default.

=== Vanilla targets and tuning ===

{|cellpadding="5" border="1" class="wikitable"
!ARCH
!Processor Type / CPU Selection / System Type
!Code Generation / Instruction Extensions
!Timer Frequency
!Preemption Model
!Bitness
|-
|s390x
|IBM zEnterprise 114 and 196
|IBM zBC12 and zEC12 (<code>-march=zEC12 -mtune=zEC12</code>)
|1000 Hz
|No Forced Preemption (Server)
|64
|-
|ppc64le
|Server processors
|POWER8 (<code>-mcpu=power8</code>), AltiVec (<code>-Wa,-maltivec</code> to assembler or <code>-maltivec -mabi=altivec</code>), VSX
|1000 HZ
|No Forced Preemption (Server)
|64
|-
|ppc
|
512x/52xx/6xx/7xx/74xx/82xx/83xx/86xx
* Apple PowerMac based machines
|AltiVec (<code>-Wa,-maltivec</code> to assembler or <code>-maltivec -mabi=altivec</code>) on >=74xx
|1000 HZ
|No Forced Preemption (Server)
|32
|-
|x86_64
|Generic-x86-64
|(-mtune=generic ; SIMD assembly modules enabled based on simple compile test and/or presence of CPU flag)
|1000 HZ
|Voluntary Kernel Preemption (Desktop)
|64
|-
|x86
|586/K5/5x86/6x86/6x86MX
|(-mtune=generic ; SIMD assembly modules enabled based on simple compile test and/or presence of CPU flag)
|1000 HZ
|Voluntary Kernel Preemption (Desktop)
|32
|-
|armv7
|
* ARMv7 based platforms (Cortex-A, PJ4, Scorpion, Krait)
* Freescale i.MX family -- Cortex A (i.MX51, i.MX53, i.MX6 Quad/DualLite, i.MX6 SoloLite, i.MX6 SoloX, i.MX6 UltraLite, i.MX7 Dual)
* Qualcomm -- (MSM8X60, MSM8960, MSM8974)
* Allwinner SoCs -- (A10 (sun4i), A10s / A13 (sun5i), A31 (sun6i), A20 (sun7i), sun8i Family, (sun9i))
* ARM Ldt Versatile Express family --
|Either <code>-march=armv7-a</code> or <code>-march=armv5t -Wa,-march=armv7-a</code> based on a compile test. <code>-mfpu=vfp</code>
|1000 Hz
|Voluntary Kernel Preemption (Desktop)
|32
|-
|aarch64
|
* Allwinner sunxi 64-bit SoC Family
* Broadcom BCM2835 family
* Marvell Berlin SoC Family
* ARMv8 based Samsung Exynos SoC family
* ARMv8 based Freescale Layerscape SoC family
* Hisilicon SoC Family
* Mediatek MT65xx & MT81xx ARMv8 SoC
* Marvell EBU SoC Family
* Qualcomm Platforms
* Rockchip Platforms
* AMD Seattle SoC Family
* Altera's Stratix 10 SoCFPGA Family
* NVIDIA Tegra SoC Family
* Spreadtrum SoC platform
* Cavium Inc. Thunder SoC Family
* ARMv8 software model (Versatile Express)
* AppliedMicro X-Gene SOC Family
* Xilinx ZynqMP Family
|
|1000 HZ
|Voluntary Kernel Preemption (Desktop)
|64
|-
|}
If you do desktop multitasking, you may want to switch to Voluntary Kernel Preemption (Desktop) or Preemptible Kernel (Low-Latency Desktop) and up the Timer Frequency. If you run a dedicated render farm node or a dedicated bitcoin miner use No Forced Preemption (Server) and decrease the Timer Frequency.

Optimized modules (most are already compiled as modules):
* raid6 -- altivec, avx512, ssse3, avx2, mmx, sse, sse2, neon
* some operations of raid5 -- mmx (32 bit), sse (64 bit), avx
For Kernel API:
* 32-bit memcpy -- 3dnow
* 32-bit memory page clearing and copying -- sse (Athlon/K7 only), mmx
From x86/crypto, arm/crypto, powerpc/crypto:
* CAMELLIA -- avx2, avx, aes-ni
* CHACHA20 -- avx2, neon
* CAST5 -- avx
* CAST6 -- avx
* TWOFISH -- avx
* SERPENT -- avx2, avx, sse2
* SHA1 -- avx2, ssse3, neon, spe
* SHA2 -- avx2
* SHA256 -- ssse3, neon, spe
* SHA512 -- avx2, ssse3, neon
* POLY1305 -- avx2
* GHASH -- pclmulqdq (part of aes-ni), vmx (power8)
* AES -- aes-ni, neon, vmx (power8), spe
* CRC32 -- pclmulqdq, sse, neon, vmx (power8)
* CRCT10DIF -- pclmulqdq, sse, neon, vmx (power8)

=== Fast reboots with kexec ===
{{main|kexec}}

If you want to reboot the kernel fast avoiding the POST test, you need {{ic|doas apk add {{pkg|kexec-tools}}}} and enable kexec in the kernel:

Processor type and features
[*] kexec system call

=== Hibernation to prevent data loss ===

Power management and ACPI options
[*] Hibernation (aka 'suspend to disk')

Hibernation should be used if you have a laptop. You don't want the laptop to suddenly shut off resulting in data loss, you want it to save your work based on a percentage of battery life (this requires special script). When hibernation resumes, should lock and ask for credentials. Depending on your needs, the hibernated image can be encrypted/decrypted which again requires additional customization to scripts.

Hibernation with an unsanitized swap file is generally insecure because data and unlocked memory pages are swapped out in plaintext. To increase the security either disable swap or use an encrypted swap. The swap file/partition is typically used as the hibernation resume image.

== Building ==

Before building, make sure you have ccache installed. This should reduce compile time on multiple builds.
You may also want to read up on compile flags - perhaps you want your build to be optimized for speed (or size which is the default).
Review <code>/etc/abuild.conf</code> and set:
<pre>
PACKAGER_PRIVKEY="/home/MY_USER_HERE/.abuild/my-email@mydomain.com-ID.rsa"
USE_CCACHE=1
</pre>
Please note that kernel build process may not use some of the settings you set in `abuild.conf` so in order to customize compiler or linker and/or it's flags you may need to edit APKBUILD.
If you have your config ready, first try building with:

<pre>
FLAVOR=lts-my_custom abuild -rK 2>&1 | tee build1.log
</pre>

This will install most of the dependencies and keep buildtime temp dirs and files (srcdir/pkgdir/deps).

If it complains about a dependency like {{pkg|elfutils-dev}} use <code>-rKd</code>.
Then, when it prompts for values for new found config options just hold enter till it starts compiling the kernel.
Unless you use <code>FLAVOR</code> or removed or commented out <code>virt.*.config</code> configs in APKBUILD there should be two sets one for -lts and the other for the -virt.
Just {{Key|Ctrl}}+{{Key|C}} out of the compilation process after
the second set so you can further customize the config.
Then you go into the {{path|src/linux-VER}} and edit the config file.
Copy the {{path|.config}} file overriding the {{path|lts.ARCH.config}} in the srcdir.

== Installing ==

If the build was successful the kernel packages are located in <code>~/packages/main/ARCH</code>
You probably already know how to install a package in your Alpine Linux...

== Testing ==

In case your new kernel may be missing a module and can't boot it is generally a good idea to keep the default <code>linux-lts</code>, so make sure you have it installed using the command: {{Cmd|# apk add {{pkg|linux-lts}}}}

Normally during the installation using with [[apk]], there are tools that will update initramfs and a boot loader automatically for you. For easier debugging you may want to change boot loader config and remove <code>quiet</code> from linux command line in {{Path|/etc/default/grub}} or {{Path|/etc/update-extlinux.conf}}.
Or perhaps you want to change other kernel options in {{Path|/etc/mkinitfs/mkinitfs.conf}}. Please remember to generate initramfs and update your bootloader manually, eg:
{{Cmd|<nowiki># mkinitfs
# grub-mkconfig -o /boot/grub/grub.cfg</nowiki>}}

In case something goes wrong with a boot process it is also a good idea to have a bootable rescue Alpine USB ready.

Once you have the default lts kernel and rescue USB <code>reboot</code> the computer.

If you are curious about correctness testing, some kernel modules or components do preform self tests at the beginning of the boot process. The tools may have test suites that you run with the make command.

== See Also ==
* [[Kernels]]
* [https://wiki.archlinux.org/title/Kernel Archwiki Kernels]
* [https://wiki.gentoo.org/wiki/Kernel Gentoo Wiki Kernel]
* [https://wiki.gentoo.org/wiki/Kernel/Configuration Gentoo Wiki Kernel Configuration]
* [[How to build the Alpine Linux kernel]]
* [[Kernel_live_patching|Kernel Live Patching (KLP)]]

[[Category:Kernel]]

User:Darkfader/distcc

2026-03-29T14:25:22Z

Darkfader: /* Alternatives */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
links for further review:

- https://repology.org/project/selinux-distcc/versions
- https://gitlab.com/liguros/liguros-repo/-/tree/stable/sec-policy/selinux-distcc

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
There are references to some 'forever free' tier, also, but I could not yet find the terms and conditions for having a 'forever free thing'. it also seems you need a windows-based 'coordinator' host.
opinion: generally if there's some professional context with high (business) pressure for faster builds I would check that out, and then without time pressure build a great distcc setup for the long-term.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-29T14:23:32Z

Darkfader: /* Alternatives */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
links for further review:

- https://repology.org/project/selinux-distcc/versions
- https://gitlab.com/liguros/liguros-repo/-/tree/stable/sec-policy/selinux-distcc

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
There are references to some 'forever free' tier, also, but I could not yet find the terms and conditions for having a 'forever free thing'. it also seems you need a windows-based 'coordinator' host.
generally if there's some professional context with high (business) pressure for faster builds I would check that out, and then there's more time to get a great distcc setup built in the meantime.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T22:42:55Z

Darkfader: /* selinux */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
links for further review:

- https://repology.org/project/selinux-distcc/versions
- https://gitlab.com/liguros/liguros-repo/-/tree/stable/sec-policy/selinux-distcc

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T18:14:36Z

Darkfader: /* selinux */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions
https://gitlab.com/liguros/liguros-repo/-/tree/stable/sec-policy/selinux-distcc

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T18:13:39Z

Darkfader: /* Gitlab integration */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

Maybe work from this k8s howto, it's a bit more prod-grade than most:

https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T16:33:18Z

Darkfader:

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & receptio

==== untested build container system ====

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T16:32:18Z

Darkfader: /* Operation */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

=== Gitlab integration ===

TBA

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T16:31:00Z

Darkfader: /* Alternatives */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching. At least you'd wanna know they have a higher commitment to maintenance than the understaffed distcc 'team' of current.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T16:30:17Z

Darkfader: /* Alternatives */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell. There's no mention if maybe SUSE's OBS uses it or something. Without more knowledge of their use I don't know how big a benefit will come from switching.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.
THey advertise some 'forever free' tier; generally if there's high (business) pressure for faster builds I would check that out.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T16:28:24Z

Darkfader: /* Alternatives */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

A commercial alternative (incredibuild https://www.incredibuild.com/) exists, but I have not tried it.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-28T16:22:32Z

Darkfader: /* general posture */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

== Alternatives ==

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

A commercial alternative (incredibuild) exists, but I have not tried it.

Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader

2026-03-28T16:20:55Z

Darkfader:

==About me==

UNIX Sysadmin, Server hugger and cluster fixer, Monitoring addict (Check_MK).

Blog: http://deranfangvomende.wordpress.com

=== Links ===

* DistCC howto: https://wiki.alpinelinux.org/wiki/User:Darkfader/distcc
* Collection of public Octeon SDKs: https://codeberg.org/darkfader/octeon-sdk-all

===Love Alpine Linux because===
* Only thing on par or close to 'proper' Carrier Grade Linux Distros
* the virtualization choices and DMVPN
* boot to ram
* the community
* the speed
* APK
* only Linux distro never driving me crazy.

===Areas of interest===

Alpine things I keep trying to do things with

* Xen (a decade ago :/)
* Re-do OpenNebula frontend port
* CFEngine / Rudder ports
* Elastic Agent

===Odd systems===

* MIPS64/Octeon3,
* CubieTruck Xen host
* RISC-V on Milk-V duo 64MB
* rental VPS running in memory

User:Darkfader/distcc

2026-03-26T01:13:47Z

Darkfader:

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

A commercial alternative exists, but I have not tried it.
Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

== Other sources ==

https://retroflux.net/blog/distcc-adventures/

User:Darkfader/distcc

2026-03-26T01:12:03Z

Darkfader: /* installation */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

=== untested build container system ===

See this one https://github.com/bensuperpc/docker-distcc
not yet tested, but this could be a good basis for a 'best practice' container.
There's others, especially for crossbuilds, but those are also very complex to modify.

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

A commercial alternative exists, but I have not tried it.
Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

User:Darkfader/distcc

2026-03-26T01:09:24Z

Darkfader: /* tcpwrapper style ip range filter */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost, but see here: https://gitlab.com/postmarketOS/pmbootstrap/-/work_items/1619) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

A commercial alternative exists, but I have not tried it.
Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

User:Darkfader/distcc

2026-03-26T01:08:21Z

Darkfader: /* general posture */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

A commercial alternative exists, but I have not tried it.
Architecturally it seems to be taking the ccache approach with a shared cache and smarter redistribution.
So anything that can call ccache is something they can scale out. I'll add a link later, for those whom it helps.
Naturally it is out of scope for this article.
Nonetheless it's interesting since the ordering of how to run ccache and how to run distcc is an issue that also exists in the OSS world. Not just that we don't have an automatic setup and integration of the tools (ccache, redis, distccd, distcc-pump) but we also don't have collected sufficient data to aid in deciding which aproach is the best.
Integrating into CIs (gitlab-runner specifically) is another item where cache persistence becomes very fragile.
Having an alpine build container that knows how to use ccache with redis and distcc-pump would be a possible step forward.

User:Darkfader/distcc

2026-03-26T01:01:43Z

Darkfader: /* tcpwrapper style ip range filter */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this (link lost) suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T19:25:12Z

Darkfader: /* ccache and memcached */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency
Upon looking at the ccache website, it seems the correct mechanism is not memcached but Redis. This would work just as well. Generally a cache shared in this way would be optimal for performance.

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T19:22:39Z

Darkfader: /* seccomp */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

I will update the entry once it's clear if the patch was later added or instead nothing was done.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T19:21:39Z

Darkfader: /* general posture */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project at its core relies on accepting foreign input over the network, has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, if they were actually upstreamed, I'd say they came through.

A kind of overengineered solulion would be to use netlabel to ensure application integrity across hosts, meaning only a valid process would be able to send packets to the distcc nodes :-)

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:37:01Z

Darkfader: /* troubleshooting / analysis */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

==== Node Selection algorithm ====

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:36:29Z

Darkfader:

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

=== Node Selection algorithm ===

as per my understanding, a flowchart goes here:

# compile task
# evaluate whether to run locally by job nature
# determine if local host's load is notable
# look at distcc hosts list
# do something based off localhost entry if first
# filter for nodes with cpp flag
# do something based off localhost entry if not first
# further prioritize by server order, first is handled in some way
# skip nodes in backoff prisons
# balance and priotize by server thread number, if given
# send to suitable host
# if compile not successful, proceed on other node
# if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:35:29Z

Darkfader: /* failed to distribute, running locally instead */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning. graceful performance degradation is the goal, and degrading it is while we try to figure this out.

A flowchart goes here:

1. compile task
2. evaluate whether to run locally by job nature
3. determine if local host's load is notable
4. look at distcc hosts list
5. do something based off localhost entry if first
6. filter for nodes with cpp flag
7. do something based off localhost entry if not first
8. further prioritize by server order, first is handled in some way
9. skip nodes in backoff prisons
10. balance and priotize by server thread number, if given
11. send to suitable host
12. if compile not successful, proceed on other node
13. if nodes depleted, proceed on localhost

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:29:49Z

Darkfader: /* failed to distribute, running locally instead */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.
if the curse is not lifted, despair may befall them and all they see is the need to investigate and ruminate further on the workings of this tool, ignoring thereby the obivous flaws that stem from of its alchemic origins.

just wait till you find out it has a backoff alghoritm deciding whether to call out to a remote server independent of that server functioning.

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:25:52Z

Darkfader: /* failed to distribute, running locally instead */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more error-prone at that same work, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:25:19Z

Darkfader: /* failed to distribute, running locally instead */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges to prey on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more fragile at that, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:24:44Z

Darkfader: /* failed to distribute, running locally instead */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient, wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more fragile at that, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:24:19Z

Darkfader: /* failed to distribute, running locally instead */

Hi,

I'm preparing this page. It can take a long time till I finish.
If you are also wishing to write on this topic, feel free to integrate the content.

== Document overview ==

I noticed that almost every distro has one partially complete, partially helpful document on how to use distcc on the distro.
Usually they also have one for ccache.
In either case, they're enough to get started, but not really a reliable watertight thing.
We definitely needed one of our own.

=== goal ===

to describe a working setup for building aports in easiest/fastests fashion
not planning to add versatility or features where it would make the setup more errorprone.
the page should describe enough of the steps to successfully compile an LTS kernel via aports and have that job be distributed over multiple nodes.
Logs should be set up and able to display errors, but not show any errors during the test compile.

To include a path for analysis via testing components.

distcc can greatly improve compile speeds for large software, it comes with a different set of features than ccache; it focusses not avoiding unneccessary compile work, but on a way to speed up the necessary one.

There's valuable info in the docs of other distros, it should be referenced here (i.e. the arch wiki troubleshooting), when it makes sense, add a TOC for their content.

=== audience ===
people running software builds on alpine and have multiple computers

This page will show a specific installation, specific configuration, and specific tests, resulting in a specific set of functionality that can be tested to be working.

== installation ==

=== Packages ===

you need, on each host
* distcc
* distccd-openrc
* distcc-pump
* distcc-pump-pyc

the .pyc will speed up the invocations, without it
idk if one or both should be installed in that case. but it appears to be also automatically be precompiled in /usr/lib/python3.12/site-packages/include_server/__pycache__/ so what does the package do exactly?

There's some references to cpython-312, so maybe it actually still uses classy python-c conversion or matbe that's just a component for reading C source code. I have zero idea.

you also need stuff to do compiles
* alpine sdk
* clang
* binutils
...
* elfutils(-dev)

=== Settings ===
==== settings for distcc ====

* there's /etc/default/distcc
* there's /etc/conf.d/distcc

make all your settings here
take care with the listen address, if you specify an IP it'll not be on 127.0.0.1 in case you would have localhost in your list...

* command_whitelist.sh

this is half functional, you need to set things here but you also need to maintain the symlinks that are collected under /usr/lib/distcc (for your compilers) and /usr/lib/distcc/bin (for itself)

you MUST run the script to update the compilers!
Info for script and what files it will create

/usr/sbin/update-distcc-symlinks

tschike:/usr/bin# ls -l /usr/lib/distcc/
total 4
drwxr-xr-x 2 root root 4096 Mar 2 18:14 bin
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c89 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 c99 -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 cc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-g++ -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc -> ../../bin/distcc
lrwxrwxrwx 1 root root 16 Mar 2 07:02 x86_64-alpine-linux-musl-gcc-15.2.0 -> ../../bin/distcc

distcc itself is in bin
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 cpp -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 18:14 gcc -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Mar 2 06:49 x86_64-alpine-linux-musl-gcc -> /usr/bin/distcc

the last symplink here is wrong, made by me and would not work...
BAD symlink.

=== distcc hosts file ===

idk about that thing it's odd

=== abuild.conf ===

settings for aports
* cc=
* cxx=
* cpp=
* cflags=
* njobs

== detail infos ==

???

=== hosts syntax ===

* myhost otherhost
* myhost,cpp,lzo myotherhost,cpp,lzo

==== the host ====

hostname/ip
localhost
127.0.0.1
::1 - does not work

==== protocol ====

* no protocol given
* ,cpp,lzo protocol

cpp implies lzo, it requires compression, even if you have 10gbit/s or more, it's just hardcoded

=== threads ===
/number of workers

== architecture ==
it can handle C, C++, ObjC, maybe some other stuff

* what happens with normal xmit
* what happens with pump mode
* at which step the include server is used and how it collects the includes

=== distribution algorithm ===
honestly I simply don't get it

* The order matters
* The number of threads matters

==== localhost ====

* localhost precedence
* localhost fallback

variable: DISTCC_FALLBACK

0 = Fail to compile if it would need to fallback to a normal local gcc call
1 = If remote compile fails, just do it yourself

== Operation ==

=== startup and shutdown ===
service distcc stop is not entirely reliable (it can take a minute after the stop until the processes are gone and sometimes it will never stop
this is very bad with openrc, the openrc script returns after a second and only relies on its service flags, not the process status.
manually check after stopping, wait a min, if needed, kill it all.
at some point the rc file needs to be rewritten, it can't stay like it is.

if you used a pump mode session, that also needs a logout (pump --shutdown)
avoid running multiple startups without shutdown in one session. it's safe as far as I can tell but nothing cleans up these processes.

=== ccache and memcached ===
CCACHE is said to be conflicting with pump mode unless when you call them in the backend
so, where you start the compile, you don't use it
where the compile happens, you use it
they can share the cache via memcached, this is a nice trick for consistency

=== dockerized / native ===

it remains mostly the same, a container needs to make sure it monitors the right services (distccd, nginx, include_server)
if you're using zeroconf, you need to somehow expose the mdns service broadcasts & reception

Processs list:

* TBA

Workdir list:

* TBA

=== alpine-chroot ===

when using the 'official' script there's still some odd pieces, seemed to be the processes died on logout. but not all of them.

==== manual launch ====

Starting the daemon would be using
/usr/bin/distccd --pid-file /var/run/distccd/distccd.pid -N 15 --user distcc --port 3632 --log-level=debug --log-file=/var/log/distccd.log --allow my-sub-net/24

==== localhost? =====
unclear if you need to use --allow for 127.0.0.1/32 or something to allow the remote preproccesor.

== Kernel specific settings ==

currently (distcc 3.4-r9 on Alpine) you need a patch to build the kernel.
See

=== Include server Settings ===

cache reset triggers
This ought to be set before enabling pump mode.

export INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"

link to explanation TBA

=== Disabling GCC Plugins ===

KConfig unselect HAVE_GCC_PLUGINS

Some info is here in the troubleshooting part of the Arch Wiki
https://wiki.archlinux.org/title/Distcc#Troubleshooting

=== Patches ===

Other things (for 6.6LTS)
PCIe Stub patch

=== Autoconf ===

No known setup examples
Add whatever it publishes in mdns

== troubleshooting / analysis ==

=== testing ===

# turn off fallback DISTCC_FALLBACK=1
# set distcc up to point at specific system under test DISTCC_HOSTS="mytestbox,cpp,lzo"
# GCC example compiles
## code example C
## same with included header
## code example C++
## same with included header
## code example ObjC
## same with included header
# Cmake example compiles

docoument the thing with compile launcher ccache;distcc
there's some blog post, point to that
no ccache here yet
show $CC differences distcc vs gcc, what configure scripts see

=== Latency ===

Latency of pump mode startups and fallbacks needs to be investigated.
LZO is enforced even if you have faster network
DNS Requests, very old bug report from Gcode, one request per call, is it true? how to get rid of it?
TMPDIR is respected, make sure it's on ramdisk even on the remote nodes.
Compile ideally never goes to disk when it doesn't have to.
How efficient is the include server collection and unpacking?

=== failed to distribute, running locally instead ===

the curse of the ancient and wise bulgarian witch compildora nottherea has befallen people all over the world. only strict and mindless adherence to rituals passed down from generation to generation has given hope to those who are under her ages old spell. again and again it reemerges on idealistic young men and women who spend so much time at their unholy computers that they try to spend less time there by spending a lot of time trying to optimize what the computer does, slowly, instead of reflecting on why they are there, while the computer is busy working, and why they try to solve this by adding a component that makes the computer be more fragile at that, increasing the need of their presense at this idolized thinking machine to oversee and often repair, or worse, mindlessly restart its doing.

== security ==

=== tcpwrapper style ip range filter ===

the original security model consists of ip restrictions.
there seems to also be some GSSAPI user auth.
further, commands that can be called are restricted by name and location.
this appears to be a runtime whitelist lookup, meaning it's done and authorized by the same parts of the daemon as processes the compile request along with the intended compiler.
so the main weaknesses against malicious clients seem to be in sending things to compile, and in overriding the remote compiler to use.
it can be assumed that a malicious client able to exploit the compiler handshake can then run arbitrary stuff.
There's at least a github issue regarding this google.com/search?q=distcc+seccomp&rlz=1C5CHFA_enDE1121DE1121&oq=distcc+seccomp&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigAdIBCDM1NjRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8 suggesting running over ssh. That does only partitally alleviate this risk with regard to a key based verfication of a client versus a the standard ip restrictions which always include some parsing.
So this protects against someone directly exploiting the TCP code of distcc.
It does not protect against malicious clients.
(ssh force command can't be used or you'll not compile anything)

The basic step for protecting access should be filtering who can access the distcc server, so use nftables etc. to restrict access to port 3262 (??) set up the internal filter the same way.

=== seccomp ===

The next thing is to confine the compiler calls to only write in their temp directory and that they can only run compilers (using nsjail, apparmor, selinux etc)

the above issue also references the second bit of security, namely a seccomp filter which will already cover a good bit of the above
https://github.com/distcc/distcc/pull/235
the commit got closed without merge, from what I see.

=== privs ===

The other internal security bit is that they do some priviledge dropping. it runs as a dedicated user (distcc), so you can also have an audit policy, and can/could use something like iptables' to ensure it can only connect to the other distcc/memcached hosts, but nothing else.

=== compiler list ===

One needs to investigate when compiler_whitelist.sh is exactly called. as far as I recall it doesn't close stdin/stdout.

=== distcc-hardened ===

Alpine adds some hardening patch, idk what nature/origina that has.

=== selinux ===

there's also a selinux policy for distcc from gentoo or liguros if one is so inclined.
https://repology.org/project/selinux-distcc/versions

=== general posture ===

Some security measures like the above should definitely be used since the project has only a few part-time maintainers that cannot easily drive the project forward or do large refactors.
But the seccomp changes were done almost 10 years ago, so, to be fair, they still came through.

Arch wiki refers to a fork by SUSE
https://github.com/icecc/icecream

It appears 'maybe better' but hard to tell.

User:Darkfader/distcc

2026-03-03T11:19:59Z

Darkfader: /* troubleshooting / analysis */

User:Darkfader/distcc

2026-03-03T11:08:42Z

Darkfader: /* goal */

User:Darkfader/distcc

2026-03-03T11:08:08Z

Darkfader: /* goal */

User:Darkfader/distcc

2026-03-03T11:07:13Z

Darkfader: /* general posture */

User:Darkfader/distcc

2026-03-03T11:04:53Z

Darkfader:

User:Darkfader/distcc

2026-03-03T11:03:03Z

Darkfader: /* troubleshooting / analysis */

User:Darkfader/distcc

2026-03-03T11:01:34Z

Darkfader:

User:Darkfader/distcc

2026-03-03T10:49:52Z

Darkfader:

User:Darkfader/distcc

2026-03-03T10:46:28Z

Darkfader: /* general posture */

User:Darkfader/distcc

2026-03-03T10:44:30Z

Darkfader: /* general posture */

User:Darkfader/distcc

2026-03-03T10:44:03Z

Darkfader: /* security */

User:Darkfader/distcc

2026-03-03T10:32:10Z

Darkfader:

User:Darkfader/distcc

2026-03-03T10:24:17Z

Darkfader: /* Disabling GCC Plugins */

User:Darkfader/distcc

2026-03-03T10:22:24Z

Darkfader:

User:Darkfader/distcc

2026-03-03T10:19:46Z

Darkfader:

User:Darkfader/distcc

2026-03-03T10:15:58Z

Darkfader: /* architecture */