User talk:Jch

NFS bug study

All debian used are fresh install of wheezy 7.8.
All alpine used are fresh install of edge. (will also try vanilla kernel in KVM)
All boxes are supermicro servers with bi-Xeon running AL from USB key.
I do not have physical access to the boxes!

The NFS-servers are configured to export

/srv/home       192.168.1.0/24(rw,sync,no_subtree_check)

The nfs-clients are configured to mount from fstab

storage:/srv/home /home nfs noauto,defaults,noexec 0 0 

"storage" is defined in /etc/hosts to point to the right server.

The test is done with

mount /home

We will compare the dmesg outputs, the ls -ld /home outputs, the cat /home/test and touch /home/toto ones. /home/test is prepared on the server (just a text file containing "do you see me?"). Those tests are run as root user.

Will redo usage tests with non root user because of the default squashroot of NFS...

NFS-server in KVM-Debian

fresh install with tasksel "file server"
this KVM in running on bare metal alpine

nfs-client in KVM AL

mount /home gives
in dmesg

[73460.112383] RPC: Registered named UNIX socket transport module.
[73460.112386] RPC: Registered udp transport module.
[73460.112388] RPC: Registered tcp transport module.
[73460.112389] RPC: Registered tcp NFSv4.1 backchannel transport module.
[73460.165060] svc: failed to register lockdv1 RPC service (errno 111).
[73460.165069] lockd_up: makesock failed, error=-111
[73460.217513] NFS: Registering the id_resolver key type
[73460.217524] Key type id_resolver registered
[73460.217525] Key type id_legacy registered

in ls -ld /home/

drwxr-xr-x    2 42949672 42949672         6 Jan 23 12:27 /home

in cat /home/test

 Do you see me?

in touch /home/toto

touch: /home/toto: Permission denied

But with some user with the right real uid:gid,

webhosting:~$ ls -ln /homebis/
total 0
drwxr-xr-x    2 4294967294 4294967294        17 Feb  4 09:55 tests
webhosting:~$ ls -ln /homebis//tests/
total 0
-rw-r--r--    1 4294967294 4294967294         0 Feb  4 09:55 toto
webhosting:~$ vi /homebis//tests/toto 
webhosting:~$ ls -ln /homebis//tests/
total 4
-rw-r--r--    1 4294967294 4294967294         5 Feb  4 10:06 toto

The uid:gid does not appear right but access rights seem good.

nfs-client in KVM debian

dmesg is empty
ls -ld /home

drwxr-xr-x 2 root root 17 Jan 23 08:39 /home

cat /home/test

Do you see me?

touch /home/toto (even after adding rw to the mount options in fstab)

touch: cannot touch `/home/toto': Permission denied

Some pointers to investigate this permission problem:

• http://unix.stackexchange.com/questions/79172/nfs-permission-denied

To begin using machine as an NFS client, you will need the portmapper running on that machine, and to use NFS file locking, you will also need rpc.statd and rpc.lockd running on both the client and the server.

'

nfs-client in LXC AL (on bare metal AL)

apk add nfs-utils

dmesg empy sofar

mount /home

dmesg

[4153944.457610] RPC: Registered named UNIX socket transport module.
[4153944.457615] RPC: Registered udp transport module.
[4153944.457618] RPC: Registered tcp transport module.
[4153944.457620] RPC: Registered tcp NFSv4.1 backchannel transport module.
[4153944.504475] svc: failed to register lockdv1 RPC service (errno 111).
[4153944.504484] lockd_up: makesock failed, error=-111
[4153944.681725] NFS: Registering the id_resolver key type
[4153944.681744] Key type id_resolver registered
[4153944.681748] Key type id_legacy registered

ls -ld /home

drwxr-xr-x    2 42949672 42949672        17 Jan 23 14:39 /home

cat /home/test

Do you see me?

touch /home/toto

touch: /home/toto: Permission denied

nfs-client in LXC AL (in KVM AL)

apk add nfs-utils

but

# mount /home
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
mount: permission denied (are you root?)

and

# /etc/init.d/rpc.statd start
 * Caching service dependencies ... [ ok ]
 * Starting rpcbind ... [ ok ]
 * Starting NFS statd ... * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'
 [ !! ]
 * ERROR: rpc.statd failed to start

dmesg

[74747.135827] rpcbind[6718]: segfault at 7ccfe7b0 ip 000072977ccef5cd sp 00007c6b3e329a68 error 4 in ld-musl-x86_64.so.1[72977cca0000+85000]
[74747.135841] grsec: Segmentation fault occurred at 000000007ccfe7b0 in /sbin/rpcbind[rpcbind:6718] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1831] uid/euid:0/0 gid/egid:0/0
[74747.135887] grsec: bruteforce prevention initiated due to crash of /sbin/rpcbind against uid 100, banning suid/sgid execs for 15 minutes.  Please investigate the crash report for /sbin/rpcbind[rpcbind:6718] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1831] uid/euid:0/0 gid/egid:0/0

nfs-client in LXC debian (in KVM AL)

apt-get install nfs-commonn

gives

[FAIL] Starting NFS common utilities: statd idmapd failed!

then mount /home gives same results in guest as in host

NFS-server in KVM-Alpine

Done from a KVM running in memory straight from the iso

CDROM="/my/path/alpine-mini-3.1.1-x86_64.iso"
qemu-system-x86_64 -name test -enable-kvm -cpu qemu64 -m 256 -smp 1 -curses \
  -net nic,vlan=0,model=virtio,macaddr=52:54:32:a0:a0:a0 \
  -net tap,vlan=0,script=/etc/openvswitch/ovs-ifup-lan,downscript=/etc/openvswitch/ovs-ifdown-lan,ifname=test0 \
  -cdrom ${CDROM}

do not forget to issue "grsec nomedeset" at SYSLINUX prompt or you loose the output (I'm doing it trough ssh term)

# setup-alpine # no disk install at all, no apk cache but proxy
# . /etc/profile.d/proxy.sh
# apk add nfs-utils
# echo "/home   192.168.1.0/24(rw,no_root_squash)" >> /etc/exports
# echo "Do you see me?" > /home/test
# /etc/init.d/nfs start
 * Caching service dependencies ...                                       [ ok ]
 * Starting rpcbind ...                                                   [ ok ]
 * Starting NFS statd ...
 * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'               [ !! ]
 * ERROR: rpc.statd failed to start
 * ERROR: cannot start nfs as rpc.statd would not start
# dmesg # only relevant lines displayed
[  462.262020] rpcbind[1890]: segfault at 1e783940 ip 000070591e773f1d sp 00007dc1da01a4d8 error 4 in ld-musl-x86_64.so.1[70591e724000+86000]
[  462.262032] grsec: Segmentation fault occurred at 000000001e783940 in /sbin/rpcbind[rpcbind:1890] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1] uid/euid:0/0 gid/egid:0/0              
[  462.262043] grsec: bruteforce prevention initiated due to crash of /sbin/rpcbind against uid 100, banning suid/sgid execs for 15 minutes.  Please investigate the crash report for /sbin/rpcbind[rpcbind:1890] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1] uid/euid:0/0 gid/egid:0/0
# poweroff

Let's try with the vanilla kernel

CDROM="/my/path/alpine-vanilla-3.1.1-x86_64.iso"

with same command line and same sequence of instructions

test:~# /etc/init.d/nfs start
 * Caching service dependencies ...                                       [ ok ]
 * Starting rpcbind ...                                                   [ ok ]
 * Starting NFS statd ...
 * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'               [ !! ]
 * ERROR: rpc.statd failed to start
 * ERROR: cannot start nfs as rpc.statd would not start
test:~# dmesg
[  243.445710] rpcbind[1930]: segfault at 33f30940 ip 00007f5a33f20f1d sp 00007fffa4290e48 error 4 in ld-musl-x86_64.so.1[7f5a33ed1000+86000] 
test:~# poweroff

Obviously I will not be able to test clients now...

UPDATE 2015-02-20 with http://dev.alpinelinux.org/~clandmeter/rpcbind-0.2.3_rc2-r0.apk NFS works on AlpineLinux x86_64 stable both as server and client.

How to automate KVM creation

The goal is not only to have a working install but to have it at the after setup-alpine stage without human intervention... Tis is the first stages of a work in progress...

I want to pass a Block Device and a name as parameters. The block device could be an image file, a LV, a NBD, a hdd, a raid array, whatever.
Everything else should be fully automatic according to some config file (stating the http-proxy, the time server, the log server, ...).

The I will just run the script, watch my dhcp logs to discover the new IP assigned (that's why the name is a parameter), then log in with ssh without password to customize it further but at high level only (will be a robot and not me in fact).

I guess it would be something like emulate boot from usb key with specific overlay already on key...
then run setup-disk with proper parameters on the command line to avoid the interactive process (like setup-alpine does)...
Methink this could be done from a couple of scripts put in /etc/local.d/. The last.stop one deleting all of them to be clean at next reboot.
Let's start easy ;)

How to prepare a img file to emulate an USB key

first a working example done in console (accessed trough ssh).
Will build a script from it...

First, lets's prepare somme block device (here an image file but could be something else)

apk add qemu-img
qemu-img create -f raw usbkey.img 512M
apk del qemu-img
T="usbkey.img"

Next, let's install AL on this $T

apk add multipath-tools syslinux dosfstools
fdisk $T
kpartx -av $T
mkdosfs -F32 /dev/mapper/loop1p1
dd if=/usr/share/syslinux/mbr.bin of=/dev/mapper/loop1
syslinux /dev/mapper/loop1p1
mkdir key
mount -t vfat /dev/mapper/loop1p1 key
wget http://wiki.alpinelinux.org/cgi-bin/dl.cgi/v3.1/releases/x86_64/alpine-mini-3.1.1-x86_64.iso
mkdir cdrom
mount alpine-mini-3.1.1-x86_64.iso cdrom
cd cdrom
cp -a .alpine-release * ../key/
cd ..
umount key
umount cdrom
kpartx -d $T
apk del multipath-tools syslinux dosfstools
rm alpine-mini-3.1.1-x86_64.iso

This block device may now be use to boot some KVM for instance like:

screen -d -m -S KVM-builder \
qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
-device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick

This is working fine. The problem is when adding a HDD to the lot, qemu try to boot from the hdd and does not even try to boot from the usb key. Enabling menu in boot let's one access the emulated bios which allows to select USB device to boot interactively but this break the goal of fully automated boot :( The stanza is for instance

screen -d -m -S KVM-builder \
qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
-device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick \
-drive file=$T2 boot menu=on

qemu-doc states that very clearly:
> -boot [order=drives][,once=drives][,menu=on|off][,splash=sp_name][,splash-time=sp_time][,reboot-timeout=rb_timeout][,strict=on|off]
> Specify boot order drives as a string of drive letters. Valid drive letters depend on the target achitecture. The x86 PC uses: a, b (floppy 1 and 2), c (first hard disk), d (first CD-ROM), n-p (Etherboot from network adapter 1-4), hard disk boot is the default

Starting AL from network

As it does not seems possible to start qemu with a virtual USB key *and* a virtual HDD attached to the VM. Let's try something different: to start AL from the network and mount the HDD later on...

Usually this kind of setup needs

• a DHCP server to get an IP address and the location of the TFTP server
• a TFTP server to download the kernel and tje root file system to boot from
• a NFS server or a HTTP one to get the overlay used to configure the machine
• a NFS server to share files with others
• a NBD server to get his own block devices as storage
• a machine where to prepare initramfs

First, let's check what is vailable in AL and what is not...

• dhcpcd-6.6.7-r0
• tftp-hpa-5.2-r1
• nfs-utils-1.3.1-r2
• darkhttpd-1.10-r1
• qemu-nbd (not really good but exists)

PXE_boot

We are trying to do something as in PXE_boot.

We did it on separate machine for each service. It forces us to deeply understand all interactions between processes.

dhcpd

192.168.1.1

with package dhcp from repo. Nothing special.

  filename "pxelinux.0";
  next-server 192.168.1.2;

tftp

192.168.1.2

tftp-hpa configured to serve some SYSLINUX files.

The config is in /etc/conf.d/in.tftpd
Then to issue:

rc-update add in.tftpd
rc-service in.tftpd start

We serve from /var/tftpboot.

We add to temporary install the syslinux apk to get pxelinix.0 and other libs needed.
We did prepare a "pxerd" initramfs file with virtio_net.ko, dhcp and nfs included; made sure loop and squashfs are included.

pxelinux.cfg/default looks like

PROMPT 0
TIMEOUT 3
default alpine
LABEL alpine
LINUX alpine/vmlinuz-grsec
INITRD alpine/pxerd
APPEND ip=dhcp alpine_dev=nfs:192.168.1.3:/srv/boot/alpine modloop=/boot/grsec.modloop.squashfs nomodeset quiet apkovl=http://192.168.1.4/localhost.apkovl.tar.gz
#APPEND modloop=http:/192.168.1.4/grsec.modloop.squashfs
#APPEND apkovl=http://192.168.1.4/localhost.apkovl.tar.gz # including the modloop hack
#APPEND alpine_repo=http://repo-url

Modules are loaded

/ # lsmod
Module                  Size  Used by    Not tainted
nfsv3                  22784  1
nfs                   144376  2 nfsv3
lockd                  71917  2 nfsv3,nfs
sunrpc                225574  6 nfsv3,nfs,lockd
af_packet              28735  0
sr_mod                 13487  0
cdrom                  40424  1 sr_mod
pata_acpi               3326  0
ata_piix               25601  0
ata_generic             3554  0
libata                181955  3 pata_acpi,ata_piix,ata_generic
virtio_net             19684  0
scsi_mod              113710  2 sr_mod,libata
virtio_pci              6485  0
virtio                  4933  2 virtio_net,virtio_pci
virtio_ring             9161  2 virtio_net,virtio_pci
squashfs               25893  1
loop                   18243  2

Network is up

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:33:B0:C2:D2
inet addr:192.168.1.108  Bcast:0.0.0.0  Mask:255.255.255.0  
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1  
RX packets:322 errors:0 dropped:0 overruns:0 frame:0  
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0  
collisions:0 txqueuelen:1000
RX bytes:20514 (20.0 KiB)  TX bytes:684 (684.0 B)

but modloop does not load This patch fix this issue (hope to see it mainstream soon)

localhost:~# diff /etc/init.d/modloop modloop.new 
--- /etc/init.d/modloop 
+++ modloop.new
@@ -32,7 +32,7 @@
        local search_dev="$1" fstab="$2" 
        local dev mnt fs mntopts chk
        case "$search_dev" in
-               UUID=*|LABEL=*|/dev/*);;
+               UUID=*|LABEL=*|/dev/*|nfs);;
                *) search_dev=/dev/$search_dev;;
        esac
        local search_real_dev=$(resolve_dev $search_dev)
@@ -49,6 +49,10 @@
                                fi
                        done
                done
+               if [ "$fs" = "$search_dev" ]; then
+                       echo "$mnt" 
+                       return
+               fi
        done < $fstab 2>/dev/null
 } 

References

http://www.syslinux.org/wiki/index.php/PXELINUX

nfs

192.168.1.3

see http://wiki.alpinelinux.org/wiki/User_talk:Jch#NFS_bug_study
It is now working with http://dev.alpinelinux.org/~clandmeter/rpcbind-0.2.3_rc2-r0.apk

We serve the content of an usb key (iso) in ro as

/srv/boot/alpine	*(ro,no_root_squash,no_subtree_check)

http

192.168.1.4

With package Darkhttpd from repo serving from /var/tftpboot/ to serve files needed to boot (kernel, rootfs, apkovl.tar.gz)

nbd

192.168.1.5

I really would like to have xnbd-server in AL.
For now, we have a qcow2 debian image added to the apkovl with lbu add; lbu ci.
This image is used to launch a first KVM with /dev/mdX as second drive.
In turn, inside the KVM, vdb is used to define a lvm2 volume.
The LV are published with xnbd-server.

Later on, the same KVM will be able to connect to RBD device and re-publish it as NBD.

xnbd-server allows live migration of Block Devices while live. And has a powerfull proxy mode.

All other KVM are running from FS accessed trough NBD from such SAN. Even other SAN.
As soon as those KVM-NBD are up, they may be used to launch others or to provide datastores.

We put that image on every USB key we use along with mdadm and OpenVSwitch (and collectd).

dns

192.168.1.6

to be developped

Building a complete infrastucture with AL

I'm doing it. It's for real! That's my daily job at present ^^

I'm building a full private cloud bootstraped with only an AlpineLinux USB key for each physical machine. But next ones will be able to boot from network; not even USB keys will be needed. As a matter of fact, we used more than only one physical USB key because we didn't started from scratch but had a live migration from Debian to Alpine for most of the services and machines...

If there is some feed-back, I may develop config files and so on ;)

As I started from scratch and OpenVSwitch was not available in Alpine at that time yet, It took me a while to build everything. But to reproduce it, it would be piece of cake!

We use qemu-kvm for KVM. But I guess one may use whatever Virtual Machine technology one likes.

This is the presentation of a use case. Not a HOW TO. And it's still a work in progess...

Network

Firewall

We put a dedicated physical machine on each link between our LAN and other networks. It just run iptables and some paquets accounting metrology.

Router

Physical machine connected to our LAN and other networks (trough a firewall). A static routing table do the trick.

Switches

All physical machines run OpenVSwitch reproducing virtually all physical switches we have plus some virtuals only.

VPN

All physical machines run openVPN as client to as many switch defined less the physical interfaces of the machine. There is an openVPN server somewhere running in a KVM connected to needed switches.

Storage

SAN

On each physical machine, a couple of HDD are mounted in raid1 witch mdadm. This raid array is passed as parameter to a KVM who in turn mount it as physical volume for LVM. The created LV are published as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as xnbd is not in Alpine (yet?)..

The SAN also connects to the CEPH cluster as client and publish reached RBD as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as no xnbd nor RBD are in Alpine (yet?)..

NAS

Running on the same physical machine, another KVM is mounting some NBD (with qemu-nbd) as local drives and publishing some directories as NFS shares. For the time being, this KVM is running debian 7.8 as there is no good nbd-client in Alpine (yet?)... We now have nfs-server and nfs-client in AL.

CEPH

KVM with physical HDD as parameters are used for building OSD and MON needed to operate a CEPH cluster. One KVM is the "console" to drive it from a single point of presence (usefull but not "needed").For the time being, those KVM are running debian 7.8 as CEPH and RBD are not in Alpine (yet?)..

Low-level services

No service at all is running in the AL on bare metal. All are running is some KVM connected to needed switches by the means of the OpenVSwitches. The apkovl on the USB keys contains only the scripts to launch KVM and one image file to launch the first SAN. Other KVM are launched from LV in the SAN.

dhcp

Exactly two KVM stored in different SAN, primary and secondary in failover mode, are running dhcpd from repo.
We just have to configure it properly.

We have to test if dhcpd may run in a LXC instead of a KVM?

DNS

Nothing to say here because still running on debian.

Resolver

With dnscache from repo.

Those KVM have manually assigned IP address in the LAN and does know a gateway to the Internet.
They use themselves as resolver...
They know the direct manually assigned IP address in the LAN of the main DNS server of selected domains (for split dns configuration).

PXEboot

Need to try ;) It is THE NEXT STEP!
Must prepare needed files in tftp server.
Must prepare needed files in nfs server.
Must prepare needed files in darkhttpd server.
Must start both dhcpd server (primary and secondary) with prepared config in a row.

Time server

The router (who has access to internet) usr ntpd (or similar) from repo, to act as client to the WAN and server to the LAN.

syslog

With syslog-ng from repo, we receive the logs from all machines be it physical or virtual.
It's the only place who needs logrotate from repo.

HTTP proxy/cache

The web proxy/cache squid, from repo, uses a NBD as cache. It has a link to the internet to forward requests and one to the LAN.

Because of him, no machine, as they are all connected to the LAN, be it physical or virtual, needs a published default gateway. And all machines are able to install/upgrade packages or to see the WWW as client.

We point all AL boxes to this KVM with setup-proxy.

Monitoring

shinken from sources in some LXC with barely only the python package installed

Metrology

Collectd (one KVM as server, all other machines, be it physical or virtual, as client) with collectd-network from repo.
A couple of lines in CGP config file is enough for now.

Backups

with common tools: rsync, tar, nc, bzip2, openssh, cron

High-level services

in LXC AL whenever possible.
in LXC Debian as second choice
in KVM otherwise.

edge is currently broken and is unfortunately needed for several services :( Albeit Alpine is fixed ASAP albeit we will need to switch...

x2goserver

AL edge proposes the package x2goserver.
I would like to give it a try ;)
It seems to be running (at least installed) in a AL LXC inside a AL KVM and connected trough OVS ^^

but unfortunately, x2goclient pops up "kex error : did not find one of algos diffie-hellman-group1-sha1 in list curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 for kex algos"

Master key

We want to be able to bootstrap the full infrastructure from only one usb key and one machine with physical access (to insert the usb key obviously).

This key will run AL stable. With only very few packages installed. But some images on the sorage.

Initial packages

dhcpd tftp syslinux nfs darkhttp openssh vim openvswitch mdadm qemu screen collectd collectd-network gptdisk irqbalance ssmtp mailx

Bootstrap

First, we setup the network. Remember, this is a bootstrap. We assume nothing exect to have (or not) an internet connexion on some ethernet cable plugged into some NIC. It means we may take any decision we see fit.

Our primary machine is the only fixed point for now. Let's give it the number 1.
All machines are connected to the LAN. We know nothing yet about other NICs.
First we must decide about the LAN IP range. For instance be it 192.168.1.0/24.
We will use complicated network setup, let's start by installing openvswitch on bare metal

apk add openvswitch
rc-update add ovs-modules
rc-update add ovsdb-server
rc-update add ovs-vswitch
rc-service ovs-modules start
rc-service ovsdb-server start
rc-service ovs-vswitch start
ovs-vsctl add-br lan
ovs-vsctl add-port lan eth0
vi /etc/network/interfaces #iface eth0 inet manual
                           #iface lan inet dhcp

No machine will offer any service from bare metal.

apk add qemu-system-x86_64 screen
modprobe kvm
modprobe kvm-intel
modprobe kvm-amd
modprobe tun
screen -m -d -S KVM-infra qemu-system-x86_64 -kvm -kernel /kernel -initrd /initrd -append alpine_dev=...,apkovl=... -net -net -drive /dev/usb

La suite immédiate se fait dans cette VM

screen -r KVM-infra

We need the storage space from the usb key to handle boot images and apkovl files.
In KVM-infra

setup-alpine (fixed IP)
apk add openvswitch
rc-update add ovs-modules
rc-update add ovsdb-server
rc-update add ovs-vswitch
rc-service ovs-modules start
rc-service ovsdb-server start
rc-service ovs-vswitch start
ovs-vsctl add-br lan
ovs-vsctl add-port lan eth0
vi /etc/network/interfaces #iface eth0 inet manual
                           #iface lan inet static
mkdir -p /srv
mount /vda2 /srv
mkdir -p /srv/nfs/alpine
mount /vda1 /srv/nfs/alpine

Next we will have other networked devices. We need dhcpd (in KVM-infra)

apk add dhcpd 
rc-update dhcpd
vi /etc/dhcp/dhcpd.conf  #filename "pxelinux.0";
                         #next-server 192.168.1.1;
rc-service dhcpd start

To boot others, we need tftp, nfs and http (tftp and http in LXC, nfs in KVM-infra)

apk add tftp-hla nfs-utils darkhttp
rc-update add nfs
rc-update add tftp
rc-update add darkhttpd
mkdir -p /srv/nfs
mkdir -p /srv/tftp
mkdir -p /srv/http
vi /etc/exports
vi /etc/tftp 
vi /etc/darkhttp
rc-service tftp start
rc-service darkhttpd start
rc-service nfs start

Let's populate those servers.

mkdir -p /srv/tftp/alpine
cp /media/usb/boot/vmlinuz* /srv/tftp/alpine/
cp /media/usb/boot/modloop* /srv/tftp/alpine/
apk add mkinitfs
cd /etc/mkinitfs
vi features.d/network.modules
vi features.d/dhcp.files
vi features.d/dhcp.modules
vi features.d/nfs.modules
vi mkinitfs.conf # add network, dhcp, nfs and squashfs
mkinitfs -o /srv/tftp/alpine/pxerd
apk del mkinitfs
apk add syslinux
cp /usr/share/syslinux/pxelinux.0 /srv/tftp/
cp /usr/share/syslinux/ldlinux.c32 /srv/tftp/
apk del syslinux
mkdir -p /srv/tftp/pxelinux.cfg
vi /srv/tftp/pxelinux.cfg/default

All we need now to boot another AL machines (be it physical or virtual) are some {MAC}.apkovl.tar.gz files served by darkhttpd. We badly need name resolution at this stage. DNS and resolver are needed. DNS to be updated dynamically by dhcp server with split-dns. Resolver knowing the fixed IP address on the DNS and the default route if known at this stage. Both may run in LXC inside this KVM-infra (like the other previous services). DNS will be djbdns and resolver will be dnscache (both from repo).

Then (for now) we need image of a debian install with xnbd-server and lvm2 to build SAN.
Also, on bare-metal we need mdadm to assemble raid1 arrays.
A new SAN is therefore, a MAC address (for debian boot as san), some BD as vda (raid1 from mdadm).
A new server is therefore, a MAC address (for AL boot), a apkovl file (MAC named), some data NBD from some SAN.
The apkovl will be downloaded at boot time with PXE provided address; before launching openvswitch! The IP address will then change because of the apparent MAC change when OVS becomes active.
We may use symlinks to MAC named config files to have a more human friendly view.

It is to be noted that after bootstrap KVM may move to other physical machines. While some KVM-infra is somehow connected to the LAN, everything stay alive! This precise image will be reproduced in every SAN build.