View source for User talk:Jch

= NFS bug study =

All debian used are fresh install of wheezy 7.8.<br/>
All alpine used are fresh install of edge. (will also try vanilla kernel in KVM)<br/>
All boxes are supermicro servers with bi-Xeon running AL from USB key.<br/>
I do not have physical access to the boxes!

The NFS-servers are configured to export 
 /srv/home       192.168.1.0/24(rw,sync,no_subtree_check)

The nfs-clients are configured to mount from fstab
 storage:/srv/home /home nfs noauto,defaults,noexec 0 0 

"storage" is defined in /etc/hosts to point to the right server.

The test is done with
 mount /home

We will compare the '''dmesg''' outputs, the '''ls -ld /home''' outputs, the '''cat /home/test''' and '''touch /home/toto''' ones. /home/test is prepared on the server (just a text file containing "do you see me?"). Those tests are run as root user.

'''Will redo usage tests with non root user because of the default squashroot of NFS...'''

== NFS-server in KVM-Debian ==

fresh install with tasksel "file server"<br/>
this KVM in running on bare metal alpine

=== nfs-client in KVM AL ===

mount /home gives <br/>
in '''dmesg'''
<pre>
[73460.112383] RPC: Registered named UNIX socket transport module.
[73460.112386] RPC: Registered udp transport module.
[73460.112388] RPC: Registered tcp transport module.
[73460.112389] RPC: Registered tcp NFSv4.1 backchannel transport module.
[73460.165060] svc: failed to register lockdv1 RPC service (errno 111).
[73460.165069] lockd_up: makesock failed, error=-111
[73460.217513] NFS: Registering the id_resolver key type
[73460.217524] Key type id_resolver registered
[73460.217525] Key type id_legacy registered
</pre>
in '''ls -ld /home/
 drwxr-xr-x    2 42949672 42949672         6 Jan 23 12:27 /home
in '''cat /home/test'''
  Do you see me?
in '''touch /home/toto'''
 touch: /home/toto: Permission denied

=== nfs-client in KVM debian ===

'''dmesg''' is empty<br/>
'''ls -ld /home'''
 drwxr-xr-x 2 root root 17 Jan 23 08:39 /home
'''cat /home/test'''
 Do you see me?
'''touch /home/toto''' (even after adding rw to the mount options in fstab)
 touch: cannot touch `/home/toto': Permission denied

<u>Some pointers to investigate this permission problem</u>:
* http://unix.stackexchange.com/questions/79172/nfs-permission-denied

''To begin using machine as an NFS client, you will need the portmapper running on that machine, and to use NFS file locking, you will also need rpc.statd and rpc.lockd running on both the client and the server.''

''''

=== nfs-client in LXC AL (on bare metal AL) ===

 apk add nfs-utils
dmesg empy sofar
 mount /home
'''dmesg'''
<pre>
[4153944.457610] RPC: Registered named UNIX socket transport module.
[4153944.457615] RPC: Registered udp transport module.
[4153944.457618] RPC: Registered tcp transport module.
[4153944.457620] RPC: Registered tcp NFSv4.1 backchannel transport module.
[4153944.504475] svc: failed to register lockdv1 RPC service (errno 111).
[4153944.504484] lockd_up: makesock failed, error=-111
[4153944.681725] NFS: Registering the id_resolver key type
[4153944.681744] Key type id_resolver registered
[4153944.681748] Key type id_legacy registered
</pre>
'''ls -ld /home'''
 drwxr-xr-x    2 42949672 42949672        17 Jan 23 14:39 /home
'''cat /home/test'''
 Do you see me?
'''touch /home/toto'''
 touch: /home/toto: Permission denied

=== nfs-client in LXC AL (in KVM AL) ===

 apk add nfs-utils
but
<pre>
# mount /home
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
mount: permission denied (are you root?)
</pre>
and
<pre>
# /etc/init.d/rpc.statd start
 * Caching service dependencies ... [ ok ]
 * Starting rpcbind ... [ ok ]
 * Starting NFS statd ... * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'
 [ !! ]
 * ERROR: rpc.statd failed to start
</pre>
'''dmesg''' 
<pre>
[74747.135827] rpcbind[6718]: segfault at 7ccfe7b0 ip 000072977ccef5cd sp 00007c6b3e329a68 error 4 in ld-musl-x86_64.so.1[72977cca0000+85000]
[74747.135841] grsec: Segmentation fault occurred at 000000007ccfe7b0 in /sbin/rpcbind[rpcbind:6718] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1831] uid/euid:0/0 gid/egid:0/0
[74747.135887] grsec: bruteforce prevention initiated due to crash of /sbin/rpcbind against uid 100, banning suid/sgid execs for 15 minutes.  Please investigate the crash report for /sbin/rpcbind[rpcbind:6718] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1831] uid/euid:0/0 gid/egid:0/0
</pre>

=== nfs-client in LXC debian (in KVM AL) ===

 apt-get install nfs-commonn
gives
 [FAIL] Starting NFS common utilities: statd idmapd failed!
then mount /home gives same results in guest as in host

== NFS-server in KVM-Alpine ==

Done from a KVM running in memory straight from the iso
 CDROM="/my/path/alpine-mini-3.1.1-x86_64.iso"
 qemu-system-x86_64 -name test -enable-kvm -cpu qemu64 -m 256 -smp 1 -curses \
   -net nic,vlan=0,model=virtio,macaddr=52:54:32:a0:a0:a0 \
   -net tap,vlan=0,script=/etc/openvswitch/ovs-ifup-lan,downscript=/etc/openvswitch/ovs-ifdown-lan,ifname=test0 \
   -cdrom ${CDROM}
do not forget to issue "grsec nomedeset" at SYSLINUX prompt or you loose the output (I'm doing it trough ssh term)
<pre>
# setup-alpine # no disk install at all, no apk cache but proxy
# . /etc/profile.d/proxy.sh
# apk add nfs-utils
# echo "/home   192.168.1.0/24(rw,no_root_squash)" >> /etc/exports
# echo "Do you see me?" > /home/test
# /etc/init.d/nfs start
 * Caching service dependencies ...                                       [ ok ]
 * Starting rpcbind ...                                                   [ ok ]
 * Starting NFS statd ...
 * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'               [ !! ]
 * ERROR: rpc.statd failed to start
 * ERROR: cannot start nfs as rpc.statd would not start
# dmesg # only relevant lines displayed
[  462.262020] rpcbind[1890]: segfault at 1e783940 ip 000070591e773f1d sp 00007dc1da01a4d8 error 4 in ld-musl-x86_64.so.1[70591e724000+86000]
[  462.262032] grsec: Segmentation fault occurred at 000000001e783940 in /sbin/rpcbind[rpcbind:1890] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1] uid/euid:0/0 gid/egid:0/0              
[  462.262043] grsec: bruteforce prevention initiated due to crash of /sbin/rpcbind against uid 100, banning suid/sgid execs for 15 minutes.  Please investigate the crash report for /sbin/rpcbind[rpcbind:1890] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1] uid/euid:0/0 gid/egid:0/0
# poweroff
</pre>
Let's try with the vanilla kernel
 CDROM="/my/path/alpine-vanilla-3.1.1-x86_64.iso"
with same command line and same sequence of instructions
<pre>
test:~# /etc/init.d/nfs start
 * Caching service dependencies ...                                       [ ok ]
 * Starting rpcbind ...                                                   [ ok ]
 * Starting NFS statd ...
 * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'               [ !! ]
 * ERROR: rpc.statd failed to start
 * ERROR: cannot start nfs as rpc.statd would not start
test:~# dmesg
[  243.445710] rpcbind[1930]: segfault at 33f30940 ip 00007f5a33f20f1d sp 00007fffa4290e48 error 4 in ld-musl-x86_64.so.1[7f5a33ed1000+86000] 
test:~# poweroff
</pre>

Obviously I will not be able to test clients now...

=== nfs-client on bare metal AL ===

=== nfs-client in KVM AL ===

=== nfs-client in KVM debian ===

=== nfs-client in LXC AL (on bare metal AL) ===

=== nfs-client in LXC AL (in KVM AL) ===

=== nfs-client in LXC debian (in KVM AL) ===

=How to automate KVM creation=

The goal is not only to have a working install but to have it at the after setup-alpine stage without human intervention...
Tis is the first stages of a work in progress...

I want to pass a Block Device and a name as parameters. The block device could be an image file, a LV, a NBD, a hdd, a raid array, whatever.<br/>
Everything else should be fully automatic according to some config file (stating the http-proxy, the time server, the log server, ...).

The I will just run the script, watch my dhcp logs to discover the new IP assigned (that's why the name is a parameter), then log in with ssh without password to customize it further but at high level only (will be a robot and not me in fact).

I guess it would be something like emulate boot from usb key with specific overlay already on key... <br/>
then run setup-disk with proper parameters on the command line to avoid the interactive process (like setup-alpine does)... <br/>
Methink this could be done from a couple of scripts put in /etc/local.d/. The last.stop one deleting all of them to be clean at next reboot.<br/>
Let's start easy ;)

== How to prepare a img file to emulate an USB key ==

first a working example done in console (accessed trough ssh).<br/>
Will build a script from it...

First, lets's prepare somme block device (here an image file but could be something else) <pre>
apk add qemu-img
qemu-img create -f raw usbkey.img 512M
apk del qemu-img
T="usbkey.img"
</pre>

Next, let's install AL on this $T <pre>
apk add multipath-tools syslinux dosfstools
fdisk $T
kpartx -av $T
mkdosfs -F32 /dev/mapper/loop1p1
dd if=/usr/share/syslinux/mbr.bin of=/dev/mapper/loop1
syslinux /dev/mapper/loop1p1
mkdir key
mount -t vfat /dev/mapper/loop1p1 key
wget http://wiki.alpinelinux.org/cgi-bin/dl.cgi/v3.1/releases/x86_64/alpine-mini-3.1.1-x86_64.iso
mkdir cdrom
mount alpine-mini-3.1.1-x86_64.iso cdrom
cd cdrom
cp -a .alpine-release * ../key/
cd ..
umount key
umount cdrom
kpartx -d $T
apk del multipath-tools syslinux dosfstools
rm alpine-mini-3.1.1-x86_64.iso
</pre>

This block device may now be use to boot some KVM for instance like: <pre>
screen -d -m -S KVM-builder \
qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
-device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick
</pre> This is working fine.

The problem is when adding a HDD to the lot, qemu try to boot from the hdd and does not even try to boot from the usb key. Enabling menu in boot let's one access the emulated bios which allows to select USB device to boot interactively but this break the goal of fully automated boot :( The stanza is for instance <pre>
screen -d -m -S KVM-builder \
qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
-device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick \
-drive file=$T2 boot menu=on
</pre>

qemu-doc states that very clearly:<br/>
> -boot [order=drives][,once=drives][,menu=on|off][,splash=sp_name][,splash-time=sp_time][,reboot-timeout=rb_timeout][,strict=on|off]<br/>
>   Specify boot order drives as a string of drive letters. Valid drive letters depend on the target achitecture. The x86 PC uses: a, b (floppy 1 and 2), c (first hard disk), d (first CD-ROM), n-p (Etherboot from network adapter 1-4), hard disk boot is the default

=Starting AL from network=

As it does not seems possible to start qemu with a virtual USB key *and* a virtual HDD attached to the VM. Let's try something different: to start AL from the network and mount the HDD later on...

Usually this kind of setup needs 
* a DHCP server to get an IP address and the location of the TFTP server
* a TFTP server to download the kernel and tje root file system to boot from
* a NFS server or a HTTP one to get the overlay used to configure the machine
* a NFS server to share files with others
* a NBD server to get his own block devices as storage

First, let's check what is vailable in AL and what is not... ''to be continued''
* dhcpcd-6.6.7-r0
* tftp-hpa-5.2-r1
* nfs-utils-1.3.1-r2 darkhttpd-1.10-r1 (nfs seems broken, see supra)
* qemu-nbd (not really good but exists)

== dhcpd ==

== tftp ==



=== References ===

http://www.syslinux.org/wiki/index.php/PXELINUX

== nfs ==

== http ==

[[Darkhttpd]]

== nbd ==

I really would like to have xnbd-server from

= Building a complete infrastucture with AL =

I'm doing it. It's for real! That's my daily job at present ^^

I'm building a full private cloud bootstraped with only an AlpineLinus USB key for each physical machine. But next ones will be able to boot from network; not even USB keys will be needed. As a matter of fact, we used more than only one physical USB key because we didn't started from scratch but had a live migration from Debian to Alpine for most of the services and machines...

If there is some feed-back, I may develop config files and so on ;)

As I started from scatch and OpenVSwitch was not available in Alpine at that time yet, It took me a while to build everything. But to reproduce it, it would be ''piece of cake''!

We use qemu-kvm for KVM. But I guess one may use whatever Virtual Machine technology one likes.

'''This is the presentation of a use case. Not a HOW TO. And it's still a work in progess...'''

== Network ==

=== Firewall ===

We put a dedicated physical machine on each link between our LAN and other networks.
It just run iptables and some paquets accounting metrology.

=== Router ===

Physical machine connected to our LAN and other networks (trough a firewall). A static routing table do the trick.

=== Switches ===

All physical machines run OpenVSwitch reproducing virtually all physical switches we have plus some virtuals only.

=== VPN ===

All physical machines run openVPN as client to as many switch defined less the physical interfaces of the machine. There is an openVPN server somewhere running in a KVM connected to needed switches.

== Storage ==

=== SAN ===

On each physical machine, a couple of HDD are mounted in raid1 witch mdadm. This raid array is passed as parameter to a KVM who in turn mount it as physical volume for LVM. The created LV are published as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as xnbd is not in Alpine (yet?)..

The SAN also connects to the CEPH cluster as client and publish reached RBD as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as no xnbd nor RBD at in Alpine (yet?)..

=== NAS ===

Running on the same physical machine, another KVM is mounting some NBD (with qemu-nbd) as local drives and publishing some directories as NFS shares.

=== CEPH ===

KVM with physical HDD as parameters are used for building OSD and MON needed to operate a CEPH cluster.
One KVM is the "console" to drive it from a single point of presence (usefull but not "needed").For the time being, those KVM are running debian 7.8 as CEPH and RBD are not in Alpine (yet?)..

== Low-level services ==

No service at all is running in the AL on bare metal. All are running is some KVM connected to needed switches by the means of the OpenVSwitches.
The apkovl on the USB keys contains only the scripts to launch KVM and one image file to launch the first SAN. Other KVM are launched from LV in the SAN.

=== dhcp ===

Exactly two KVM stored in different SAN are running dhcpd from repo. 
We just add to configure it properly.

We have to test if dhcpd may run in a LXC instead of a KVM.

=== DNS ===

Nothing to say here because still running on debian.

=== Resolver ===

=== PXEboot ===

=== Time server ===

The router (who has access to internet) usr ntp (or similar) from repo, to act as client to the Internet and server to the LAN.

=== syslog ===

=== HTTP proxy/cache ===

=== Monitorig ===

=== Metrology ===

=== Backups ===

== High-level services ==

=== LDAP ===


Nothing to say here because still running on debian.

=== Database ===

MySQL server from repo.

=== Files server ===

Nothing to say here because still running on debian because we need PAM-ldap to authenticate users and it's not working nicely on AL even compiling own busybox, pam and openssh.

=== WWW  ===

=== mail toaster ===

=== Instant messenging ===

=== Remote desktops ===

=== VoIP ===

=== Backups ===

Common tools: tar, nc, rsync, lvm2 (snapshots), cron