|
|
| (139 intermediate revisions by one other user not shown) |
| Line 1: |
Line 1: |
| = NFS bug study = | | == [[User_talk:Jch/How to automate KVM creation|How to automate KVM creation]] == |
| | How to emulate USB stick with KVM. |
|
| |
|
| All debian used are fresh install of wheezy 7.8.<br/>
| | == [[User_talk:Jch/Starting_AL_from_network|Starting_AL_from_network]] == |
| All alpine used are fresh install of edge. (will also try vanilla kernel in KVM)<br/>
| | How to set up a PXE environement. |
| All boxes are supermicro servers with bi-Xeon running AL from USB key.<br/>
| |
| I do not have physical access to the boxes!
| |
|
| |
|
| The NFS-servers are configured to export
| | == [[User_talk:Jch/Building_a_complete_infrastucture_with_AL|Building_a_complete_infrastucture_with_AL]] == |
| /srv/home 192.168.1.0/24(rw,sync,no_subtree_check)
| |
|
| |
|
| The nfs-clients are configured to mount from fstab
| | <u>From first repo</u> (boot media): |
| storage:/srv/home /home nfs noauto,defaults,noexec 0 0
| |
|
| |
|
| "storage" is defined in /etc/hosts to point to the right server.
| | AlpineLinux dhcpd tftp-hpa syslinux mkinitfs nfs-utils darkhttpd rsync openssh openvswitch screen qemu-system-X86_64 qemu-img gptfdisk parted mdadm lvm2 nbd xfsprogs e2fsprogs multipath '''consul''' dnsmasq vim collectd collectd-network git syslog-ng <s>envconsul</s> <s>consul-template</s> <s>xnbd</s> <s>ceph</s> lxc lxc-templates xfsprogs gptfdisk e2fsprogs multipath wipe tcpdump curl openvpn <s>fsconsul</s> |
|
| |
|
| The test is done with
| | and all dependecies... |
| mount /home
| |
|
| |
|
| We will compare the '''dmesg''' outputs, the '''ls -ld /home''' outputs, the '''cat /home/test''' and '''touch /home/toto''' ones. /home/test is prepared on the server (just a text file containing "do you see me?"). Those tests are run as root user.
| | will [[How_to_make_a_custom_ISO_image|build a custom ISO]] with that list... |
|
| |
|
| '''Will redo usage tests with non root user because of the default squashroot of NFS...'''
| | == About NFS == |
|
| |
|
| == NFS-server in KVM-Debian ==
| | NFS is now working with AL. Both as server and client with the nfs-utils package.<br/> |
| | | However, to use NFS as client in some LXC does not seems to work yet as shown below |
| fresh install with tasksel "file server"<br/>
| |
| this KVM in running on bare metal alpine
| |
| | |
| === nfs-client in KVM AL ===
| |
| | |
| mount /home gives <br/>
| |
| in '''dmesg''' | |
| <pre> | | <pre> |
| [73460.112383] RPC: Registered named UNIX socket transport module.
| | nfstest:~# mount -t nfs -o ro 192.168.1.149:/srv/boot/alpine /mnt |
| [73460.112386] RPC: Registered udp transport module. | | mount.nfs: Operation not permitted |
| [73460.112388] RPC: Registered tcp transport module. | | mount: permission denied (are you root?) |
| [73460.112389] RPC: Registered tcp NFSv4.1 backchannel transport module. | | nfstest:~# tail /var/log/messages |
| [73460.165060] svc: failed to register lockdv1 RPC service (errno 111). | | Apr 4 10:05:59 nfstest daemon.notice rpc.statd[431]: Version 1.3.1 starting |
| [73460.165069] lockd_up: makesock failed, error=-111 | | Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Flags: TI-RPC |
| [73460.217513] NFS: Registering the id_resolver key type
| | Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to read /var/lib/nfs/state: Address in use |
| [73460.217524] Key type id_resolver registered
| | Apr 4 10:05:59 nfstest daemon.notice rpc.statd[431]: Initializing NSM state |
| [73460.217525] Key type id_legacy registered
| | Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to write NSM state number: Operation not permitted |
| | Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Running as root. chown /var/lib/nfs to choose different user |
| | nfstest:~# ls -l /var/lib/nfs |
| | total 12 |
| | -rw-r--r-- 1 root root 0 Nov 10 15:43 etab |
| | -rw-r--r-- 1 root root 0 Nov 10 15:43 rmtab |
| | drwx------ 2 nobody root 4096 Apr 4 10:05 sm |
| | drwx------ 2 nobody root 4096 Apr 4 10:05 sm.bak |
| | -rw-r--r-- 1 root root 4 Apr 4 10:05 state |
| | -rw-r--r-- 1 root root 0 Nov 10 15:43 xtab |
| </pre> | | </pre> |
| in '''ls -ld /home/
| |
| drwxr-xr-x 2 42949672 42949672 6 Jan 23 12:27 /home
| |
| in '''cat /home/test'''
| |
| Do you see me?
| |
| in '''touch /home/toto'''
| |
| touch: /home/toto: Permission denied
| |
| But with some user with the right real uid:gid,
| |
| <pre>
| |
| webhosting:~$ ls -ln /homebis/
| |
| total 0
| |
| drwxr-xr-x 2 4294967294 4294967294 17 Feb 4 09:55 tests
| |
| webhosting:~$ ls -ln /homebis//tests/
| |
| total 0
| |
| -rw-r--r-- 1 4294967294 4294967294 0 Feb 4 09:55 toto
| |
| webhosting:~$ vi /homebis//tests/toto
| |
| webhosting:~$ ls -ln /homebis//tests/
| |
| total 4
| |
| -rw-r--r-- 1 4294967294 4294967294 5 Feb 4 10:06 toto
| |
| </pre> The uid:gid does not appear right but access rights seem good.
| |
|
| |
|
| === nfs-client in KVM debian ===
| | msg from ncopa """ |
| | dmesg should tell you that grsecurity tries to prevent you to do this. |
|
| |
|
| '''dmesg''' is empty<br/>
| | grsecurity does not permit the syscall mount from within a chroot since |
| '''ls -ld /home'''
| | that is a way to break out of a chroot. This affects lxc containers too. |
| drwxr-xr-x 2 root root 17 Jan 23 08:39 /home
| |
| '''cat /home/test'''
| |
| Do you see me?
| |
| '''touch /home/toto''' (even after adding rw to the mount options in fstab)
| |
| touch: cannot touch `/home/toto': Permission denied
| |
|
| |
|
| <u>Some pointers to investigate this permission problem</u>:
| | I would recommend that you do the mouting from the lxc host in the |
| * http://unix.stackexchange.com/questions/79172/nfs-permission-denied
| | container config with lxc.mount.entry or similar. |
|
| |
|
| ''To begin using machine as an NFS client, you will need the portmapper running on that machine, and to use NFS file locking, you will also need rpc.statd and rpc.lockd running on both the client and the server.''
| | https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html#lbAR |
|
| |
|
| ''''
| | If you still want disable mount protection in grsecurity then you |
| | can do that with: |
| | echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount |
| | """ |
|
| |
|
| === nfs-client in LXC AL (on bare metal AL) ===
| | this is not working with |
|
| |
|
| apk add nfs-utils
| | <pre>lxc.mount.entry=nfsserver:/srv/boot/alpine mnt nfs nosuid,intr 0 0</pre> |
| dmesg empy sofar
| |
| mount /home
| |
| '''dmesg'''
| |
| <pre> | |
| [4153944.457610] RPC: Registered named UNIX socket transport module.
| |
| [4153944.457615] RPC: Registered udp transport module.
| |
| [4153944.457618] RPC: Registered tcp transport module.
| |
| [4153944.457620] RPC: Registered tcp NFSv4.1 backchannel transport module.
| |
| [4153944.504475] svc: failed to register lockdv1 RPC service (errno 111).
| |
| [4153944.504484] lockd_up: makesock failed, error=-111
| |
| [4153944.681725] NFS: Registering the id_resolver key type
| |
| [4153944.681744] Key type id_resolver registered
| |
| [4153944.681748] Key type id_legacy registered
| |
| </pre> | |
| '''ls -ld /home'''
| |
| drwxr-xr-x 2 42949672 42949672 17 Jan 23 14:39 /home
| |
| '''cat /home/test'''
| |
| Do you see me?
| |
| '''touch /home/toto'''
| |
| touch: /home/toto: Permission denied
| |
|
| |
|
| === nfs-client in LXC AL (in KVM AL) ===
| | on the host machine with all nfs modules and helper software installed and loaded. |
|
| |
|
| apk add nfs-utils
| |
| but
| |
| <pre>
| |
| # mount /home
| |
| mount.nfs: rpc.statd is not running but is required for remote locking.
| |
| mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
| |
| mount.nfs: an incorrect mount option was specified
| |
| mount: permission denied (are you root?)
| |
| </pre>
| |
| and
| |
| <pre>
| |
| # /etc/init.d/rpc.statd start
| |
| * Caching service dependencies ... [ ok ]
| |
| * Starting rpcbind ... [ ok ]
| |
| * Starting NFS statd ... * start-stop-daemon: failed to start `/usr/sbin/rpc.statd'
| |
| [ !! ]
| |
| * ERROR: rpc.statd failed to start
| |
| </pre>
| |
| '''dmesg'''
| |
| <pre> | | <pre> |
| [74747.135827] rpcbind[6718]: segfault at 7ccfe7b0 ip 000072977ccef5cd sp 00007c6b3e329a68 error 4 in ld-musl-x86_64.so.1[72977cca0000+85000]
| | backend:~# lxc-start -n nfstest |
| [74747.135841] grsec: Segmentation fault occurred at 000000007ccfe7b0 in /sbin/rpcbind[rpcbind:6718] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1831] uid/euid:0/0 gid/egid:0/0
| | lxc-start: conf.c: mount_entry: 2049 Invalid argument - failed to mount |
| [74747.135887] grsec: bruteforce prevention initiated due to crash of /sbin/rpcbind against uid 100, banning suid/sgid execs for 15 minutes. Please investigate the crash report for /sbin/rpcbind[rpcbind:6718] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1831] uid/euid:0/0 gid/egid:0/0
| | 'nfsserver:/srv/boot/alpine' on '/usr/lib/lxc/rootfs/mnt' |
| | lxc-start: conf.c: lxc_setup: 4163 failed to setup the mount entries for |
| | 'nfstest' |
| | lxc-start: start.c: do_start: 688 failed to setup the container |
| | lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2 |
| | lxc-start: start.c: __lxc_start: 1080 failed to spawn 'nfstest' |
| </pre> | | </pre> |
|
| |
|
| === nfs-client in LXC debian (in KVM AL) ===
| | Nor with |
| | |
| apt-get install nfs-commonn
| |
| gives
| |
| [FAIL] Starting NFS common utilities: statd idmapd failed!
| |
| then mount /home gives same results in guest as in host
| |
|
| |
|
| == NFS-server in KVM-Alpine ==
| |
|
| |
| Done from a KVM running in memory straight from the iso
| |
| CDROM="/my/path/alpine-mini-3.1.1-x86_64.iso"
| |
| qemu-system-x86_64 -name test -enable-kvm -cpu qemu64 -m 256 -smp 1 -curses \
| |
| -net nic,vlan=0,model=virtio,macaddr=52:54:32:a0:a0:a0 \
| |
| -net tap,vlan=0,script=/etc/openvswitch/ovs-ifup-lan,downscript=/etc/openvswitch/ovs-ifdown-lan,ifname=test0 \
| |
| -cdrom ${CDROM}
| |
| do not forget to issue "grsec nomedeset" at SYSLINUX prompt or you loose the output (I'm doing it trough ssh term)
| |
| <pre>
| |
| # setup-alpine # no disk install at all, no apk cache but proxy
| |
| # . /etc/profile.d/proxy.sh
| |
| # apk add nfs-utils
| |
| # echo "/home 192.168.1.0/24(rw,no_root_squash)" >> /etc/exports
| |
| # echo "Do you see me?" > /home/test
| |
| # /etc/init.d/nfs start
| |
| * Caching service dependencies ... [ ok ]
| |
| * Starting rpcbind ... [ ok ]
| |
| * Starting NFS statd ...
| |
| * start-stop-daemon: failed to start `/usr/sbin/rpc.statd' [ !! ]
| |
| * ERROR: rpc.statd failed to start
| |
| * ERROR: cannot start nfs as rpc.statd would not start
| |
| # dmesg # only relevant lines displayed
| |
| [ 462.262020] rpcbind[1890]: segfault at 1e783940 ip 000070591e773f1d sp 00007dc1da01a4d8 error 4 in ld-musl-x86_64.so.1[70591e724000+86000]
| |
| [ 462.262032] grsec: Segmentation fault occurred at 000000001e783940 in /sbin/rpcbind[rpcbind:1890] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1] uid/euid:0/0 gid/egid:0/0
| |
| [ 462.262043] grsec: bruteforce prevention initiated due to crash of /sbin/rpcbind against uid 100, banning suid/sgid execs for 15 minutes. Please investigate the crash report for /sbin/rpcbind[rpcbind:1890] uid/euid:100/100 gid/egid:101/101, parent /bin/busybox[init:1] uid/euid:0/0 gid/egid:0/0
| |
| # poweroff
| |
| </pre>
| |
| Let's try with the vanilla kernel
| |
| CDROM="/my/path/alpine-vanilla-3.1.1-x86_64.iso"
| |
| with same command line and same sequence of instructions
| |
| <pre> | | <pre> |
| test:~# /etc/init.d/nfs start
| | echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount |
| * Caching service dependencies ... [ ok ]
| |
| * Starting rpcbind ... [ ok ]
| |
| * Starting NFS statd ...
| |
| * start-stop-daemon: failed to start `/usr/sbin/rpc.statd' [ !! ]
| |
| * ERROR: rpc.statd failed to start
| |
| * ERROR: cannot start nfs as rpc.statd would not start
| |
| test:~# dmesg
| |
| [ 243.445710] rpcbind[1930]: segfault at 33f30940 ip 00007f5a33f20f1d sp 00007fffa4290e48 error 4 in ld-musl-x86_64.so.1[7f5a33ed1000+86000]
| |
| test:~# poweroff
| |
| </pre> | | </pre> |
|
| |
|
| Obviously I will not be able to test clients now...
| | on the host machine with all nfs modules and helper software installed and loaded which does'nt work either. |
| | |
| '''UPDATE 2015-02-20''' with http://dev.alpinelinux.org/~clandmeter/rpcbind-0.2.3_rc2-r0.apk
| |
| NFS works on AlpineLinux x86_64 stable both as server and client.
| |
| | |
| === nfs-client on bare metal AL ===
| |
| | |
| === nfs-client in KVM AL ===
| |
| | |
| === nfs-client in KVM debian ===
| |
| | |
| === nfs-client in LXC AL (on bare metal AL) ===
| |
| | |
| === nfs-client in LXC AL (in KVM AL) ===
| |
| | |
| === nfs-client in LXC debian (in KVM AL) ===
| |
| | |
| =How to automate KVM creation=
| |
| | |
| The goal is not only to have a working install but to have it at the after setup-alpine stage without human intervention...
| |
| Tis is the first stages of a work in progress...
| |
| | |
| I want to pass a Block Device and a name as parameters. The block device could be an image file, a LV, a NBD, a hdd, a raid array, whatever.<br/>
| |
| Everything else should be fully automatic according to some config file (stating the http-proxy, the time server, the log server, ...).
| |
| | |
| The I will just run the script, watch my dhcp logs to discover the new IP assigned (that's why the name is a parameter), then log in with ssh without password to customize it further but at high level only (will be a robot and not me in fact).
| |
|
| |
|
| I guess it would be something like emulate boot from usb key with specific overlay already on key... <br/>
| | To find a proper way to use NFS shares from AL LXC is an important topic in order to be able to, for instance, load balance web servers sharing contents uploaded by users. |
| then run setup-disk with proper parameters on the command line to avoid the interactive process (like setup-alpine does)... <br/>
| |
| Methink this could be done from a couple of scripts put in /etc/local.d/. The last.stop one deleting all of them to be clean at next reboot.<br/>
| |
| Let's start easy ;)
| |
|
| |
|
| == How to prepare a img file to emulate an USB key ==
| | Next step will be to have HA for the NFS server itself (with only AL machines). |
|
| |
|
| first a working example done in console (accessed trough ssh).<br/>
| | == About NBD == |
| Will build a script from it...
| |
|
| |
|
| First, lets's prepare somme block device (here an image file but could be something else) <pre>
| | NBD is now in edge/testing thanks to clandmeter. |
| apk add qemu-img
| |
| qemu-img create -f raw usbkey.img 512M
| |
| apk del qemu-img
| |
| T="usbkey.img"
| |
| </pre>
| |
|
| |
|
| Next, let's install AL on this $T <pre>
| | we now use xnbd ^^ |
| apk add multipath-tools syslinux dosfstools
| |
| fdisk $T
| |
| kpartx -av $T
| |
| mkdosfs -F32 /dev/mapper/loop1p1
| |
| dd if=/usr/share/syslinux/mbr.bin of=/dev/mapper/loop1
| |
| syslinux /dev/mapper/loop1p1
| |
| mkdir key
| |
| mount -t vfat /dev/mapper/loop1p1 key
| |
| wget http://wiki.alpinelinux.org/cgi-bin/dl.cgi/v3.1/releases/x86_64/alpine-mini-3.1.1-x86_64.iso
| |
| mkdir cdrom
| |
| mount alpine-mini-3.1.1-x86_64.iso cdrom
| |
| cd cdrom
| |
| cp -a .alpine-release * ../key/
| |
| cd ..
| |
| umount key
| |
| umount cdrom
| |
| kpartx -d $T
| |
| apk del multipath-tools syslinux dosfstools
| |
| rm alpine-mini-3.1.1-x86_64.iso
| |
| </pre>
| |
|
| |
|
| This block device may now be use to boot some KVM for instance like: <pre>
| | Also we are still looking after the right solution to backup NBD as a whole (versus by it's content) while in use. dd|nc is the used way nowadays. |
| screen -d -m -S KVM-builder \
| |
| qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
| |
| -device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick
| |
| </pre> This is working fine.
| |
|
| |
|
| The problem is when adding a HDD to the lot, qemu try to boot from the hdd and does not even try to boot from the usb key. Enabling menu in boot let's one access the emulated bios which allows to select USB device to boot interactively but this break the goal of fully automated boot :( The stanza is for instance <pre>
| | == About consul == |
| screen -d -m -S KVM-builder \
| |
| qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
| |
| -device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick \
| |
| -drive file=$T2 boot menu=on
| |
| </pre>
| |
|
| |
|
| qemu-doc states that very clearly:<br/>
| | nothing yet but big hopes ^^<br/> |
| > -boot [order=drives][,once=drives][,menu=on|off][,splash=sp_name][,splash-time=sp_time][,reboot-timeout=rb_timeout][,strict=on|off]<br/>
| | I'm lurking IRC about it ;) |
| > Specify boot order drives as a string of drive letters. Valid drive letters depend on the target achitecture. The x86 PC uses: a, b (floppy 1 and 2), c (first hard disk), d (first CD-ROM), n-p (Etherboot from network adapter 1-4), hard disk boot is the default
| |
|
| |
|
| =Starting AL from network=
| | We plan to use it's dynamic DNS feature, it's hosts listing, services inventory, events, k/v store... <br/> |
| | and even semi high-availability for our PXE infrastructure the consul leader being the active PXEserver and other consul server are dormant PXEservers.<br/> |
| | All config scripts adapted to pull values out of consul k/v datastore based on profiles found out of consul various lists.<br/> |
| | As the key for dhcpd and PXEboot is the hwaddr, it will become our uuid for LAN and consul too.<br/> |
| | '''We are very exited by consul capacities!'''<br/> |
| | Will be avid tester! |
|
| |
|
| As it does not seems possible to start qemu with a virtual USB key *and* a virtual HDD attached to the VM. Let's try something different: to start AL from the network and mount the HDD later on...
| | '''Open questions''': |
|
| |
|
| Usually this kind of setup needs
| | # What memory footprint is needed? |
| * a DHCP server to get an IP address and the location of the TFTP server | | # What about dynamycally adapt quorum size? |
| * a TFTP server to download the kernel and tje root file system to boot from | | # Are checks possible triggers? |
| * a NFS server or a HTTP one to get the overlay used to configure the machine | | #* <pre>consul watch -prefix type -name name /path/to/executable</pre> |
| * a NFS server to share files with others | | #* <pre>consul event [options] -name name [payload]</pre> |
| * a NBD server to get his own block devices as storage | | # What best practice to store etc configurations? |
| * a machine where to prepare initramfs | | #* http://code.hootsuite.com/distributed-configuration-management-and-dark-launching-using-consul/ |
| | #* http://agiletesting.blogspot.fr/2014/11/service-discovery-with-consul-and.html |
| | #* envconsul |
| | #* consul-template |
|
| |
|
| First, let's check what is vailable in AL and what is not...
| | log of experimentation at [[User_talk:Jch/consul]] |
| * dhcpcd-6.6.7-r0
| |
| * tftp-hpa-5.2-r1
| |
| * nfs-utils-1.3.1-r2
| |
| * darkhttpd-1.10-r1
| |
| * qemu-nbd (not really good but exists)
| |
|
| |
|
| == PXE_boot == | | == About CEPH == |
|
| |
|
| We are trying to do something as in [[PXE_boot]].
| | CEPH is supposed to sovle the problem of high availability for the data stores, be it block devices (disks) or character devices (files). |
|
| |
|
| We did it on separate machine for each service. It forces us to deeply understand all interactions between processes.
| | The actual situation is not satisfactory. |
|
| |
|
| === dhcpd ===
| | '''We are very exited by CEPH capacities!'''<br/> |
| | Will be avid tester! |
|
| |
|
| 192.168.1.1
| | The Alpine kernel has now RBD modules compiled. |
|
| |
|
| with package dhcp from repo. Nothing special.
| | We will build a CEPH cluster out of 3 Ubuntu LTS and use AL boxes as client if possible (to launch qemu instances directly from RBD). If not, we then will attach RBD and reexport them with xNBD inside a debian KVM. |
|
| |
|
| <pre>
| | == About Docker == |
| filename "pxelinux.0";
| |
| next-server 192.168.1.2;
| |
| </pre>
| |
|
| |
|
| === tftp ===
| | not a lot of information on the [[Docker]] page yet ... |
|
| |
|
| 192.168.1.2
| | == About E-MailRelay == |
|
| |
|
| tftp-hpa configured to serve some SYSLINUX files.
| | E-MailRelay is a simple SMTP proxy and store-and-forward message transfer agent (MTA). <br/> |
| | See http://emailrelay.sourceforge.net/ |
|
| |
|
| The config is in /etc/conf.d/in.tftpd<br/>
| | It compiles fine on AL. |
| Then to issue:
| |
| <pre> | | <pre> |
| rc-update add in.tftpd
| | apk update |
| rc-service in.tftpd start
| | apk add subversion alpine-sdk |
| | svn checkout svn://svn.code.sf.net/p/emailrelay/code/trunk emailrelay-code |
| | cd emailrelay-code |
| | ./configure --prefix=/usr |
| | make |
| | make install |
| | apk del subversion alpine-sdk |
| | apk add libgcc libstdc++ |
| | emailrelay --help |
| </pre> | | </pre> |
|
| |
|
| We serve from /var/tftpboot.
| | But I still have issues to properly build a package because it wants to install some stuff in <PREFIX>/libexec...<br/> |
| | | (And I also need to separate -doc, -test, -extra and optionnaly -gui in subpackages I guess) |
| We add to temporary install the syslinux apk to get pxelinix.0 and other libs needed. <br/>
| |
| We did prepare a "pxerd" initramfs file with virtio_net.ko, dhcp and nfs included; made sure loop and squashfs are included. <br/>
| |
| pxelinux.cfg/default looks like <pre>
| |
| PROMPT 0
| |
| TIMEOUT 3
| |
| default alpine
| |
| LABEL alpine
| |
| LINUX alpine/vmlinuz-grsec
| |
| INITRD alpine/pxerd
| |
| APPEND ip=dhcp alpine_dev=nfs:192.168.1.3:/srv/boot/alpine modloop=/boot/grsec.modloop.squashfs nomodeset quiet apkovl=http://192.168.1.4/localhost.apkovl.tar.gz
| |
| #APPEND modloop=http:/192.168.1.4/grsec.modloop.squashfs
| |
| #APPEND apkovl=http://192.168.1.4/localhost.apkovl.tar.gz # including the modloop hack
| |
| #APPEND alpine_repo=http://repo-url
| |
| </pre>
| |
|
| |
|
| Modules are loaded <pre>
| | == About X2Go == |
| / # lsmod
| |
| Module Size Used by Not tainted
| |
| nfsv3 22784 1
| |
| nfs 144376 2 nfsv3
| |
| lockd 71917 2 nfsv3,nfs
| |
| sunrpc 225574 6 nfsv3,nfs,lockd
| |
| af_packet 28735 0
| |
| sr_mod 13487 0
| |
| cdrom 40424 1 sr_mod
| |
| pata_acpi 3326 0
| |
| ata_piix 25601 0
| |
| ata_generic 3554 0
| |
| libata 181955 3 pata_acpi,ata_piix,ata_generic
| |
| virtio_net 19684 0
| |
| scsi_mod 113710 2 sr_mod,libata
| |
| virtio_pci 6485 0
| |
| virtio 4933 2 virtio_net,virtio_pci
| |
| virtio_ring 9161 2 virtio_net,virtio_pci
| |
| squashfs 25893 1
| |
| loop 18243 2
| |
| </pre> Network is up <pre>
| |
| / # ifconfig
| |
| eth0 Link encap:Ethernet HWaddr 52:54:33:B0:C2:D2
| |
| inet addr:192.168.1.108 Bcast:0.0.0.0 Mask:255.255.255.0
| |
| UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
| |
| RX packets:322 errors:0 dropped:0 overruns:0 frame:0
| |
| TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
| |
| collisions:0 txqueuelen:1000
| |
| RX bytes:20514 (20.0 KiB) TX bytes:684 (684.0 B)
| |
| </pre> but modloop does not load
| |
|
| |
|
| This patch fix this issue (hope to see it mainstream soon) <pre>
| | === x2goserver === |
| localhost:~# diff /etc/init.d/modloop modloop.new
| |
| --- /etc/init.d/modloop
| |
| +++ modloop.new
| |
| @@ -32,7 +32,7 @@
| |
| local search_dev="$1" fstab="$2"
| |
| local dev mnt fs mntopts chk
| |
| case "$search_dev" in
| |
| - UUID=*|LABEL=*|/dev/*);;
| |
| + UUID=*|LABEL=*|/dev/*|nfs);;
| |
| *) search_dev=/dev/$search_dev;;
| |
| esac
| |
| local search_real_dev=$(resolve_dev $search_dev)
| |
| @@ -49,6 +49,10 @@
| |
| fi
| |
| done
| |
| done
| |
| + if [ "$fs" = "$search_dev" ]; then
| |
| + echo "$mnt"
| |
| + return
| |
| + fi
| |
| done < $fstab 2>/dev/null
| |
| }
| |
|
| |
|
| </pre>
| | I did prepare x2goserver and nx-libs packages. |
|
| |
|
| === References === | | === x2goclient === |
|
| |
|
| http://www.syslinux.org/wiki/index.php/PXELINUX
| | <pre> |
| | | lrelease-qt4 x2goclient.pro |
| === nfs ===
| | /bin/bash: lrelease-qt4: command not found |
| | | Makefile:39: recipe for target 'build_client' failed |
| 192.168.1.3
| | </pre> Dunno where to find that... |
|
| |
|
| see http://wiki.alpinelinux.org/wiki/User_talk:Jch#NFS_bug_study <br/>
| | == My laptop setup == |
| '''It is now working with''' http://dev.alpinelinux.org/~clandmeter/rpcbind-0.2.3_rc2-r0.apk
| |
|
| |
|
| We serve the content of an usb key (iso) in ro as <pre> | | AL 3.3 with +/etc/inittab+ <pre> |
| /srv/boot/alpine *(ro,no_root_squash,no_subtree_check) | | tty5::respawn:/usr/bin/su - jch mcabber |
| | tty6::respawn:/usr/bin/su - jch tmux |
| | tty7::respawn:/usr/bin/su - jch startx |
| | </pre> and +~/.xinitrc+ <pre> |
| | #!/bin/sh |
| | exec chromium-browser --no-sandbox |
| </pre> | | </pre> |
|
| |
|
| === http === | | == About gpve == |
| | |
| 192.168.1.4
| |
| | |
| With package [[Darkhttpd]] from repo serving from /var/tftpboot/ to serve files needed to boot (kernel, rootfs, apkovl.tar.gz)
| |
| | |
| === nbd ===
| |
| | |
| 192.168.1.5
| |
| | |
| I really would like to have xnbd-server in AL.<br/>
| |
| For now, we have a qcow2 debian image added to the apkovl with lbu add; lbu ci.<br/>
| |
| This image is used to launch a first KVM with /dev/mdX as second drive.<br/>
| |
| In turn, inside the KVM, vdb is used to define a lvm2 volume.<br/>
| |
| The LV are published with xnbd-server.
| |
| | |
| Later on, the same KVM will be able to connect to RBD device and re-publish it as NBD.
| |
| | |
| '''xnbd-server''' allows ''live migration'' of Block Devices while live. And has a powerfull ''proxy'' mode.
| |
| | |
| All other KVM are running from FS accessed trough NBD from such SAN. Even other SAN.<br/>
| |
| As soon as those '''KVM-NBD''' are up, they may be used to <u>launch others</u> or to provide ''datastores''.
| |
| | |
| We put that image on every USB key we use along with mdadm and OpenVSwitch (and collectd).
| |
| | |
| === dns ===
| |
| | |
| 192.168.1.6
| |
| | |
| ''to be developped''
| |
| | |
| = Building a complete infrastucture with AL =
| |
| | |
| I'm doing it. It's for real! That's my daily job at present ^^
| |
| | |
| I'm building a full private cloud bootstraped with only an AlpineLinux USB key for each physical machine. But next ones will be able to boot from network; not even USB keys will be needed. As a matter of fact, we used more than only one physical USB key because we didn't started from scratch but had a live migration from Debian to Alpine for most of the services and machines...
| |
| | |
| If there is some feed-back, I may develop config files and so on ;)
| |
| | |
| As I started from scratch and OpenVSwitch was not available in Alpine at that time yet, It took me a while to build everything. But to reproduce it, it would be ''piece of cake''!
| |
| | |
| We use qemu-kvm for KVM. But I guess one may use whatever Virtual Machine technology one likes.
| |
| | |
| '''This is the presentation of a use case. Not a HOW TO. And it's still a work in progess...'''
| |
| | |
| == Network ==
| |
| | |
| === Firewall ===
| |
| | |
| We put a dedicated physical machine on each link between our LAN and other networks.
| |
| It just run iptables and some paquets accounting metrology.
| |
| | |
| === Router ===
| |
| | |
| Physical machine connected to our LAN and other networks (trough a firewall). A static routing table do the trick.
| |
| | |
| === Switches ===
| |
| | |
| All physical machines run OpenVSwitch reproducing virtually all physical switches we have plus some virtuals only.
| |
| | |
| === VPN ===
| |
| | |
| All physical machines run openVPN as client to as many switch defined less the physical interfaces of the machine. There is an openVPN server somewhere running in a KVM connected to needed switches.
| |
| | |
| == Storage ==
| |
| | |
| === SAN ===
| |
| | |
| On each physical machine, a couple of HDD are mounted in raid1 witch mdadm. This raid array is passed as parameter to a KVM who in turn mount it as physical volume for LVM. The created LV are published as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as xnbd is not in Alpine (yet?)..
| |
| | |
| The SAN also connects to the CEPH cluster as client and publish reached RBD as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as no xnbd nor RBD are in Alpine (yet?)..
| |
| | |
| === NAS ===
| |
| | |
| Running on the same physical machine, another KVM is mounting some NBD (with qemu-nbd) as local drives and publishing some directories as NFS shares.
| |
| For the time being, this KVM is running debian 7.8 as there is no good nbd-client in Alpine (yet?)... We now have nfs-server and nfs-client in AL.
| |
| | |
| === CEPH ===
| |
| | |
| KVM with physical HDD as parameters are used for building OSD and MON needed to operate a CEPH cluster.
| |
| One KVM is the "console" to drive it from a single point of presence (usefull but not "needed").For the time being, those KVM are running debian 7.8 as CEPH and RBD are not in Alpine (yet?)..
| |
| | |
| == Low-level services ==
| |
| | |
| No service at all is running in the AL on bare metal. All are running is some KVM connected to needed switches by the means of the OpenVSwitches.
| |
| The apkovl on the USB keys contains only the scripts to launch KVM and one image file to launch the first SAN. Other KVM are launched from LV in the SAN.
| |
| | |
| === dhcp ===
| |
| | |
| Exactly two KVM stored in different SAN, ''primary'' and ''secondary'' in <u>failover mode</u>, are running '''dhcp'''d from repo. <br/>
| |
| We just have to configure it properly.
| |
| | |
| We have to test if '''dhcp'''d may run in a LXC instead of a KVM?
| |
| | |
| === DNS ===
| |
| | |
| Nothing to say here because still running on debian.
| |
| | |
| === Resolver ===
| |
| | |
| With '''dnscache''' from repo.
| |
| | |
| Those KVM have <u>manually assigned IP address in the LAN</u> and does know a gateway to the Internet.<br/>
| |
| They use themselves as resolver... <br/>
| |
| They know the direct manually assigned IP address in the LAN of the main DNS server of selected domains (for split dns configuration).
| |
| | |
| === PXEboot ===
| |
| | |
| Need to try ;) It is '''<u>THE NEXT STEP</u>'''!<br/>
| |
| Must prepare needed files in '''tftp''' server.<br/>
| |
| Must prepare needed files in '''nfs''' server.<br/>
| |
| Must prepare needed files in '''darkhttpd''' server.<br/>
| |
| Must start both '''dhcpd''' server (''primary'' and '' secondary'') with prepared config in a row.
| |
| | |
| === Time server ===
| |
| | |
| The router (who has access to internet) usr '''ntpd''' (or similar) from repo, to act as <u>client to the WAN</u> and <u>server to the LAN</u>.
| |
| | |
| === syslog ===
| |
| | |
| With '''syslog-ng''' from repo, we receive the logs from all machines be it physical or virtual.<br/>
| |
| It's the only place who needs '''logrotate''' from repo.
| |
| | |
| === HTTP proxy/cache ===
| |
| | |
| The web proxy/cache '''squid''', from repo, uses a NBD as cache.
| |
| It has a link to the internet to forward requests and one to the LAN.
| |
| | |
| Because of him, no machine, as they are all connected to the LAN, be it physical or virtual, needs a published default gateway.
| |
| And all machines are able to install/upgrade packages or to see the WWW as client.
| |
| | |
| We point all AL boxes to this KVM with '''setup-proxy'''.
| |
| | |
| === Monitoring ===
| |
| | |
| shinken from sources in some LXC with barely only the python package installed
| |
|
| |
|
| === Metrology ===
| | {{pkg|gvpe}}<br> |
| | http://software.schmorp.de/pkg/gvpe.html |
|
| |
|
| '''Collectd''' (one KVM as server, all other machines, be it physical or virtual, as client) with collectd-network from repo.<br/>
| | Plan to use it to interconnect about 5 sites. |
| A couple of lines in CGP config file is enough for now.
| |
|
| |
|
| === Backups === | | == About freeswitch == |
|
| |
|
| with common tools: '''rsync''', '''tar''', '''nc''', '''bzip2''', '''openssh''', '''cron'''
| | I have a request to run a SIP server for a couple of users.<br/> |
| | I'm doing it in some LXC accessed trough an openVPN from Jolla phones. |
|
| |
|
| == High-level services == | | == New rollout of our infra == |
|
| |
|
| in LXC AL whenever possible.<br/>
| | This week, we will upgrade some hardware and also redo all the infrastructure based on the fresh 3.3 serie. |
| in LXC Debian as second choice<br/>
| |
| in KVM otherwise.
| |
|
| |
|
| ''edge'' is currently broken and is unfortunately needed for several services :(
| | The compute nodes will run (on baremetal) with mdadm, openvswitch, qemu, consul, collectd, screen (maybe tmux) and openssh. |
| Albeit Alpine is fixed ASAP albeit we will need to switch...
| |
|
| |
|
| === x2goserver ===
| | The storage nodes will run a CEPH cluster (unfortunately not based on AL). |
|
| |
|
| AL edge proposes the package '''x2goserver'''.<br/>
| | Everything else will run in various KVM on the compute nodes. |
| I would like to give it a try ;)<br/>
| |
| It seems to be running (at least installed) in a AL LXC inside a AL KVM and connected trough OVS ^^
| |
|
| |
|
| but unfortunately, '''x2goclient''' pops up "kex error : did not find one of algos diffie-hellman-group1-sha1 in list curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 for kex algos"
| | First, let's check if the needed package are available in the basic ISOs. If yes we will be able to run from USB keys. If not we will need to have sys install on the HDD... |
How to emulate USB stick with KVM.
How to set up a PXE environement.
From first repo (boot media):
AlpineLinux dhcpd tftp-hpa syslinux mkinitfs nfs-utils darkhttpd rsync openssh openvswitch screen qemu-system-X86_64 qemu-img gptfdisk parted mdadm lvm2 nbd xfsprogs e2fsprogs multipath consul dnsmasq vim collectd collectd-network git syslog-ng envconsul consul-template xnbd ceph lxc lxc-templates xfsprogs gptfdisk e2fsprogs multipath wipe tcpdump curl openvpn fsconsul
and all dependecies...
will build a custom ISO with that list...
About NFS
NFS is now working with AL. Both as server and client with the nfs-utils package.
However, to use NFS as client in some LXC does not seems to work yet as shown below
nfstest:~# mount -t nfs -o ro 192.168.1.149:/srv/boot/alpine /mnt
mount.nfs: Operation not permitted
mount: permission denied (are you root?)
nfstest:~# tail /var/log/messages
Apr 4 10:05:59 nfstest daemon.notice rpc.statd[431]: Version 1.3.1 starting
Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Flags: TI-RPC
Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to read /var/lib/nfs/state: Address in use
Apr 4 10:05:59 nfstest daemon.notice rpc.statd[431]: Initializing NSM state
Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to write NSM state number: Operation not permitted
Apr 4 10:05:59 nfstest daemon.warn rpc.statd[431]: Running as root. chown /var/lib/nfs to choose different user
nfstest:~# ls -l /var/lib/nfs
total 12
-rw-r--r-- 1 root root 0 Nov 10 15:43 etab
-rw-r--r-- 1 root root 0 Nov 10 15:43 rmtab
drwx------ 2 nobody root 4096 Apr 4 10:05 sm
drwx------ 2 nobody root 4096 Apr 4 10:05 sm.bak
-rw-r--r-- 1 root root 4 Apr 4 10:05 state
-rw-r--r-- 1 root root 0 Nov 10 15:43 xtab
msg from ncopa """
dmesg should tell you that grsecurity tries to prevent you to do this.
grsecurity does not permit the syscall mount from within a chroot since
that is a way to break out of a chroot. This affects lxc containers too.
I would recommend that you do the mouting from the lxc host in the
container config with lxc.mount.entry or similar.
https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html#lbAR
If you still want disable mount protection in grsecurity then you
can do that with:
echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount
"""
this is not working with
lxc.mount.entry=nfsserver:/srv/boot/alpine mnt nfs nosuid,intr 0 0
on the host machine with all nfs modules and helper software installed and loaded.
backend:~# lxc-start -n nfstest
lxc-start: conf.c: mount_entry: 2049 Invalid argument - failed to mount
'nfsserver:/srv/boot/alpine' on '/usr/lib/lxc/rootfs/mnt'
lxc-start: conf.c: lxc_setup: 4163 failed to setup the mount entries for
'nfstest'
lxc-start: start.c: do_start: 688 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1080 failed to spawn 'nfstest'
Nor with
echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount
on the host machine with all nfs modules and helper software installed and loaded which does'nt work either.
To find a proper way to use NFS shares from AL LXC is an important topic in order to be able to, for instance, load balance web servers sharing contents uploaded by users.
Next step will be to have HA for the NFS server itself (with only AL machines).
About NBD
NBD is now in edge/testing thanks to clandmeter.
we now use xnbd ^^
Also we are still looking after the right solution to backup NBD as a whole (versus by it's content) while in use. dd|nc is the used way nowadays.
About consul
nothing yet but big hopes ^^
I'm lurking IRC about it ;)
We plan to use it's dynamic DNS feature, it's hosts listing, services inventory, events, k/v store...
and even semi high-availability for our PXE infrastructure the consul leader being the active PXEserver and other consul server are dormant PXEservers.
All config scripts adapted to pull values out of consul k/v datastore based on profiles found out of consul various lists.
As the key for dhcpd and PXEboot is the hwaddr, it will become our uuid for LAN and consul too.
We are very exited by consul capacities!
Will be avid tester!
Open questions:
- What memory footprint is needed?
- What about dynamycally adapt quorum size?
- Are checks possible triggers?
- What best practice to store etc configurations?
log of experimentation at User_talk:Jch/consul
About CEPH
CEPH is supposed to sovle the problem of high availability for the data stores, be it block devices (disks) or character devices (files).
The actual situation is not satisfactory.
We are very exited by CEPH capacities!
Will be avid tester!
The Alpine kernel has now RBD modules compiled.
We will build a CEPH cluster out of 3 Ubuntu LTS and use AL boxes as client if possible (to launch qemu instances directly from RBD). If not, we then will attach RBD and reexport them with xNBD inside a debian KVM.
About Docker
not a lot of information on the Docker page yet ...
About E-MailRelay
E-MailRelay is a simple SMTP proxy and store-and-forward message transfer agent (MTA).
See http://emailrelay.sourceforge.net/
It compiles fine on AL.
apk update
apk add subversion alpine-sdk
svn checkout svn://svn.code.sf.net/p/emailrelay/code/trunk emailrelay-code
cd emailrelay-code
./configure --prefix=/usr
make
make install
apk del subversion alpine-sdk
apk add libgcc libstdc++
emailrelay --help
But I still have issues to properly build a package because it wants to install some stuff in <PREFIX>/libexec...
(And I also need to separate -doc, -test, -extra and optionnaly -gui in subpackages I guess)
About X2Go
x2goserver
I did prepare x2goserver and nx-libs packages.
x2goclient
lrelease-qt4 x2goclient.pro
/bin/bash: lrelease-qt4: command not found
Makefile:39: recipe for target 'build_client' failed
Dunno where to find that...
My laptop setup
AL 3.3 with +/etc/inittab+
tty5::respawn:/usr/bin/su - jch mcabber
tty6::respawn:/usr/bin/su - jch tmux
tty7::respawn:/usr/bin/su - jch startx
and +~/.xinitrc+
- !/bin/sh
exec chromium-browser --no-sandbox
About gpve
gvpe
http://software.schmorp.de/pkg/gvpe.html
Plan to use it to interconnect about 5 sites.
About freeswitch
I have a request to run a SIP server for a couple of users.
I'm doing it in some LXC accessed trough an openVPN from Jolla phones.
New rollout of our infra
This week, we will upgrade some hardware and also redo all the infrastructure based on the fresh 3.3 serie.
The compute nodes will run (on baremetal) with mdadm, openvswitch, qemu, consul, collectd, screen (maybe tmux) and openssh.
The storage nodes will run a CEPH cluster (unfortunately not based on AL).
Everything else will run in various KVM on the compute nodes.
First, let's check if the needed package are available in the basic ISOs. If yes we will be able to run from USB keys. If not we will need to have sys install on the HDD...