User talk:Jch: Difference between revisions

From Alpine Linux
mNo edit summary
m (→‎About gpve: Rescued dead link by replacing it with pkg template.)
 
(109 intermediate revisions by one other user not shown)
Line 1: Line 1:
==How to automate KVM creation==
== [[User_talk:Jch/How to automate KVM creation|How to automate KVM creation]] ==
How to emulate USB stick with KVM.


The goal is not only to have a working install but to have it at the after setup-alpine stage without human intervention...
== [[User_talk:Jch/Starting_AL_from_network|Starting_AL_from_network]] ==
Tis is the first stages of a work in progress...
How to set up a PXE environement.


I want to pass a Block Device and a name as parameters. The block device could be an image file, a LV, a NBD, a hdd, a raid array, whatever.<br/>
== [[User_talk:Jch/Building_a_complete_infrastucture_with_AL|Building_a_complete_infrastucture_with_AL]] ==
Everything else should be fully automatic according to some config file (stating the http-proxy, the time server, the log server, ...).


The I will just run the script, watch my dhcp logs to discover the new IP assigned (that's why the name is a parameter), then log in with ssh without password to customize it further but at high level only (will be a robot and not me in fact).
<u>From first repo</u> (boot media):


I guess it would be something like emulate boot from usb key with specific overlay already on key... <br/>
AlpineLinux dhcpd tftp-hpa syslinux mkinitfs nfs-utils darkhttpd rsync openssh openvswitch screen qemu-system-X86_64 qemu-img gptfdisk parted mdadm lvm2 nbd xfsprogs e2fsprogs multipath '''consul''' dnsmasq vim collectd collectd-network git syslog-ng <s>envconsul</s> <s>consul-template</s> <s>xnbd</s> <s>ceph</s> lxc lxc-templates xfsprogs gptfdisk e2fsprogs multipath wipe tcpdump curl openvpn <s>fsconsul</s>
then run setup-disk with proper parameters on the command line to avoid the interactive process (like setup-alpine does)... <br/>
Methink this could be done from a couple of scripts put in /etc/local.d/. The last.stop one deleting all of them to be clean at next reboot.<br/>
Let's start easy ;)


=== How to prepare a img file to emulate an USB key ===
and all dependecies...


first a working example done in console (accessed trough ssh).<br/>
will [[How_to_make_a_custom_ISO_image|build a custom ISO]] with that list...
Will build a script from it...


First, lets's prepare somme block device (here an image file but could be something else) <pre>
== About NFS ==
apk add qemu-img
qemu-img create -f raw usbkey.img 512M
apk del qemu-img
T="usbkey.img"
</pre>


Next, let's install AL on this $T <pre>
NFS is now working with AL. Both as server and client with the nfs-utils package.<br/>
apk add multipath-tools syslinux dosfstools
However, to use NFS as client in some LXC does not seems to work yet as shown below
fdisk $T
<pre>
kpartx -av $T
nfstest:~# mount -t nfs -o ro 192.168.1.149:/srv/boot/alpine /mnt
mkdosfs -F32 /dev/mapper/loop1p1
mount.nfs: Operation not permitted
dd if=/usr/share/syslinux/mbr.bin of=/dev/mapper/loop1
mount: permission denied (are you root?)
syslinux /dev/mapper/loop1p1
nfstest:~# tail /var/log/messages
mkdir key
Apr  4 10:05:59 nfstest daemon.notice rpc.statd[431]: Version 1.3.1 starting
mount -t vfat /dev/mapper/loop1p1 key
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Flags: TI-RPC
wget http://wiki.alpinelinux.org/cgi-bin/dl.cgi/v3.1/releases/x86_64/alpine-mini-3.1.1-x86_64.iso
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to read /var/lib/nfs/state: Address in use
mkdir cdrom
Apr  4 10:05:59 nfstest daemon.notice rpc.statd[431]: Initializing NSM state
mount alpine-mini-3.1.1-x86_64.iso cdrom
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to write NSM state number: Operation not permitted
cd cdrom
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Running as root. chown /var/lib/nfs to choose different user
cp -a .alpine-release * ../key/
nfstest:~# ls -l /var/lib/nfs
cd ..
total 12
umount key
-rw-r--r--    1 root    root            0 Nov 10 15:43 etab
umount cdrom
-rw-r--r--    1 root    root            0 Nov 10 15:43 rmtab
kpartx -d $T
drwx------    2 nobody  root          4096 Apr  4 10:05 sm
apk del multipath-tools syslinux dosfstools
drwx------   2 nobody  root          4096 Apr  4 10:05 sm.bak
rm alpine-mini-3.1.1-x86_64.iso
-rw-r--r--    1 root    root            4 Apr  4 10:05 state
-rw-r--r--    1 root    root            0 Nov 10 15:43 xtab
</pre>
</pre>


This block device may now be use to boot some KVM for instance like: <pre>
msg from ncopa """
screen -d -m -S KVM-builder \
dmesg should tell you that grsecurity tries to prevent you to do this.
qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
-device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick
</pre> This is working fine.
 
The problem is when adding a HDD to the lot, qemu try to boot from the hdd and does not even try to boot from the usb key. Enabling menu in boot let's one access the emulated bios which allows to select USB device to boot interactively but this break the goal of fully automated boot :( The stanza is for instance <pre>
screen -d -m -S KVM-builder \
qemu-system-x86_64 -name KVM-usb -enable-kvm -cpu qemu64 -curses \
-device nec-usb-xhci -drive if=none,id=usbstick,file=$T -device usb-storage,drive=usbstick \
-drive file=$T2 boot menu=on
</pre>
 
qemu-doc states that very clearly:<br/>
> -boot [order=drives][,once=drives][,menu=on|off][,splash=sp_name][,splash-time=sp_time][,reboot-timeout=rb_timeout][,strict=on|off]<br/>
>  Specify boot order drives as a string of drive letters. Valid drive letters depend on the target achitecture. The x86 PC uses: a, b (floppy 1 and 2), c (first hard disk), d (first CD-ROM), n-p (Etherboot from network adapter 1-4), hard disk boot is the default
 
==Starting AL from network==
 
As it does not seems possible to start qemu with a virtual USB key *and* a virtual HDD attached to the VM. Let's try something different: to start AL from the network and mount the HDD later on...


Usually this kind of setup needs
grsecurity does not permit the syscall mount from within a chroot since
* a DHCP server to get an IP address and the location of the TFTP server
that is a way to break out of a chroot. This affects lxc containers too.
* a TFTP server to download the kernel and tje root file system to boot from
* a NFS server or a HTTP one to get the overlay used to configure the machine
* a NFS server to share files with others
* a NBD server to get his own block devices as storage
* a machine where to prepare initramfs


First, let's check what is vailable in AL and what is not...
I would recommend that you do the mouting from the lxc host in the
* dhcpcd-6.6.7-r0
container config with lxc.mount.entry or similar.
* tftp-hpa-5.2-r1
* nfs-utils-1.3.1-r2
* darkhttpd-1.10-r1
* nbd-3-10-r0


=== PXE_boot ===
https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html#lbAR


We are trying to do something as in [[PXE_boot]].
If you still want disable mount protection in grsecurity then you
can do that with:
echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount
"""


We did it on separate machine for each service. It forces us to deeply understand all interactions between processes.
this is not working with


==== dhcpd ====
<pre>lxc.mount.entry=nfsserver:/srv/boot/alpine mnt nfs nosuid,intr 0 0</pre>


192.168.1.1
on the host machine with all nfs modules and helper software installed and loaded.
 
with package dhcp from repo. <s>Nothing special.</s>


<pre>
<pre>
  filename "pxelinux.0";
backend:~# lxc-start -n nfstest
  next-server 192.168.1.2;
lxc-start: conf.c: mount_entry: 2049 Invalid argument - failed to mount
'nfsserver:/srv/boot/alpine' on '/usr/lib/lxc/rootfs/mnt'
lxc-start: conf.c: lxc_setup: 4163 failed to setup the mount entries for
'nfstest'
lxc-start: start.c: do_start: 688 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1080 failed to spawn 'nfstest'
</pre>
</pre>


and
Nor with
<pre>
# Disable RFC 2136 dynamic DNS updates.
ddns-update-style none;
 
# Define actions to take when leases are committed, released, or expired to
# accomplish dynamic DNS updates to djbdns. This does not use the RFC 2136
# update mechanism, because djbdns does not support it. However, it
# accomplishes the same thing.
# syntax "execute(cmd, arg, ...)"
### need to check if the two "on EVENT" must be nested or in sequence...
on commit {
  execute ("/usr/local/bin/dns-update-djb",
          "commit",
          lcase (option host-name),
          config-option domain-name,
          binary-to-ascii (10, 8, ".", leased-address));
  on release or expiry {
    execute ("/usr/local/bin/dns-update-djb",
            "release",
            binary-to-ascii (10, 8, ".", leased-address));
  }
}
</pre>
 
with a custom /usr/local/bin/dns-update-djb script largely inspired from https://sites.google.com/site/dmoulding/dns-update-djb but adapted for a distant tinydns server and to the AL way.
 
==== tftp ====
 
192.168.1.2
 
tftp-hpa configured to serve some SYSLINUX files.


The config is in /etc/conf.d/in.tftpd<br/>
Then to issue:
<pre>
<pre>
rc-update add in.tftpd
echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount
rc-service in.tftpd start
</pre>
 
We serve from /var/tftpboot.
 
We add to temporary install the syslinux apk to get pxelinix.0 and other libs needed. <br/>
We did prepare a "pxerd" initramfs file with virtio_net.ko, dhcp and nfs included; made sure loop and squashfs are included. <br/>
pxelinux.cfg/default looks like <pre>
PROMPT 0
TIMEOUT 3
default alpine
LABEL alpine
LINUX alpine/vmlinuz-grsec
INITRD alpine/pxerd
APPEND ip=dhcp alpine_dev=nfs:192.168.1.3:/srv/boot/alpine modloop=/boot/grsec.modloop.squashfs nomodeset quiet apkovl=http://192.168.1.4/localhost.apkovl.tar.gz
#APPEND modloop=http:/192.168.1.4/grsec.modloop.squashfs
#APPEND apkovl=http://192.168.1.4/localhost.apkovl.tar.gz # including the modloop hack
#APPEND alpine_repo=http://repo-url
</pre>
 
Modules are loaded <pre>
/ # lsmod
Module                  Size  Used by    Not tainted
nfsv3                  22784  1
nfs                  144376  2 nfsv3
lockd                  71917  2 nfsv3,nfs
sunrpc                225574  6 nfsv3,nfs,lockd
af_packet              28735  0
sr_mod                13487  0
cdrom                  40424  1 sr_mod
pata_acpi              3326  0
ata_piix              25601  0
ata_generic            3554  0
libata                181955  3 pata_acpi,ata_piix,ata_generic
virtio_net            19684  0
scsi_mod              113710  2 sr_mod,libata
virtio_pci              6485  0
virtio                  4933  2 virtio_net,virtio_pci
virtio_ring            9161  2 virtio_net,virtio_pci
squashfs              25893  1
loop                  18243  2
</pre> Network is up <pre>
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:33:B0:C2:D2
inet addr:192.168.1.108  Bcast:0.0.0.0  Mask:255.255.255.0 
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1 
RX packets:322 errors:0 dropped:0 overruns:0 frame:0 
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 
collisions:0 txqueuelen:1000
RX bytes:20514 (20.0 KiB)  TX bytes:684 (684.0 B)
</pre> but modloop does not load
 
This patch fix this issue (hope to see it mainstream soon) <pre>
localhost:~# diff /etc/init.d/modloop modloop.new
--- /etc/init.d/modloop
+++ modloop.new
@@ -32,7 +32,7 @@
        local search_dev="$1" fstab="$2"
        local dev mnt fs mntopts chk
        case "$search_dev" in
-              UUID=*|LABEL=*|/dev/*);;
+              UUID=*|LABEL=*|/dev/*|nfs);;
                *) search_dev=/dev/$search_dev;;
        esac
        local search_real_dev=$(resolve_dev $search_dev)
@@ -49,6 +49,10 @@
                                fi
                        done
                done
+              if [ "$fs" = "$search_dev" ]; then
+                      echo "$mnt"
+                      return
+              fi
        done < $fstab 2>/dev/null
}
 
</pre>
 
==== References ====
 
http://www.syslinux.org/wiki/index.php/PXELINUX
 
==== nfs ====
 
192.168.1.3
 
see http://wiki.alpinelinux.org/wiki/User_talk:Jch#NFS_bug_study <br/>
'''It is now working with''' http://dev.alpinelinux.org/~clandmeter/rpcbind-0.2.3_rc2-r0.apk
 
We serve the content of an usb key (iso) in ro as <pre>
/srv/boot/alpine *(ro,no_root_squash,no_subtree_check)
</pre>
</pre>


==== http ====
on the host machine with all nfs modules and helper software installed and loaded which does'nt work either.
 
192.168.1.4
 
With package [[Darkhttpd]] from repo serving from /var/tftpboot/ to serve files needed to boot (kernel, rootfs, apkovl.tar.gz)
 
==== nbd ====
 
192.168.1.5
 
I really would like to have xnbd-server in AL. nbd-3.1.0 was just added to edge/testing repo; need to try it in real situation...<br/>
For now, we have a qcow2 debian image  added to the apkovl with lbu add; lbu ci.<br/>
This image is used to launch a first KVM with /dev/mdX as second drive.<br/>
In turn, inside the KVM, vdb is used to define a lvm2 volume.<br/>
The LV are published with xnbd-server.
 
Later on, the same KVM will be able to connect to RBD device and re-publish it as NBD.
 
'''xnbd-server''' allows ''live migration'' of Block Devices while live. And has a powerfull ''proxy'' mode.
 
All other KVM are running from FS accessed trough NBD from such SAN. Even other SAN.<br/>
As soon as those '''KVM-NBD''' are up, they may be used to <u>launch others</u> or to provide ''datastores''.
 
We put that image on every USB key we use along with mdadm and OpenVSwitch (and collectd).
 
==== dns ====
 
192.168.1.6
 
tinydns from repo with split-dns config.
 
== Building a complete infrastucture with AL ==
 
I'm doing it. It's for real! That's my daily job at present ^^
 
I'm building a full private cloud bootstraped with only an AlpineLinux USB key for each physical machine. But next ones will be able to boot from network; not even USB keys will be needed. As a matter of fact, we used more than only one physical USB key because we didn't started from scratch but had a live migration from Debian to Alpine for most of the services and machines...
 
If there is some feed-back, I may develop config files and so on ;)
 
As I started from scratch and OpenVSwitch was not available in Alpine at that time yet, It took me a while to build everything. But to reproduce it, it would be ''piece of cake''!
 
We use qemu-kvm for KVM. But I guess one may use whatever Virtual Machine technology one likes.
 
'''This is the presentation of a use case. Not a HOW TO. And it's still a work in progess...'''
 
=== Network ===
 
==== Firewall ====
 
We put a dedicated physical machine on each link between our LAN and other networks.
It just run iptables and some paquets accounting metrology.
 
==== Router ====
 
Physical machine connected to our LAN and other networks (trough a firewall). A static routing table do the trick.
 
==== Switches ====
 
All physical machines run OpenVSwitch reproducing virtually all physical switches we have plus some virtuals only.
 
==== VPN ====
 
All physical machines run openVPN as client to as many switch defined less the physical interfaces of the machine. There is an openVPN server somewhere running in a KVM connected to needed switches.
 
=== Storage ===
 
==== SAN ====
 
On each physical machine, a couple of HDD are mounted in raid1 witch mdadm. This raid array is passed as parameter to a KVM who in turn mount it as physical volume for LVM. The created LV are published as NBD with '''xnbd-server'''. For the time being, this KVM is running debian 7.8 as xnbd is not in Alpine (yet?)..
 
The SAN also connects to the CEPH cluster as client and publish reached RBD as NBD with xnbd-server. For the time being, this KVM is running debian 7.8 as no xnbd nor RBD are in Alpine (yet?)..
 
==== NAS ====
 
Some KVM is mounting some NBD as local drives and publishing some directories as NFS shares.<br/>
We now have nfs and nbd in AL.
 
==== CEPH ====
 
KVM with physical HDD as parameters are used for building OSD and MON needed to operate a CEPH cluster.
One KVM is the "console" to drive it from a single point of presence (usefull but not "needed").For the time being, those KVM are running debian 7.8 as CEPH and RBD are not in Alpine (yet?)..
 
=== Low-level services ===
 
No service at all is running in the AL on bare metal. All are running is some KVM connected to needed switches by the means of the OpenVSwitches.
The apkovl on the USB keys contains only the scripts to launch KVM and one image file to launch the first SAN. Other KVM are launched from LV in the SAN.
 
==== dhcp ====
 
Exactly two KVM stored in different SAN, ''primary'' and ''secondary'' in <u>failover mode</u>, are running '''dhcp'''d from repo. <br/>
We just have to configure it properly.
 
We have to test if '''dhcp'''d may run in a LXC instead of a KVM?
 
==== DNS ====
 
tinydns from repo with split-dns config.
 
==== Resolver ====
 
With '''dnscache''' from repo.
 
Those KVM have <u>manually assigned IP address in the LAN</u> and does know a gateway to the Internet.<br/>
They use themselves as resolver... <br/>
They know the direct manually assigned IP address in the LAN of the main DNS server of selected domains (for split dns configuration).
 
==== PXEboot ====
 
kernel and initrd files in '''tftp''' server.<br/>
copy of usb content in '''nfs''' server.<br/>
apkovl files in '''darkhttpd''' server.
 
==== Time server ====
 
The router (who has access to internet) usr '''ntpd''' (or similar) from repo, to act as <u>client to the WAN</u> and <u>server to the LAN</u>.
 
==== syslog ====


With '''syslog-ng''' from repo, we receive the logs from all machines be it physical or virtual.<br/>
To find a proper way to use NFS shares from AL LXC is an important topic in order to be able to, for instance, load balance web servers sharing contents uploaded by users.
It's the only place who needs '''logrotate''' from repo.


==== HTTP proxy/cache ====
Next step will be to have HA for the NFS server itself (with only AL machines).


The web proxy/cache '''squid''', from repo, uses a NBD as cache.
== About NBD ==
It has a link to the internet to forward requests and one to the LAN.


Because of him, no machine, as they are all connected to the LAN, be it physical or virtual, needs a published default gateway.
NBD is now in edge/testing thanks to clandmeter.
And all machines are able to install/upgrade packages or to see the WWW as client.


We point all AL boxes to this KVM with '''setup-proxy'''.
we now use xnbd ^^


==== Monitoring ====
Also we are still looking after the right solution to backup NBD as a whole (versus by it's content) while in use. dd|nc is the used way nowadays.


'''shinken''' from sources in some LXC with barely only the python package installed
== About consul ==


==== Metrology ====
nothing yet but big hopes ^^<br/>
I'm lurking IRC about it ;)


'''Collectd''' (one LXC as server, all other machines, be it physical or virtual, as client) with collectd-network from repo.<br/>
We plan to use it's dynamic DNS feature, it's hosts listing, services inventory, events, k/v store... <br/>
A couple of lines in CGP config file is enough for now.
and even semi high-availability for our PXE infrastructure the consul leader being the active PXEserver and other consul server are dormant PXEservers.<br/>
All config scripts adapted to pull values out of consul k/v datastore based on profiles found out of consul various lists.<br/>
As the key for dhcpd and PXEboot is the hwaddr, it will become our uuid for LAN and consul too.<br/>
'''We are very exited by consul capacities!'''<br/>
Will be avid tester!


==== Backups ====
'''Open questions''':


with common tools: '''rsync''', '''tar''', '''nc''', '''bzip2''', '''openssh''', '''cron'''
# What memory footprint is needed?
# What about dynamycally adapt quorum size?
# Are checks possible triggers?
#* <pre>consul watch -prefix type -name name /path/to/executable</pre>
#* <pre>consul event [options] -name name [payload]</pre>
# What best practice to store etc configurations?
#* http://code.hootsuite.com/distributed-configuration-management-and-dark-launching-using-consul/
#* http://agiletesting.blogspot.fr/2014/11/service-discovery-with-consul-and.html
#* envconsul
#* consul-template


==== LDAP ====
log of experimentation at [[User_talk:Jch/consul]]


'''openldap''' with openldap-back-hdb, both from repo.
== About CEPH ==


http://www.openldap.org/doc/admin24/backends.html states<br/>
CEPH is supposed to sovle the problem of high availability for the data stores, be it block devices (disks) or character devices (files).
> The hdb backend to slapd(8) is the recommended primary backend for a normal slapd database.<br/>
And<br/>
> Note: The hdb backend has superseded the bdb backend, and both will soon be deprecated in favor of the new mdb backend.


> The mdb backend to slapd(8) is the upcoming primary backend for a normal slapd database. It uses OpenLDAP's own Lightning Memory-Mapped Database (LMDB) library to store data and is intended to replace the Berkeley DB backends.
The actual situation is not satisfactory.


Unfortunately there is no ''openldap-back-mdb'' package in AL yet.
'''We are very exited by CEPH capacities!'''<br/>
Will be avid tester!


=== High-level services ===
The Alpine kernel has now RBD modules compiled.


in LXC AL whenever possible.<br/>
We will build a CEPH cluster out of 3 Ubuntu LTS and use AL boxes as client if possible (to launch qemu instances directly from RBD). If not, we then will attach RBD and reexport them with xNBD inside a debian KVM.
in LXC Debian as second choice<br/>
in KVM otherwise.


==== x2goserver ====
== About Docker ==


I did package nx-libs and x2goserver. I'm waiting for the packages to be included in edge/testing. They are already being used for single app access. Next step is full desktop but we are not sure if AL is the right choice for full desktop usage for our customers...
not a lot of information on the [[Docker]] page yet ...


unfortunately, '''x2goclient''' pops up "kex error : did not find one of algos diffie-hellman-group1-sha1 in list curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 for kex algos"
== About E-MailRelay ==
need to specify diffie-hellman-group1-sha1 in sshd_config. Luckyly a fix exists and my business partner is looking after a way to enhance it's security upstream.


==== ejabberd ====
E-MailRelay is a simple SMTP proxy and store-and-forward message transfer agent (MTA). <br/>
 
See http://emailrelay.sourceforge.net/
with ejabberd from edge/testing repo.
I migrate the mnesia DB from an old debian squeeze just copying the files and changing ownership in a LXC-AL. I just had to disable mod_pubsub to have it run properly. Authentification is done with openLDAP.
I now plan to migrate a very very old jabberd (11 years I guess) running on a debian etch to it if I find a way to keep users's password and rosters...
I also would like to use it as a gateway to IRC to follow #alpine, #alpine-devel and #x2go channels ;)
Some other ejabberd features are interesting to my organisation and we will experiment more in depth, namely mod_sip, mod_stun, mod_proxy65...
 
==== redmine ====
 
in a brand new LXC with edge/main and edge/testing repos<br/>
mostly following [[Redmine]] page<br/>
I use a mariaDB server on another host where I did create the user and pushed the sqldump from a running redmine 3.0.0 instance


It compiles fine on AL.
<pre>
<pre>
apk update
apk update
apk upgrade
apk add subversion alpine-sdk
reboot
svn checkout svn://svn.code.sf.net/p/emailrelay/code/trunk emailrelay-code
setup-timezone
cd emailrelay-code
apk add redmine
./configure --prefix=/usr
apk add ruby-unicorn
make
cp /etc/unicorn/redmine.conf.rb.sample /etc/unicorn/redmine.conf.rb
make install
vi /etc/conf.d/unicorn
apk del subversion alpine-sdk
vi /etc/redmine/database.yml
apk add libgcc libstdc++
apk add sudo
emailrelay --help
apk add ruby-mysql2
apk add ruby-yard
apk add tzdata
cd /usr/share/webapps/redmine
sudo -u redmine rake generate_secret_token
sudo -u redmine RAILS_ENV=production rake db:migrate
</pre>
</pre>


== Master key ==
But I still have issues to properly build a package because it wants to install some stuff in <PREFIX>/libexec...<br/>
(And I also need to separate -doc, -test, -extra and optionnaly -gui in subpackages I guess)


We want to be able to bootstrap the full infrastructure from only one usb key and one machine with physical access (to insert the usb key obviously).
== About X2Go ==


This key will run AL stable.
=== x2goserver ===
With only very few packages installed.
But some images on the sorage.


=== Initial packages ===
I did prepare x2goserver and nx-libs packages.


dhcpd tftp syslinux nfs darkhttp openssh vim
=== x2goclient ===
openvswitch mdadm qemu screen collectd collectd-network gptdisk irqbalance ssmtp mailx


=== Bootstrap ===
<pre>
lrelease-qt4 x2goclient.pro
/bin/bash: lrelease-qt4: command not found
Makefile:39: recipe for target 'build_client' failed
</pre> Dunno where to find that...


First, we setup the network. Remember, this is a bootstrap. We assume nothing exect to have (or not) an internet connexion on some ethernet cable plugged into some NIC.
== My laptop setup ==
It means we may take any decision we see fit.


Our primary machine is the only fixed point for now. Let's give it the number 1.<br/>
AL 3.3 with +/etc/inittab+ <pre>
All machines are connected to the LAN. We know nothing yet about other NICs.<br/>
tty5::respawn:/usr/bin/su - jch mcabber
First we must decide about the LAN IP range. For instance be it 192.168.1.0/24.<br/>
tty6::respawn:/usr/bin/su - jch tmux
We will use complicated network setup, let's start by installing openvswitch on bare metal
tty7::respawn:/usr/bin/su - jch startx
apk add openvswitch
</pre> and +~/.xinitrc+ <pre>
rc-update add ovs-modules
#!/bin/sh
rc-update add ovsdb-server
exec chromium-browser --no-sandbox
rc-update add ovs-vswitch
</pre>
rc-service ovs-modules start
rc-service ovsdb-server start
rc-service ovs-vswitch start
ovs-vsctl add-br lan
ovs-vsctl add-port lan eth0
vi /etc/network/interfaces #iface eth0 inet manual
                            #iface lan inet dhcp
 
No machine will offer any service from bare metal.
apk add qemu-system-x86_64 screen libusb
modprobe kvm
modprobe kvm-intel
modprobe kvm-amd
modprobe tun
screen -m -d -S KVM-infra qemu-system-x86_64 -kvm -kernel /kernel -initrd /initrd -append alpine_dev=...,apkovl=... -net -net -drive /dev/usb
 
La suite immédiate se fait dans cette VM
screen -r KVM-infra
 
We need the storage space from the usb key to handle boot images and apkovl files.<br/>
In KVM-infra
setup-alpine (fixed IP)
apk add openvswitch
rc-update add ovs-modules
rc-update add ovsdb-server
rc-update add ovs-vswitch
rc-service ovs-modules start
rc-service ovsdb-server start
rc-service ovs-vswitch start
ovs-vsctl add-br lan
ovs-vsctl add-port lan eth0
vi /etc/network/interfaces #iface eth0 inet manual
                            #iface lan inet static
mkdir -p /srv
mount /vda2 /srv
mkdir -p /srv/nfs/alpine
mount /vda1 /srv/nfs/alpine
 
Next we will have other networked devices. We need dhcpd (in KVM-infra)
apk add dhcpd
rc-update dhcpd
vi /etc/dhcp/dhcpd.conf  #filename "pxelinux.0";
                          #next-server 192.168.1.1;
rc-service dhcpd start
 
To boot others, we need tftp, nfs and http (tftp and http in LXC, nfs in KVM-infra)
apk add tftp-hla nfs-utils darkhttp
rc-update add nfs
rc-update add tftp
rc-update add darkhttpd
mkdir -p /srv/nfs
mkdir -p /srv/tftp
mkdir -p /srv/http
vi /etc/exports
vi /etc/tftp
vi /etc/darkhttp
rc-service tftp start
rc-service darkhttpd start
rc-service nfs start
 
Let's populate those servers.
mkdir -p /srv/tftp/alpine
cp /media/usb/boot/vmlinuz* /srv/tftp/alpine/
cp /media/usb/boot/modloop* /srv/tftp/alpine/
apk add mkinitfs
cd /etc/mkinitfs
vi features.d/network.modules
vi features.d/dhcp.files
vi features.d/dhcp.modules
vi features.d/nfs.modules
vi mkinitfs.conf # add network, dhcp, nfs and squashfs
mkinitfs -o /srv/tftp/alpine/pxerd
apk del mkinitfs
apk add syslinux
cp /usr/share/syslinux/pxelinux.0 /srv/tftp/
cp /usr/share/syslinux/ldlinux.c32 /srv/tftp/
apk del syslinux
mkdir -p /srv/tftp/pxelinux.cfg
vi /srv/tftp/pxelinux.cfg/default
 
All we need now to boot another AL machines (be it physical or virtual) are some {MAC}.apkovl.tar.gz files served by darkhttpd. We badly need name resolution at this stage. DNS and resolver are needed. DNS to be updated dynamically by dhcp server with split-dns. Resolver knowing the fixed IP address on the DNS and the default route if known at this stage. Both may run in LXC inside this KVM-infra (like the other previous services). DNS will be djbdns and resolver will be dnscache (both from repo).
 
Then (for now) we need image of a debian install with xnbd-server and lvm2 to build SAN. <br/>
Also, on bare-metal we need mdadm to assemble raid1 arrays. <br/>
A new SAN is therefore, a MAC address (for debian boot as san), some BD as vda (raid1 from mdadm). <br/>
A <u>new server is</u> therefore, a '''MAC address''' (for AL boot), a '''apkovl file''' (MAC named), some '''data NBD''' from some SAN. <br/>
The apkovl will be downloaded at boot time with PXE provided address; before launching openvswitch! The IP address will then change because of the apparent MAC change when OVS becomes active.<br/>
We may use symlinks to MAC named config files to have a more human friendly view.
 
It is to be noted that after bootstrap KVM may move to other physical machines. While some KVM-infra is somehow connected to the LAN, everything stay alive! This precise image will be reproduced in every SAN build.
 
=== Deploy ===
 
After bootstraping, we dispose of a way to boot any AL KVM or bare-bone in about 10 sec.
 
<u>First</u> we deploy KVM-SAN on bare-metal.
 
Next we deploy KVM-AL grouping (or not) some LXC (AL or debian).


<u>Second</u> we deploy low-level services: syslog-ng, fail2ban, openVPN, '''la_console''', http-reverse-proxy (primary and secondary), http-proxy, smtp relay, secondary resolver, secondary dns, ldap (primary and secondary), NAS, mariaDB, backups, collectd, shinken, local AL repo, git
== About gpve ==


<u>Third</u> intermediary services: smtp in, smtp out, antivirus, antispam, smtp store, imap, pop3, http, php, sip, jabber
{{pkg|gvpe}}<br>
http://software.schmorp.de/pkg/gvpe.html


<u>High level</u> services: x2goserver, lamp, mail toaster, webdav, redmine, etc
Plan to use it to interconnect about 5 sites.


For each of those services, we provide a template in the form of a {kvm-template}.apkovl.tar.gz.<br/>
== About freeswitch ==
After customisation, "lbu package" followed by sending the a.tgz to the central repository is all needed.<br/>
We follow a naming convention for MAC:


For bare metal, the 3 first bytes of the MAC is the manufacturer ID.<br/>We symlink that to the baremetal.apkovl.tar.gz.
I have a request to run a SIP server for a couple of users.<br/>
I'm doing it in some LXC accessed trough an openVPN from Jolla phones.


For KVM, we fix the MAC ourself. <br/>The first 2 bytes (AA:BB) are fixed. <br/>The third one (CC) is the level type of the KVM. <br/>The fourth one (DD) is the specific type of the template. <br/>The last 2 ones are incremental unique ID. <br/>So we are able to define pxelinux.cfg/AA:BB:CC:DD symlinks to config files defining use of {kvm-template}.apkovl.tar.gz.
== New rollout of our infra ==


As {kvm-template}.apkovl.tar.gz tend to be small, we can store a lot of those on the initial USB stick.<br/>
This week, we will upgrade some hardware and also redo all the infrastructure based on the fresh 3.3 serie.
Depending on available space on the USB stick, we could offer {lxc-template}s that way from the USB stick to be downloaded from darkhttpd with wget to the right KVM. Or later on from any wanted NAS.


We add a couple of other OVS (WAN, STORAGE) in every machines. Some are connected to NIC. Some are connected to VPN. Netflow will be used in the future to manage the network (naas: network as a service). One of those OVS (WAN) allows connected machines to access the internet trough a default route passing through a physical firewall. STORAGE is used for data replication between SAN and NAS.
The compute nodes will run (on baremetal) with mdadm, openvswitch, qemu, consul, collectd, screen (maybe tmux) and openssh.


We have the list of bare-metal machines.<br/>
The storage nodes will run a CEPH cluster (unfortunately not based on AL).
Those may launch KVM in one command.


We have the list of SAN KVM.<br/>
Everything else will run in various KVM on the compute nodes.
Those may create and publish NBD in two commands.<br/>
Even on ''diskless' machines those are present to offer nbd-proxy in one command.


All those command are grouped as ''one-liner'' scripts in some redundant NAS available from '''la_console'''.
First, let's check if the needed package are available in the basic ISOs. If yes we will be able to run from USB keys. If not we will need to have sys install on the HDD...
 
Waiting for CEPH, we need a strategy for duplicating NBD accros SAN.
 
== About NFS ==
 
NFS is now working with AL. Both as server and client with the nfs-utils package.<br/>
However, to use NFS as client in some LXC does not seems to work yet as shown below
<pre>
nfstest:~# mount -t nfs -o ro 192.168.1.149:/srv/boot/alpine /mnt
mount.nfs: Operation not permitted
mount: permission denied (are you root?)
nfstest:~# tail /var/log/messages
Apr  4 10:05:59 nfstest daemon.notice rpc.statd[431]: Version 1.3.1 starting
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Flags: TI-RPC
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to read /var/lib/nfs/state: Address in use
Apr  4 10:05:59 nfstest daemon.notice rpc.statd[431]: Initializing NSM state
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to write NSM state number: Operation not permitted
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Running as root.  chown /var/lib/nfs to choose different user
nfstest:~# ls -l /var/lib/nfs
total 12
-rw-r--r--    1 root    root            0 Nov 10 15:43 etab
-rw-r--r--    1 root    root            0 Nov 10 15:43 rmtab
drwx------    2 nobody  root          4096 Apr  4 10:05 sm
drwx------    2 nobody  root          4096 Apr  4 10:05 sm.bak
-rw-r--r--    1 root    root            4 Apr  4 10:05 state
-rw-r--r--    1 root    root            0 Nov 10 15:43 xtab
</pre>

Latest revision as of 01:56, 28 August 2023

How to automate KVM creation

How to emulate USB stick with KVM.

Starting_AL_from_network

How to set up a PXE environement.

Building_a_complete_infrastucture_with_AL

From first repo (boot media):

AlpineLinux dhcpd tftp-hpa syslinux mkinitfs nfs-utils darkhttpd rsync openssh openvswitch screen qemu-system-X86_64 qemu-img gptfdisk parted mdadm lvm2 nbd xfsprogs e2fsprogs multipath consul dnsmasq vim collectd collectd-network git syslog-ng envconsul consul-template xnbd ceph lxc lxc-templates xfsprogs gptfdisk e2fsprogs multipath wipe tcpdump curl openvpn fsconsul

and all dependecies...

will build a custom ISO with that list...

About NFS

NFS is now working with AL. Both as server and client with the nfs-utils package.
However, to use NFS as client in some LXC does not seems to work yet as shown below 

nfstest:~# mount -t nfs -o ro 192.168.1.149:/srv/boot/alpine /mnt
mount.nfs: Operation not permitted
mount: permission denied (are you root?)
nfstest:~# tail /var/log/messages 
Apr  4 10:05:59 nfstest daemon.notice rpc.statd[431]: Version 1.3.1 starting
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Flags: TI-RPC 
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to read /var/lib/nfs/state: Address in use
Apr  4 10:05:59 nfstest daemon.notice rpc.statd[431]: Initializing NSM state
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Failed to write NSM state number: Operation not permitted
Apr  4 10:05:59 nfstest daemon.warn rpc.statd[431]: Running as root.  chown /var/lib/nfs to choose different user
nfstest:~# ls -l /var/lib/nfs
total 12
-rw-r--r--    1 root     root             0 Nov 10 15:43 etab
-rw-r--r--    1 root     root             0 Nov 10 15:43 rmtab
drwx------    2 nobody   root          4096 Apr  4 10:05 sm
drwx------    2 nobody   root          4096 Apr  4 10:05 sm.bak
-rw-r--r--    1 root     root             4 Apr  4 10:05 state
-rw-r--r--    1 root     root             0 Nov 10 15:43 xtab

msg from ncopa """ dmesg should tell you that grsecurity tries to prevent you to do this.

grsecurity does not permit the syscall mount from within a chroot since that is a way to break out of a chroot. This affects lxc containers too.

I would recommend that you do the mouting from the lxc host in the container config with lxc.mount.entry or similar.

https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html#lbAR

If you still want disable mount protection in grsecurity then you can do that with: echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount """

this is not working with 

lxc.mount.entry=nfsserver:/srv/boot/alpine mnt nfs nosuid,intr 0 0

on the host machine with all nfs modules and helper software installed and loaded. 

backend:~# lxc-start -n nfstest
lxc-start: conf.c: mount_entry: 2049 Invalid argument - failed to mount
'nfsserver:/srv/boot/alpine' on '/usr/lib/lxc/rootfs/mnt'
lxc-start: conf.c: lxc_setup: 4163 failed to setup the mount entries for
'nfstest'
lxc-start: start.c: do_start: 688 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1080 failed to spawn 'nfstest'

Nor with 

echo 0 > /proc/sys/kernel/grsecurity/chroot_deny_mount

on the host machine with all nfs modules and helper software installed and loaded which does'nt work either.

To find a proper way to use NFS shares from AL LXC is an important topic in order to be able to, for instance, load balance web servers sharing contents uploaded by users.

Next step will be to have HA for the NFS server itself (with only AL machines).

About NBD

NBD is now in edge/testing thanks to clandmeter.

we now use xnbd ^^

Also we are still looking after the right solution to backup NBD as a whole (versus by it's content) while in use. dd|nc is the used way nowadays.

About consul

nothing yet but big hopes ^^
I'm lurking IRC about it ;)

We plan to use it's dynamic DNS feature, it's hosts listing, services inventory, events, k/v store...
and even semi high-availability for our PXE infrastructure the consul leader being the active PXEserver and other consul server are dormant PXEservers.
All config scripts adapted to pull values out of consul k/v datastore based on profiles found out of consul various lists.
As the key for dhcpd and PXEboot is the hwaddr, it will become our uuid for LAN and consul too.
We are very exited by consul capacities!
Will be avid tester!

Open questions:

  1. What memory footprint is needed?
  2. What about dynamycally adapt quorum size?
  3. Are checks possible triggers? 
    • consul watch -prefix type -name name /path/to/executable
    • consul event [options] -name name [payload]
  4. What best practice to store etc configurations?

log of experimentation at User_talk:Jch/consul

About CEPH

CEPH is supposed to sovle the problem of high availability for the data stores, be it block devices (disks) or character devices (files).

The actual situation is not satisfactory.

We are very exited by CEPH capacities!
Will be avid tester!

The Alpine kernel has now RBD modules compiled.

We will build a CEPH cluster out of 3 Ubuntu LTS and use AL boxes as client if possible (to launch qemu instances directly from RBD). If not, we then will attach RBD and reexport them with xNBD inside a debian KVM.

About Docker

not a lot of information on the Docker page yet ...

About E-MailRelay

E-MailRelay is a simple SMTP proxy and store-and-forward message transfer agent (MTA).
See http://emailrelay.sourceforge.net/

It compiles fine on AL. 

apk update
apk add subversion alpine-sdk
svn checkout svn://svn.code.sf.net/p/emailrelay/code/trunk emailrelay-code
cd emailrelay-code
./configure --prefix=/usr
make
make install
apk del subversion alpine-sdk
apk add libgcc libstdc++
emailrelay --help

But I still have issues to properly build a package because it wants to install some stuff in <PREFIX>/libexec...
(And I also need to separate -doc, -test, -extra and optionnaly -gui in subpackages I guess)

About X2Go

x2goserver

I did prepare x2goserver and nx-libs packages.

x2goclient

lrelease-qt4 x2goclient.pro
/bin/bash: lrelease-qt4: command not found
Makefile:39: recipe for target 'build_client' failed

Dunno where to find that...

My laptop setup

AL 3.3 with +/etc/inittab+ 

tty5::respawn:/usr/bin/su - jch mcabber
tty6::respawn:/usr/bin/su - jch tmux
tty7::respawn:/usr/bin/su - jch startx

and +~/.xinitrc+ 

  1. !/bin/sh


exec chromium-browser --no-sandbox

About gpve

gvpe
http://software.schmorp.de/pkg/gvpe.html

Plan to use it to interconnect about 5 sites.

About freeswitch

I have a request to run a SIP server for a couple of users.
I'm doing it in some LXC accessed trough an openVPN from Jolla phones.

New rollout of our infra

This week, we will upgrade some hardware and also redo all the infrastructure based on the fresh 3.3 serie.

The compute nodes will run (on baremetal) with mdadm, openvswitch, qemu, consul, collectd, screen (maybe tmux) and openssh.

The storage nodes will run a CEPH cluster (unfortunately not based on AL).

Everything else will run in various KVM on the compute nodes.

First, let's check if the needed package are available in the basic ISOs. If yes we will be able to run from USB keys. If not we will need to have sys install on the HDD...

Retrieved from "https://wiki.alpinelinux.org/w/index.php?title=User_talk:Jch&oldid=24778"