Talk:LXC: Difference between revisions

From Alpine Linux
(Found the problem in udhcp.default script - each guest now has networking, no sys mounting, and individual firewall rules.)
(→‎Fun inside the guest: Remove network start (fixed in 2.5 current) - add new note about missing device files)
Line 76: Line 76:
=== Fun inside the guest ===
=== Fun inside the guest ===


If you just start networking, you'll get:
* /dev/null is currently created as a regular file
cat: can't open '/sys/class/net/eth0/ifindex': No such file or directory
* /dev/zero doesn't exist
/usr/share/udhcpc/default.script: line 125: arithmetic syntax error
/usr/share/udhcpc/default.script: line 125: arithmetic syntax error
This is because /sys is not mounted, and we don't [http://blog.bofh.it/debian/id_413 really want to do that].


Fortunately, there's a simple fix:
To create these, do the following from ''the host''


--- /usr/share/udhcpc/default.script.orig
<pre>
+++ /usr/share/udhcpc/default.script
rm -f /var/lib/lxc/[guest-name]/rootfs/dev/null
@@ -39,7 +39,7 @@
rm -f /var/lib/lxc/[guest-name]/rootfs/dev/zero
  }
mknod /var/lib/lxc/[guest-name]/rootfs/dev/zero c 1 5
 
mknod  /var/lib/lxc/[guest-name]/rootfs/dev/null c 1 3
  if_index() {
</pre>
- cat /sys/class/net/$interface/ifindex
 
  + cat /sys/class/net/$interface/ifindex || echo 0
We do this in the host because our default config drops mknod capabilites in the guest.
  }
 
  calc_metric() {
* Let's make the /etc/network/interfaces proper
  guest1:~# cat - << EOF >/etc/network/interfaces
    auto lo
    iface lo inet loopback
    auto eth0
    iface eth0 inet dhcp
        hostname guest1
    EOF
* And add networking:
  guest1:~# rc-update add networking
* And test
  ctrl-a q
  lxchost# /etc/init.d/lxc.guest1 restart
  lxchost# lxc-console -n guest1
* We have Networking!
* Repeat the patch and configuration for guest2 and 3


=== What Works, What Doesnt ===
=== What Works, What Doesnt ===

Revision as of 06:00, 12 April 2013

Alternative Network Setup

These are notes on macvlan on a box with real vlans. The goal here is to have the host on a management vlan, and several guests each on other vlans. There's no need for the host to talk to the guests. The host resides on the "OOB" network, and if the host needs to talk to a guest, it does so with lxc-console, like having a KVM. Each guest should get its address from the DHCP server on the appropriate vlan.Something like this:

Setup:

host dhcp on vlan 8
guest1 dhcp on vlan 64
guest2 dhcp on vlan 129
guest3 dhcp on vlan64 (different address)
  • Host's /etc/network/interfaces file
auto lo
iface lo inet loopback
 
# MGMT vlan
auto eth0.8
iface eth0.8 inet dhcp
     hostname lxchost

# USR vlan - we bring it up, but dont assign an address
auto eth0.65
iface eth0.65 inet manual
   up ip link set $IFACE addr de:ad:be:ef:ca:fe
   up ip link set $IFACE up
   down ip link set $IFACE down

# VoIP vlan - we bring it up, but dont assign an address
auto eth0.129
iface eth0.129 inet manual
   up ip link set $IFACE addr 0f:f1:ce:c0:ff:ee
   up ip link set $IFACE up
   down ip link set $IFACE down
  • Here's /etc/lxc/lxc.conf
lxc.network.type   =   macvlan
# Allow guests on the same vlan to see each other                                   
lxc.network.macvlan.mode = bridge                                                    
lxc.network.link   =   eth0.65                     
lxc.network.name   =   eth0                                                                   
# lxc.network.hwaddr = de:ad:be:ef:c0:00    # macvlan will make one up, but possible if wanted                 
# lxc.network.flags  =   up                 # Do NOT bring up the interface, we will do so within the container
# lxc.network.ipv4   =   0.0.0.0            # Do NOT assign an address, we do so within the container          
                                                                                                     
# Capabilities to drop (for instance, to stop the guest from mounting sys)   
# Taken from http://sourceforge.net/mailarchive/message.php?msg_id=28285704  
# sys_boot is not listed here, as it causes problems when the host tries to stop the guest

# If you trust the guest, then you can get by without dropping capabilities
                                                                                  
lxc.cap.drop= sys_admin audit_control audit_write fsetid ipc_lock                 
lxc.cap.drop= ipc_owner lease linux_immutable mac_admin mac_override mknod setfcap
lxc.cap.drop= setpcap sys_module sys_nice sys_pacct sys_ptrace sys_rawio
lxc.cap.drop= sys_tty_config sys_time  
  • Create the guests
for a in `seq 1 3`; do 
  lxc-create -n guest${a} -f /etc/lxc/lxc.conf -t alpine
  ln -s /etc/init.d/lxc /etc/init.d/lxc.guest${a}
done
  • vi /var/lib/lxc/guest2/config
  change lxc.network.link to eth0.129
  • Start and enter the first guest (this is where the fun starts)
/etc/init.d/lxc.guest1 start
lxc-console -n guest1

Fun inside the guest

  • /dev/null is currently created as a regular file
  • /dev/zero doesn't exist

To create these, do the following from the host

rm -f /var/lib/lxc/[guest-name]/rootfs/dev/null
rm -f /var/lib/lxc/[guest-name]/rootfs/dev/zero
mknod  /var/lib/lxc/[guest-name]/rootfs/dev/zero c 1 5
mknod  /var/lib/lxc/[guest-name]/rootfs/dev/null c 1 3

We do this in the host because our default config drops mknod capabilites in the guest.

What Works, What Doesnt

  • Pro
    • Each guest has its own mac address
    • Network connectivity between each guest
    • No communication allowed between host and guests (this is a plus in our case - managment vlan != user vlan)
    • if iptables modules are loaded in the host, each guest can create its own iptables rules (awall for all! sweet)
  • Con
    • No communication allowed between host and guests because we are not using a bridge interface (this is a plus in our case - managment vlan != user vlan)