Alternative Network Setup

These are notes on macvlan on a box with real vlans. The goal here is to have the host on a management vlan, and several guests each on other vlans. There's no need for the host to talk to the guests. I wanted to try to see if the guest could get dhcp addresses. Something like this:

Setup:

host	dhcp on vlan 8
guest1	dhcp on vlan 64
guest2	dhcp on vlan 129
guest3	dhcp (different address) on vlan 64

Host's /etc/network/interfaces file

auto lo
iface lo inet loopback
 
# MGMT vlan
auto eth0.8
iface eth0.8 inet dhcp
     hostname lxchost

# USR vlan - we bring it up, but dont assign an address
auto eth0.65
iface eth0.65 inet manual
   up ip link set $IFACE addr de:ad:be:ef:ca:fe
   up ip link set $IFACE up
   down ip link set $IFACE down

# VoIP vlan - we bring it up, but dont assign an address
auto eth0.129
iface eth0.129 inet manual
   up ip link set $IFACE addr 0f:f1:ce:c0:ff:ee
   up ip link set $IFACE up
   down ip link set $IFACE down

Here's /etc/lxc/lxc.conf

lxc.network.type   =   macvlan
lxc.network.link   =   eth0.65
lxc.network.name   =   eth0
# lxc.network.flags  =   up       # We will bring the interface up inside the container
# lxc.network.ipv4   =   0.0.0.0  # We are going to do dhcp later

Create the guests

for a in `seq 1 3`; do 
  lxc-create -n guest${a} -f /etc/lxc/lxc.conf -t alpine
  ln -s /etc/init.d/lxc /etc/init.d/lxc.guest${a}
done

vi /var/lib/lxc/guest2/config

  change lxc.network.link to eth0.129

Start and enter the first guest (this is where the fun starts)

/etc/init.d/lxc.guest1 start
lxc-console -n guest1

Fun inside the guest

The /etc/networking/interfaces file is already set up for dhcp, so let's just restart networking:

guest1:~# /etc/init.d/networking restart
* Stopping networking ...
*   eth0 ...
cat: can't open '/var/run/udhcpc.eth0.pid': No such file or directory
ifdown: warning: no dhcp clients found and stopped  [ !! ]
 * Starting networking ...
 *   eth0 ...
cat: can't open '/sys/class/net/eth0/ifindex': No such file or directory
/usr/share/udhcpc/default.script: line 125: arithmetic syntax error
/usr/share/udhcpc/default.script: line 125: arithmetic syntax error

But.. lookie there... we do have a real ip address.
The reason for the syntax errors is we don't have sys/class/net mounted... So let's mount it and try again....

guest1:~# mount -t sysfs none /sys
guest1:~# /etc/init.d/networking restart
 * Stopping networking ...
 *   eth0 ...   [ ok ]
 * Starting networking ...
 *   eth0 ...   [ ok ]
guest1:~#

We just opened ourselves to a world of hurt. But more on that later
Let's see if we can make this 'just work'. We're going to do some weird things, don't worry... its not standard

 guest1:~# rc-update add networking
 guest1:~# echo "sysfs		/sys		sysfs	auto,defaults 0 0" >>/etc/fstab
 guest1:~# cat - << EOF >/etc/network/interfaces
   #auto lo
   iface lo inet loopback

   auto eth0
   iface eth0 inet dhcp
       pre-up /bin/mount -a
       hostname guest1
   EOF
 ctrl-a q
 lxchost# /etc/init.d/lxc.guest1 restart
 lxchost# lxc-console -n guest1

We have Networking!
Repeat the configuration for guest2 and 3

What Works, What Doesnt

Pro
- Each guest has its own mac address
Con
- Can't ping from one guest to another
- Real /sys is mounted - the guest can shut down the host