Dynamic Multipoint VPN (DMVPN): Difference between revisions

From Alpine Linux
(awall isp marks)
(replace /etc/init.d with rc-service)
 
(142 intermediate revisions by 12 users not shown)
Line 1: Line 1:
{{Draft}}
{{TOC right}}


http://alpinelinux.org/about under '''Why the Name Alpine?''' states:
https://alpinelinux.org/about under '''Why the Name Alpine?''' states:


''The first open-source implementation of Cisco's DMVPN, called OpenNHRP, was written for Alpine Linux.''
''The first open-source implementation of Cisco's DMVPN, called OpenNHRP, was written for Alpine Linux.''[https://web.archive.org/web/20110508001131/https://alpinelinux.org/about]


So the aim of this document is to be the reference DMVPN setup, with all the networking services needed for the clients that will use the DMVPN (DNS, DHCP, firewall, etc.).
So the aim of this document is to be the reference Linux DMVPN setup, with all the networking services needed for the clients that will use the DMVPN (DNS, firewall, etc.).  [[Small Office Services]] offers additional services such as DHCP for clients, http proxying, and a basic SIP telephone system.


= Terminology =
= Terminology =
'''NBMA''': ''Non-Broadcast Multi-Access'' network as described in [http://tools.ietf.org/html/rfc2332 RFC 2332]
;NBMA: ''Non-Broadcast Multi-Access'' network as described in [https://datatracker.ietf.org/doc/html/rfc2332 RFC 2332]


'''Hub''': the ''Next Hop Server'' (NHS) performing the Next Hop Resolution Protocol service within the NBMA cloud.
;Hub: the ''Next Hop Server'' (NHS) performing the Next Hop Resolution Protocol service within the NBMA cloud.


'''Spoke''': the ''Next Hop Resolution Protocol Client'' (NHC) which initiates NHRP requests of various types in order to obtain access to the NHRP service.
;Spoke: the ''Next Hop Resolution Protocol Client'' (NHC) which initiates NHRP requests of various types in order to obtain access to the NHRP service.


{{Tip|The recommended version of Alpine for building a DMVPN should be at minimum 2.4.11. Don't use 2.5.x, or 2.6.0 since it has in-tunnel IP fragmentation issues. Alpine 2.6.1 or later should be also okay}}
{{Tip|At the time of this writing the recommended Alpine version for building a DMVPN should be at minimum 2.4.11. Don't use 2.5.x, or 2.6.0 since the kernel has in-tunnel IP fragmentation issues. Alpine 3.0 has also a Musl issue in getprotobyname(). Alpine 2.7.x has been thoroughly tested and 3.0.3 hasn't shown any issue so far.}}


{{Note|This document assumes that all Alpine installations are run in [[Installation#Basics|diskless mode]] and that the configuration is saved on USB key}}
{{Note|This document assumes that all Alpine installations are run in [[Installation#Basics|diskless mode]] and that the configuration is saved on USB key}}
= Hardware =
If you need gigabits throughput you should go for a processor with [https://www.intel.com/content/www/us/en/developer/articles/technical/advanced-encryption-standard-instructions-aes-ni.html AES-NI] and [https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sha-extensions.html SHA Extensions].
= Extract Certificates =
We will use certificates for DMVPN and for OpenVPN (RoadWarrior clients). If you are in need to generate your own certificates, please see [[Generating_SSL_certs_with_ACF]]. You should use a separate machine for this purpose. If you downloaded the certificates on a Windows machine, you may use [https://winscp.net/eng/download.php WinSCP] to copy them on the DMVPN box.
Here are the general purpose instruction for extracting certificates from pfx files:
{{Cmd|openssl pkcs12 -in cert.pfx -cacerts -nokeys -out cacert.pem
openssl pkcs12 -in cert.pfx -nocerts -nodes -out serverkey.pem
openssl pkcs12 -in cert.pfx -nokeys -clcerts -out cert.pem
}}
Set appropriate permission for your certificate files:
{{Cmd|chmod 600 *.pem *.pfx}}


= Spoke Node =
= Spoke Node =
A local spoke node network has support for multiple ISP connections, along with redundant layer 2 switches.  At least one 802.1q capable switch is required, and a second is optional for redundancy purposes.  The typical spoke node network looks like:
[[File:DMVPN-Spoke.png]]
== Boot Alpine USB ==
Follow the instructions on [[Create_a_Bootable_USB]] about how to create a bootable USB.
== Alpine Setup ==
== Alpine Setup ==
We will setup the network interfaces as follows:
We will setup the network interfaces as follows:


bond0.1 = LAN<br>
{|class="wikitable"
bond0.2 = DMZ<br>
!'''Interface'''
bond0.10 = ISP1<br>
!'''Description'''
bond0.11 = ISP2<br>
!'''Subnet'''
|-
|bond0.3
|Management
|10.1.0.129/26
|-
|bond0.101
|LAN
|10.1.0.0/25
|-
|bond0.256
|Internet from ISP1
|Allocated from ISP
|-
|bond0.257
|Internet from ISP2
|Allocated from ISP
|-
|bond0.620
|Transit between wifi proxy and dmvpn spoke node
|10.1.0.252/30
|-
|bond0.701
|WiFi clients (no access to DMVPN network)
|172.17.48.0/24
|-
|bond0.1101
|Voice
|10.2.0.0/24
|}


Boot Alpine in [[Installation#Basics|diskless mode]] and run <code>setup-alpine</code>
{{Cmd|setup-alpine}}


{|class="wikitable"
{|class="wikitable"
Line 43: Line 96:
|-
|-
|<code>Available interfaces are: eth0<br>Enter '?' for help on bridges, bonding and vlans.<br>Which one do you want to initialize? (or '?' done')</code>
|<code>Available interfaces are: eth0<br>Enter '?' for help on bridges, bonding and vlans.<br>Which one do you want to initialize? (or '?' done')</code>
|''Enter'' '''bond0.1'''
|''Enter'' '''bond0.101'''
|-
|-
|<code>Available bond slaves are: eth0 eth1<br>Which slave(s) do you want to add to bond0? (or 'done') [eth0]</code>
|<code>Available bond slaves are: eth0 eth1<br>Which slave(s) do you want to add to bond0? (or 'done') [eth0]</code>
|'''eth0 eth1'''
|'''eth0 eth1'''
|-
|-
|<code>IP address for bond0? (or 'dhcp', 'none', '?') [dhcp]:</code>
|<code>IP address for bond0? (or 'dhcp', 'none', '?') [none]:</code>
|''Press Enter confirming 'none'''
|''Press Enter confirming 'none'''
|-
|-
|<code>IP address for bond0.1? (or 'dhcp', 'none', '?') [dhcp]:</code>
|<code>IP address for bond0.101? (or 'dhcp', 'none', '?') [dhcp]:</code>
|''Enter the IP address of you LAN interface, e.g.'' '''10.1.0.1'''
|''Enter the IP address of your LAN interface, e.g.'' '''10.1.0.1'''
|-
|-
|<code>Netmask? [255.255.255.0]:</code>
|<code>Netmask? [255.255.255.0]:</code>
|''Press Enter confirming '255.255.255.0' or type an appropriate another appropriate subnet mask''
|''Press Enter confirming '255.255.255.0' or type another appropriate subnet mask''
|-
|-
|<code>Gateway? (or 'none') [none]:</code>
|<code>Gateway? (or 'none') [none]:</code>
Line 63: Line 116:
|'''yes'''
|'''yes'''
|-
|-
|''Make a copy of the bond0.1 configuration for bond0.2, bond0.10 and bond0.20 (optional) interfaces.<br>Don't forget to add a gateway and a metric value for ISP interfaces when multiple gateways are set.<br>Save and close the file (:wq)''
|''Make a copy of the bond0.101 configuration for bond0.620, bond0.701, bond0.1101, bond0.256 and bond0.257 (optional) interfaces.<br>Don't forget to add a gateway and a metric value for ISP interfaces when multiple gateways are set.<br>Save and close the file (:wq)''
|-
|-
|<code>DNS domain name? (e.g. 'bar.com') []:</code>
|<code>DNS domain name? (e.g. 'bar.com') []:</code>
Line 80: Line 133:
|''Press Enter confirming 'UTC'''
|''Press Enter confirming 'UTC'''
|-
|-
|<code>HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none]</code>
|<code>HTTP/FTP proxy URL? (e.g. '<nowiki>http://proxy:8080</nowiki>', or 'none') [none]</code>
|''Press Enter confirming 'none'''
|''Press Enter confirming 'none'''
|-
|-
Line 101: Line 154:
|''Press Enter confirming '/media/usb/cache'''
|''Press Enter confirming '/media/usb/cache'''
|}
|}
== Networking ==
Update the networking configuration.
With your favorite editor open <code>/etc/network/interfaces</code> and add interfaces:
{{cat|/etc/network/interfaces|
    ...
auto bond0.101
iface bond0.101 inet static
        address 10.1.0.1
        netmask 255.255.255.192
auto bond0.620
iface bond0.620 inet static
        address 10.1.0.253
        netmask 255.255.255.252
auto bond0.701
iface bond0.101 inet static
        address 172.17.48.1
        netmask 255.255.255.0
auto bond0.1101
iface bond0.101 inet static
        address 10.2.0.1
        netmask 255.255.255.0
auto bond0.256
iface bond0.256 inet static
        address <%ISP1_IP_ADDRESS%>
        netmask <%ISP1_NETMASK%>
auto bond0.257
iface bond0.257 inet static
        address <%ISP2_IP_ADDRESS%>
        netmask <%ISP2_NETMASK%>
}}


== Bonding ==
== Bonding ==
Update the bonding configuration:
Update the bonding configuration.
 
With your favorite editor open <code>/etc/network/interfaces</code> and add <code>bond-mode</code>, <code>bond-miimon</code> and <code>bond-updelay</code> parameters to the <code>bond0</code> stanza:
 
{{cat|/etc/network/interfaces|
auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    bond-mode balance-tlb
    bond-miimon 100
    bond-updelay 500
    ...
}}
 
Bring up the new bonding settings:
 
{{Cmd|ifdown bond0
ifup bond0}}
 
 
== Physically install ==
At this point, you're ready to connect the VPN Spoke Node to the network if you haven't already done so.  Please set up an 802.1q capable switch with the VLANs listed in AlpineSetup section.  Once done, tag all of the VLANs on one port.  Connect that port to <code>eth0</code>.  Then, connect your first ISP's CPE to a switchport with VLAN 256 untagged.
 
== SSH ==
Remove password authentication and DNS reverse lookup:
 
{{Cmd|sed -i "s/.PasswordAuthentication yes/PasswordAuthentication no/" /etc/ssh/sshd_config
sed -i "s/.UseDNS yes/UseDNS no/" /etc/ssh/sshd_config}}
 
Restart ssh:
{{Cmd|rc-service sshd restart}}


echo bonding mode=balance-tlb miimon=100 updelay=500 >> /etc/modules
== NTP server ==
In order to have attached devices syncing their time agains this host, we need to do some modifications to chrony config.<BR>
Add '<code>allow all</code>' to the end of the '<code>/etc/chrony/chrony.conf</code>' so the file looks something like this:


== DNS  ==
{{cat|/etc/chrony/chrony.conf|
{{Cmd|apk add unbound}}
server pool.ntp.org
initstepslew 10 pool.ntp.org
commandkey 10
keyfile /etc/chrony/chrony.keys
driftfile /etc/chrony/chrony.drift
allow all
}}


Edit /etc/unbound/unbound.conf and add the following configuration.  If you have a domain that you want unbound to resolve but is internal to your network only, uncomment the stub-zone stanza and replace the stub-addr with the appropriate DNS server:
Restart chronyd for the changes to take effect
<pre>
{{cmd|rc-service chronyd restart}}
 
== Recursive DNS  ==
Install package(s):
 
{{Cmd|apk add -U unbound}}
 
With your favorite editor open <code>/etc/unbound/unbound.conf</code> and add the following configuration.  If you have a domain that you want unbound to resolve but is internal to your network only, the stub-zone stanza is present:
 
{{cat|/etc/unbound/unbound.conf|
server:
server:
         verbosity: 1
         verbosity: 1
Line 122: Line 261:
         access-control: 10.1.0.0/16 allow
         access-control: 10.1.0.0/16 allow
         access-control: 127.0.0.0/8 allow
         access-control: 127.0.0.0/8 allow
do-not-query-localhost: no
do-not-query-localhost: no
root-hints: "/etc/unbound/named.cache"
#stub-zone:
# name: "example.net"
# stub-addr: 10.1.0.10
python:
remote-control:
        control-enable: no
</pre>


Fetch the latest copy of root hints:
root-hints: "/etc/unbound/root.hints"
 
stub-zone:
name: "location1.example.net"
stub-addr: 10.1.0.2
 
stub-zone:
        name: "example.net"
        stub-addr: 172.16.255.1
        stub-addr: 172.16.255.2
        stub-addr: 172.16.255.3
        stub-addr: 172.16.255.4
        stub-addr: 172.16.255.5
        stub-addr: 172.16.255.7
 
stub-zone:
        name: "example2.net"
        stub-addr: 172.16.255.1
        stub-addr: 172.16.255.2
        stub-addr: 172.16.255.3
        stub-addr: 172.16.255.4
        stub-addr: 172.16.255.5
        stub-addr: 172.16.255.7
}}


{{Cmd|wget http://ftp.internic.net/domain/named.cache -O /etc/unbound/named.cache}}
Start unbound and start using unbound on this host:


{{Cmd|/etc/init.d/unbound start}}
{{Cmd|rc-service unbound start
{{Cmd|echo nameserver 10.1.0.1 > /etc/resolv.conf}}
rc-update add unbound
echo nameserver 10.1.0.1 > /etc/resolv.conf}}


== GRE Tunnel ==
== GRE Tunnel ==
With your favorite editor open <code>/etc/network/interfaces</code> and add the following:
With your favorite editor open <code>/etc/network/interfaces</code> and add the following:


auto gre1
{{cat|/etc/network/interfaces|<nowiki>
iface gre1 inet static
auto gre1
      pre-up ip tunnel add $IFACE mode gre ttl 64 tos inherit key 12.34.56.78 || true
iface gre1 inet static
      address 172.16.1.1
      pre-up ip tunnel add $IFACE mode gre ttl 64 tos inherit key 12.34.56.78 || true
      netmask 255.255.0.0
      address 172.16.1.1
      post-down ip tunnel del $IFACE || true
      netmask 255.255.0.0
      post-down ip tunnel del $IFACE || true
</nowiki>}}


Save and close the file.
Bring up the new <code>gre1</code> interface:


{{Cmd|ifup gre1}}
{{Cmd|ifup gre1}}


== IPSEC ==
== IPSEC ==
Install package(s):
{{Cmd|apk add ipsec-tools}}
{{Cmd|apk add ipsec-tools}}


With your favorite editor open <code>/etc/ipsec.conf</code> and change the content to the following:
With your favorite editor create <code>/etc/ipsec.conf</code> and set the content to the following:


spdflush;
{{cat|/etc/ipsec.conf|
spdadd 0.0.0.0/0 0.0.0.0/0 gre -P out ipsec esp/transport//require;
spdflush;
spdadd 0.0.0.0/0 0.0.0.0/0 gre -P in ipsec esp/transport//require;
spdadd 0.0.0.0/0 0.0.0.0/0 gre -P out ipsec esp/transport//require;
spdadd 0.0.0.0/0 0.0.0.0/0 gre -P in ipsec esp/transport//require;
}}


With your favorite editor open <code>/etc/racoon/racoon.conf</code> and change the content to the following:
Create missing directory:


<pre>
{{Cmd|mkdir /etc/racoon/}}
 
Extract your pfx into <code>/etc/racoon</code>, using the filenames '''<code>ca.pem</code>''', '''<code>cert.pem</code>''', and '''<code>key.pem</code>''' (see [[Dynamic_Multipoint_VPN_%28DMVPN%29#Extract_Certificates|instructions above]] for command).
 
With your favorite editor create <code>/etc/racoon/racoon.conf</code> and set the content to the following:
 
{{cat|/etc/racoon/racoon.conf|
path certificate "/etc/racoon/";
remote anonymous {
remote anonymous {
exchange_mode main;
exchange_mode main;
Line 195: Line 364:
compression_algorithm deflate;
compression_algorithm deflate;
}
}
</pre>
}}
 
Edit <code>/etc/conf.d/racoon</code> and unset <code>RACOON_PSK_FILE</code>:
 
{{cat|/etc/conf.d/racoon|
...
RACOON_PSK_FILE{{=}}
...
}}


Save and close the file.
Start service(s):


{{Cmd|/etc/init.d/racoon start}}
{{Cmd|rc-service racoon start
rc-update add racoon}}


== Next Hop Resolution Protocol (NHRP) ==
== Next Hop Resolution Protocol (NHRP) ==
Install package(s):
{{Cmd|apk add opennhrp}}
{{Cmd|apk add opennhrp}}


With your favorite editor open <code>/etc/opennhrp/opennhrp.conf</code> and change the content to the following:
With your favorite editor open <code>/etc/opennhrp/opennhrp.conf</code> and change the content to the following:


<pre>
{{cat|/etc/opennhrp/opennhrp.conf|
interface gre1
interface gre1
dynamic-map 172.16.0.0/16 hub.example.com
dynamic-map 172.16.0.0/16 hub.example.com
Line 212: Line 392:
redirect
redirect
non-caching
non-caching
interface bond0.1
 
interface bond0.8
shortcut-destination
 
interface bond0.64
shortcut-destination
shortcut-destination
interface bond0.2
 
interface bond0.620
shortcut-destination
shortcut-destination
</pre>
}}
 
You must have a DNS A record ''<code>hub.example.com</code>'' for each hub node IP address.


With your favorite editor open <code>/etc/opennhrp/opennhrp-script</code> and change the content to the following:
With your favorite editor open <code>/etc/opennhrp/opennhrp-script</code> and change the content to the following:


<pre>
{{cat|/etc/opennhrp/opennhrp-script|<nowiki>#!/bin/sh
#!/bin/sh
 
MYAS=$(sed -n 's/router bgp \(\d*\)/\1/p' < /etc/quagga/bgpd.conf)


case $1 in
case $1 in
interface-up)
interface-up)
ip route flush proto 42 dev $NHRP_INTERFACE
    echo "Interface $NHRP_INTERFACE is up"
ip neigh flush dev $NHRP_INTERFACE
    if [ "$NHRP_INTERFACE" = "gre1" ]; then
;;
        ip route flush proto 42 dev $NHRP_INTERFACE
        ip neigh flush dev $NHRP_INTERFACE
 
        vtysh -d bgpd \
            -c "configure terminal" \
            -c "router bgp $MYAS" \
            -c "no neighbor core" \
            -c "neighbor core peer-group"
    fi
    ;;
peer-register)
peer-register)
;;
    ;;
peer-up)
peer-up)
if [ -n "$NHRP_DESTMTU" ]; then
    if [ -n "$NHRP_DESTMTU" ]; then
ARGS=`ip route get $NHRP_DESTNBMA from $NHRP_SRCNBMA | head -1`
        ARGS=`ip route get $NHRP_DESTNBMA from $NHRP_SRCNBMA | head -1`
ip route add $ARGS proto 42 mtu $NHRP_DESTMTU
        ip route add $ARGS proto 42 mtu $NHRP_DESTMTU
fi
    fi
echo "Create link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
    echo "Create link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
racoonctl establish-sa -w isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA || exit 1
    racoonctl establish-sa -w isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA || exit 1
racoonctl establish-sa -w esp inet $NHRP_SRCNBMA $NHRP_DESTNBMA gre || exit 1  
    racoonctl establish-sa -w esp inet $NHRP_SRCNBMA $NHRP_DESTNBMA gre || exit 1
vtysh -d bgpd -c "clear bgp $NHRP_DESTADDR" 2>/dev/null || true
    ;;
;;
peer-down)
peer-down)
echo "Delete link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
    echo "Delete link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
if [ "$NHRP_PEER_DOWN_REASON" != "lower-down" ]; then
    racoonctl delete-sa isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA
racoonctl delete-sa isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA
    ip route del $NHRP_DESTNBMA src $NHRP_SRCNBMA proto 42
fi
    ;;
ip route del $NHRP_DESTNBMA src $NHRP_SRCNBMA proto 42
nhs-up)
;;
    echo "NHS UP $NHRP_DESTADDR"
    (
        flock -x 200
        vtysh -d bgpd \
            -c "configure terminal" \
            -c "router bgp $MYAS" \
            -c "neighbor $NHRP_DESTADDR remote-as 65000" \
            -c "neighbor $NHRP_DESTADDR peer-group core" \
            -c "exit" \
            -c "exit" \
            -c "clear bgp $NHRP_DESTADDR"
    ) 200>/var/lock/opennhrp-script.lock
    ;;
nhs-down)
    (
        flock -x 200
        vtysh -d bgpd \
            -c "configure terminal" \
            -c "router bgp $MYAS" \
            -c "no neighbor $NHRP_DESTADDR"
    ) 200>/var/lock/opennhrp-script.lock
    ;;
route-up)
route-up)
echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is up"
    echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is up"
ip route replace $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 via $NHRP_NEXTHOP dev $NHRP_INTERFACE
    ip route replace $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 via $NHRP_NEXTHOP dev $NHRP_INTERFACE
ip route flush cache
    ip route flush cache
;;
    ;;
route-down)
route-down)
echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is down"
    echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is down"
ip route del $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42
    ip route del $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42
ip route flush cache
    ip route flush cache
;;
    ;;
esac
esac


exit 0
exit 0
</nowiki>}}


</pre>
Make it executable and start service(s):
 
Save and close the file. Make it executable:


{{Cmd|chmod +x /etc/opennhrp/opennhrp-script}}
{{Cmd|chmod +x /etc/opennhrp/opennhrp-script
rc-service opennhrp start
rc-update add opennhrp}}


== BGP ==
== BGP ==
{{Cmd|apk add quagga}}
Install package(s):


{{Cmd|touch /etc/quagga/zebra.conf}}
{{Cmd|apk add quagga
touch /etc/quagga/zebra.conf}}


With your favorite editor open <code>/etc/quagga/bgpd.conf</code> and change the content to the following:
With your favorite editor open <code>/etc/quagga/bgpd.conf</code> and change the content to the following (replace <code>strongpassword</code> with a password of your choice and <code>%HUB_GRE_IP%</code> with the '''Hub''' node GRE IP address):
* Add the line <code>neighbor %HUB_GRE_IP% remote-as 65000</code> for each '''Hub''' host you have in your NBMA cloud.


<pre>
{{cat|/etc/quagga/bgpd.conf|
password strongpassword
password strongpassword
enable password strongpassword
enable password strongpassword
Line 292: Line 512:
neighbor %HUB_GRE_IP% remote-as 65000
neighbor %HUB_GRE_IP% remote-as 65000
         ...
         ...
</pre>
}}
 
Start service(s):
 
{{Cmd|rc-service bgpd start
rc-update add bgpd}}
 
== OpenVPN ==
Install package(s):
 
{{Cmd|echo tun >> /etc/modules
modprobe tun
apk add openvpn openssl
openssl dhparam -out /etc/openvpn/dh1024.pem 1024}}
 
Configure openvpn:
 
{{cat|/etc/openvpn/openvpn.conf|
dev tun
proto udp
port 1194
 
server 10.1.128.0 255.255.255.0
push "route 10.0.0.0 255.0.0.0"
push "dhcp-option DNS 10.1.0.1"
 
tls-server
ca /etc/openvpn/cacert.pem
cert /etc/openvpn/servercert.pem
key /etc/openvpn/serverkey.pem
 
crl-verify /etc/openvpn/crl.pem
 
dh /etc/openvpn/dh1024.pem
 
persist-key
persist-tun
 
keepalive 10 120
 
comp-lzo


Add lines <code>neighbor %HUB_GRE_IP%...</code> for each '''Hub''' host you have in your NBMA cloud.
status /var/log/openvpn.status
mute 20
verb 3
}}


Save and close the file.
Start service(s):


{{Cmd|/etc/init.d/bgpd start}}
{{Cmd|rc-service openvpn start
rc-update add openvpn}}


== Firewall ==
== Firewall ==
Install package(s):
{{Cmd|apk add awall}}
{{Cmd|apk add awall}}
Enable IP forwarding:
{{Cmd|sysctl -w net.ipv4.ip_forward{{=}}1
sed -i 's/.*net\.ipv4\.ip_forward.*$/net.ipv4.ip_forward {{=}} 1/g' /etc/sysctl.conf}}


With your favorite editor, edit the following files and set their contents as follows:
With your favorite editor, edit the following files and set their contents as follows:




<code>'''/etc/awall/optional/params.json'''</code>
{{cat|/etc/awall/optional/params.json|
<pre>
{
{
   "description": "params",
   "description": "params",


   "variable": {
   "variable": {
     "B_IF" = "bond0.1",
     "B_IF": "bond0.8",
     "C_IF" = "bond0.2",
     "C_IF": "bond0.64",
     "E_IF" = "bond0.10",
     "DE_IF": "bond0.620",
     "E_IF2" = "bond0.11"
     "ISP1_IF": "bond0.256",
    "ISP2_IF": "bond0.257"
   }
   }
}
}
</pre>
}}
 




<code>'''/etc/awall/optional/internet-host.json'''</code>
{{cat|/etc/awall/optional/internet-host.json|
<pre>
{
{
   "description": "Internet host",
   "description": "Internet host",
Line 329: Line 600:


   "zone": {
   "zone": {
     "E": { "iface": "$E_IF" },
     "E": { "iface": [ "$ISP1_IF", "$ISP2_IF" ] },
     "E2": { "iface": "$E_IF2" }
    "ISP1": { "iface": "$ISP1_IF" },
     "ISP2": { "iface": "$ISP2_IF" }
   },
   },


   "filter": [
   "filter": [
     {
     {
       "in": [ "E", "E2 ],
       "in": "E",
       "service": "ping",
       "service": "ping",
       "action": "accept",
       "action": "accept",
Line 341: Line 613:
     },
     },
     {
     {
       "in": [ "E", "E2" ],
       "in": "E",
       "out": "_fw",
       "out": "_fw",
       "service": "ssh",
       "service": [ "ssh", "https" ],
       "action": "accept",
       "action": "accept",
       "conn-limit": { "count": 3, "interval": 60 }
       "conn-limit": { "count": 3, "interval": 60 }
Line 350: Line 622:
     {
     {
       "in": "_fw",
       "in": "_fw",
       "out": [ "E", "E2" ],
       "out": "E",
       "service": [ "dns", "http", "ntp" ],
       "service": [ "dns", "http", "ntp" ],
       "action": "accept"
       "action": "accept"
Line 361: Line 633:
   ]
   ]
}
}
</pre>
}}
 
 
 
{{cat|/etc/awall/optional/openvpn.json|
{
  "description": "OpenVPN support",
 
  "import": "internet-host",
 
  "service": {
    "openvpn": { "proto": "udp", "port": 1194 }
  },
 
  "filter": [
    { "in": "E", "out": "_fw", "service": "openvpn", "action": "accept" }
    ]
}
}}
 
 
 
{{cat|/etc/awall/optional/clampmss.json|
{
  "description": "Deal with ISPs afraid of ICMP",
 
  "import": "internet-host",
 
  "clamp-mss": [ { "out": "E" } ]
}
}}
 




<code>'''/etc/awall/optional/mark.json'''</code>
{{cat|/etc/awall/optional/mark.json|
<pre>
{
{
   "description": "Mark traffic based on ISP",
   "description": "Mark traffic based on ISP",
Line 372: Line 674:


   "route-track": [
   "route-track": [
     { "in": "E", "mark": 1 },
     { "out": "ISP1", "mark": 1 },
     { "in": "E2", "mark": 2 }
     { "out": "ISP2", "mark": 2 }
   ]
   ]
}
}
</pre>
}}
 
 


<code>'''/etc/awall/optional/dmvpn.json'''</code>
{{cat|/etc/awall/optional/dmvpn.json|
<pre>
{
{
   "description": "DMVPN router",
   "description": "DMVPN router",
Line 386: Line 689:


   "variable": {
   "variable": {
     "A_ADDR": [ "10.0.0.0/8", "172.16.0.0/16" ],
     "A_ADDR": [ "10.0.0.0/8", "172.16.0.0/16" ]
    "A_IF": "gre1"
   },
   },


   "zone": {
   "zone": {
     "A": { "addr": "$A_ADDR", "iface": "$A_IF" }
     "A": { "addr": "$A_ADDR", "iface": "gre1" }
   },
   },


   "filter": [
   "filter": [
     { "in": [ "E", "E2" ], "out": "_fw", "service": "ipsec", "action": "accept" },
     { "in": "E", "out": "_fw", "service": "ipsec", "action": "accept" },
     { "in": "_fw", "out": [ "E", "E2" ], "service": "ipsec", "action": "accept" },
     { "in": "_fw", "out": "E", "service": "ipsec", "action": "accept" },
     {
     {
       "in": [ "E", "E2" ],
       "in": "E",
       "out": "_fw",
       "out": "_fw",
       "ipsec": "in",
       "ipsec": "in",
Line 406: Line 708:
     {
     {
       "in": "_fw",
       "in": "_fw",
       "out": [ "E", "E2" ],
       "out": "E",
       "ipsec": "out",
       "ipsec": "out",
       "service": "gre",
       "service": "gre",
Line 414: Line 716:
     { "in": "_fw", "out": "A", "service": "bgp", "action": "accept" },
     { "in": "_fw", "out": "A", "service": "bgp", "action": "accept" },
     { "in": "A", "out": "_fw", "service": "bgp", "action": "accept"},
     { "in": "A", "out": "_fw", "service": "bgp", "action": "accept"},
     { "out": [ "E", "E2" ], "dest": "$A_ADDR", "action": "reject" }
     { "out": "E", "dest": "$A_ADDR", "action": "reject" }
   ]
   ]
}
}
</pre>
}}
 


<code>'''/etc/awall/optional/vpnc.json'''</code>
{{cat|/etc/awall/optional/vpnc.json|
<pre>
{
{
   "description": "VPNc",
   "description": "VPNc",
Line 429: Line 729:
   "zone": {
   "zone": {
     "B": { "iface": "$B_IF" },
     "B": { "iface": "$B_IF" },
     "C": { "iface": "$C_IF" }
     "C": { "iface": "$C_IF" },
    "DE": { "iface": "$DE_IF" }
 
   },
   },


   "policy": [
   "policy": [
Line 437: Line 738:
     { "in": "B", "out": "A", "action": "accept" },
     { "in": "B", "out": "A", "action": "accept" },
     { "in": "C", "out": [ "A", "E" ], "action": "accept" },
     { "in": "C", "out": [ "A", "E" ], "action": "accept" },
     { "in": [ "E", "E2" ], "action": "drop" }
     { "in": "DE", "out": "E", "action": "accept" },
    { "in": "E", "action": "drop" },
     { "in": "_fw", "out": "A", "action": "accept" }
     { "in": "_fw", "out": "A", "action": "accept" }
   ],
   ],


   "snat": [
   "snat": [
     { "out": [ "E", "E2" ] }
     { "out": "E" }
   ],
   ],


Line 471: Line 773:
       "out": "_fw",
       "out": "_fw",
       "proto": "icmp",
       "proto": "icmp",
      "action": "accept"
    },
    {
      "out": "DE",
      "service": [ "ssh", "http", "https", "ping" ],
       "action": "accept"
       "action": "accept"
     }
     }
Line 476: Line 784:
   ]
   ]
}
}
}}
Activate the firewall:
{{Cmd|modprobe ip_tables
modprobe iptable_nat
awall enable clampmss
awall enable openvpn
awall enable vpnc
awall activate -f
rc-update add iptables}}
== ISP Failover ==
Install package(s):
{{Cmd|apk add pingu
echo -e "1\tisp1">> /etc/iproute2/rt_tables
echo -e "2\tisp2">> /etc/iproute2/rt_tables}}
Configure pingu to monitor our <code>bond0.256</code> and <code>bond0.257</code> interfaces in <code>/etc/pingu/pingu.conf</code>.  Add the hosts to monitor for ISP failover to <code>/etc/pingu/pingu.conf</code> and bind to primary ISP. We also set the ping timeout to 4 seconds.:
{{cat|/etc/pingu/pingu.conf|
timeout 4
required 2
retry 11
interface bond0.256 {
  # route-table must correspond with mark in /etc/awall/optional/mark.json
  route-table 1
  fwmark 1
  rule-priority 20000
  # google dns
  ping 8.8.8.8
  # opendns
  ping 208.67.222.222
}
interface bond0.257 {
  # route-table must correspond with mark in /etc/awall/optional/mark.json
  route-table 2
  fwmark 2
  rule-priority 20000
}
}}
Make sure we can reach the public IP from our LAN by adding static route rules for our private net(s). Edit <code>/etc/pingu/route-rules</code>:
{{cat|/etc/pingu/route-rules|
to 10.0.0.0/8 table main prio 1000
to 172.16.0.0/12 table main prio 1000
}}
Start service(s):
{{Cmd|rc-service pingu start
rc-update add pingu}}
Now, if both hosts stop responding to pings, ISP-1 will be considered down and all gateways via bond0.256 will be removed from main route table. Note that the gateway will not be removed from the route table '1'. This is so we can continue try ping via <code>bond0.256</code> so we can detect that the ISP is back online. When ISP starts working again, the gateways will be added back to main route table again.
== Commit Configuration ==
Commit configuration:
{{Cmd|lbu ci}}
= Hub Node =
We will document only what changes from the Spoke node setup.
== Routing Tables ==
<pre>
echo -e "42\tnhrp_shortcut\n43\tnhrp_mtu\n44\tquagga\n>> /etc/iproute2/rt_tables
</pre>
</pre>
Add the following "up" commands:
{{cat|/etc/network/interfaces|
auto gre1
...
    up ip rule add lookup nhrp_shortcut pref 11000
    up ip rule add lookup quagga pref 11001
    up ip rule add lookup nhrp_mtu pref 11999
}}
== NHRP ==
With your favorite editor open <code>/etc/opennhrp/opennhrp.conf</code> on Hub 2 and set the content as follows:
{{cat|/etc/opennhrp/opennhrp.conf|
interface gre1
  map %Hub1_GRE_IP%/%MaskBit% hub1.example.org
  route-table 44
  shortcut
  redirect
  non-caching
}}
Do the same on Hub 1 adding the data relative to Hub 2.
With your favorite editor open <code>/etc/opennhrp/opennhrp-script</code> and set the content as follows:
<pre>
#!/bin/sh
case $1 in
interface-up)
ip route flush proto 42 dev $NHRP_INTERFACE
ip neigh flush dev $NHRP_INTERFACE
;;
peer-register)
CERT=`racoonctl get-cert inet $NHRP_SRCNBMA $NHRP_DESTNBMA | openssl x509 -inform der -text -noout | egrep -o "/OU=[^/]*(/[0-9]+)?" | cut -b 5-`
if [ -z "`echo "$CERT" | grep "^GRE=$NHRP_DESTADDR"`" ]; then
logger -t opennhrp-script -p auth.err "GRE registration of $NHRP_DESTADDR to $NHRP_DESTNBMA DENIED"
exit 1
fi
logger -t opennhrp-script -p auth.info "GRE registration of $NHRP_DESTADDR to $NHRP_DESTNBMA authenticated"
(
flock -x 200
AS=`echo "$CERT" | grep "^AS=" | cut -b 4-`
vtysh -d bgpd -c "configure terminal" \
-c "router bgp 65000" \
-c "neighbor $NHRP_DESTADDR remote-as $AS" \
-c "neighbor $NHRP_DESTADDR peer-group leaf" \
-c "neighbor $NHRP_DESTADDR prefix-list net-$AS-in in"
SEQ=5
(echo "$CERT" | grep "^NET=" | cut -b 5-) | while read NET; do
vtysh -d bgpd -c "configure terminal" \
-c "ip prefix-list net-$AS-in seq $SEQ permit $NET le 26"
SEQ=$(($SEQ+5))
done
) 200>/var/lock/opennhrp-script.lock
;;
peer-up)
echo "Create link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
racoonctl establish-sa -w isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA || exit 1
racoonctl establish-sa -w esp inet $NHRP_SRCNBMA $NHRP_DESTNBMA gre || exit 1
CERT=`racoonctl get-cert inet $NHRP_SRCNBMA $NHRP_DESTNBMA | openssl x509 -inform der -text -noout | egrep -o "/OU=[^/]*(/[0-9]+)?" | cut -b 5-`
if [ -z "`echo "$CERT" | grep "^GRE=$NHRP_DESTADDR"`" ]; then
logger -p daemon.err "GRE mapping of $NHRP_DESTADDR to $NHRP_DESTNBMA DENIED"
exit 1
fi
if [ -n "$NHRP_DESTMTU" ]; then
ARGS=`ip route get $NHRP_DESTNBMA from $NHRP_SRCNBMA | head -1`
ip route add $ARGS proto 42 mtu $NHRP_DESTMTU table nhrp_mtu
fi
;;
peer-down)
echo "Delete link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
if [ "$NHRP_PEER_DOWN_REASON" != "lower-down" ]; then
racoonctl delete-sa isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA
fi
ip route del $NHRP_DESTNBMA src $NHRP_SRCNBMA proto 42 table nhrp_mtu
;;
route-up)
echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is up"
ip route replace $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 via $NHRP_NEXTHOP dev $NHRP_INTERFACE table nhrp_shortcut
ip route flush cache
;;
route-down)
echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is down"
ip route del $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 table nhrp_shortcut
ip route flush cache
;;
esac
exit 0
</pre>
== BGP ==
With your favorite editor open <code>/etc/quagga/bgpd.conf</code> on Hub 2 and set the content as follows:
{{cat|/etc/quagga/bgpd.conf|
password zebra
enable password zebra
log syslog
router bgp 65000
bgp router-id %Hub2_GRE_IP%
bgp deterministic-med
network %GRE_NETWORK%/%MASK_BITS%
neighbor hub peer-group
neighbor hub next-hop-self
neighbor hub route-map CORE-IN in
neighbor spoke peer-group
neighbor spoke passive
neighbor spoke next-hop-self
neighbor %Spoke1_GRE_IP% remote-as 65001
neighbor %Spoke1_GRE_IP% peer-group spoke
neighbor %Spoke1_GRE_IP% prefix-list net-65001-in in
...
...
...
neighbor hub remote-as 65000
neighbor %Hub1_GRE_IP% peer-group core
ip prefix-list net-65001-in seq 5 permit 10.1.0.0/16 le 26
...
route-map CORE-IN permit 10
set metric +100
}}
Add the lines <code>neighbor %Spoke1_GRE_IP%...</code> for each spoke node you have. Do the same on Hub 1, changing the relevant data for Hub 2.
= Troubleshooting the DMVPN =
== Broken [https://wikipedia.org/wiki/Path_MTU_Discovery Path MTU Discovery (PMTUD)] ==
ISPs afraid of ICMP (which is somehow legitimate) often just blindly add <code>no ip unreachables</code> in their router interfaces, effectively creating a [https://wikipedia.org/wiki/Black_hole_%28networking%29 blackhole router] that breaks PMTUD, since ICMP Type 3 Code 4 packets (Fragmentation Needed) are dropped. PMTUD is needed by ISAKMP that runs on UDP (TCP works because it uses CLAMPMSS).
For technical details see https://packetlife.net/blog/2008/oct/9/disabling-unreachables-breaks-pmtud/
PMTUD could also be broken due to badly configured DSL modem/routers or bugged firmware. Turning off the firewall on modem itself or any VPN passthrough functionality it may help.
You can easily detect which host is the blackhole router by pinging with DF bit set and with packets of standard MTU size, each hop given in your traceroute to destination:
{{Cmd|ping -M do -s 1472 %IP%}}
{{Note|"-M do" requires GNU ping, present in <code>iputils</code> package}}
If you don't get a response back (either Echo-Response or Fragmentation-Needed) there's firewall dropping ICMP packets. If it answers to normal ping packets (DF bit cleared), most likely you have hit a blackhole router.
== Kernel and NHRP Routing Cache Issues ==
{{Todo|...}}
[[category: VPN]]

Latest revision as of 10:45, 17 November 2023

https://alpinelinux.org/about under Why the Name Alpine? states:

The first open-source implementation of Cisco's DMVPN, called OpenNHRP, was written for Alpine Linux.[1]

So the aim of this document is to be the reference Linux DMVPN setup, with all the networking services needed for the clients that will use the DMVPN (DNS, firewall, etc.). Small Office Services offers additional services such as DHCP for clients, http proxying, and a basic SIP telephone system.

Terminology

NBMA
Non-Broadcast Multi-Access network as described in RFC 2332
Hub
the Next Hop Server (NHS) performing the Next Hop Resolution Protocol service within the NBMA cloud.
Spoke
the Next Hop Resolution Protocol Client (NHC) which initiates NHRP requests of various types in order to obtain access to the NHRP service.
Tip: At the time of this writing the recommended Alpine version for building a DMVPN should be at minimum 2.4.11. Don't use 2.5.x, or 2.6.0 since the kernel has in-tunnel IP fragmentation issues. Alpine 3.0 has also a Musl issue in getprotobyname(). Alpine 2.7.x has been thoroughly tested and 3.0.3 hasn't shown any issue so far.
Note: This document assumes that all Alpine installations are run in diskless mode and that the configuration is saved on USB key

Hardware

If you need gigabits throughput you should go for a processor with AES-NI and SHA Extensions.

Extract Certificates

We will use certificates for DMVPN and for OpenVPN (RoadWarrior clients). If you are in need to generate your own certificates, please see Generating_SSL_certs_with_ACF. You should use a separate machine for this purpose. If you downloaded the certificates on a Windows machine, you may use WinSCP to copy them on the DMVPN box.

Here are the general purpose instruction for extracting certificates from pfx files:

openssl pkcs12 -in cert.pfx -cacerts -nokeys -out cacert.pem openssl pkcs12 -in cert.pfx -nocerts -nodes -out serverkey.pem openssl pkcs12 -in cert.pfx -nokeys -clcerts -out cert.pem

Set appropriate permission for your certificate files:

chmod 600 *.pem *.pfx

Spoke Node

A local spoke node network has support for multiple ISP connections, along with redundant layer 2 switches. At least one 802.1q capable switch is required, and a second is optional for redundancy purposes. The typical spoke node network looks like:

Boot Alpine USB

Follow the instructions on Create_a_Bootable_USB about how to create a bootable USB.

Alpine Setup

We will setup the network interfaces as follows:

Interface Description Subnet
bond0.3 Management 10.1.0.129/26
bond0.101 LAN 10.1.0.0/25
bond0.256 Internet from ISP1 Allocated from ISP
bond0.257 Internet from ISP2 Allocated from ISP
bond0.620 Transit between wifi proxy and dmvpn spoke node 10.1.0.252/30
bond0.701 WiFi clients (no access to DMVPN network) 172.17.48.0/24
bond0.1101 Voice 10.2.0.0/24

setup-alpine

You will be prompted something like this... Suggestion on what you could enter...
Select keyboard layout [none]: Type an appropriate layout for you
Select variant: Type an appropriate layout for you (if prompted)
Enter system hostname (short form, e.g. 'foo') [localhost]: Enter the hostname, e.g. vpnc
Available interfaces are: eth0
Enter '?' for help on bridges, bonding and vlans.
Which one do you want to initialize? (or '?' done')
Enter bond0.101
Available bond slaves are: eth0 eth1
Which slave(s) do you want to add to bond0? (or 'done') [eth0]
eth0 eth1
IP address for bond0? (or 'dhcp', 'none', '?') [none]: Press Enter confirming 'none'
IP address for bond0.101? (or 'dhcp', 'none', '?') [dhcp]: Enter the IP address of your LAN interface, e.g. 10.1.0.1
Netmask? [255.255.255.0]: Press Enter confirming '255.255.255.0' or type another appropriate subnet mask
Gateway? (or 'none') [none]: Press Enter confirming 'none'
Do you want to do any manual network configuration? [no] yes
Make a copy of the bond0.101 configuration for bond0.620, bond0.701, bond0.1101, bond0.256 and bond0.257 (optional) interfaces.
Don't forget to add a gateway and a metric value for ISP interfaces when multiple gateways are set.
Save and close the file (:wq)
DNS domain name? (e.g. 'bar.com') []: Enter the domain name of your intranet, e.g., example.net
DNS nameservers(s)? []: 8.8.8.8 8.8.4.4 (we will change them later)
Changing password for root
New password:
Enter a secure password for the console
Retype password: Retype the above password
Which timezone are you in? ('?' for list) [UTC]: Press Enter confirming 'UTC'
HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] Press Enter confirming 'none'
Enter mirror number (1-9) or URL to add (or r/f/e/done) [f]: Select a mirror close to you and press Enter
Which SSH server? ('openssh', 'dropbear' or 'none') [openssh]: Press Enter confirming 'openssh'
Which NTP client to run? ('openntpd', 'chrony' or 'none') [chrony]: Press Enter confirming 'chrony'
Which disk(s) would you like to use? (or '?' for help or 'none') [none]: Press Enter confirming 'none' or type 'none' if needed
Enter where to store configs ('floppy', 'usb' or 'none') [usb]: Press Enter confirming 'usb'
Enter apk cache directory (or '?' or 'none') [/media/usb/cache]: Press Enter confirming '/media/usb/cache'

Networking

Update the networking configuration.

With your favorite editor open /etc/network/interfaces and add interfaces:

Contents of /etc/network/interfaces

... auto bond0.101 iface bond0.101 inet static address 10.1.0.1 netmask 255.255.255.192 auto bond0.620 iface bond0.620 inet static address 10.1.0.253 netmask 255.255.255.252 auto bond0.701 iface bond0.101 inet static address 172.17.48.1 netmask 255.255.255.0 auto bond0.1101 iface bond0.101 inet static address 10.2.0.1 netmask 255.255.255.0 auto bond0.256 iface bond0.256 inet static address <%ISP1_IP_ADDRESS%> netmask <%ISP1_NETMASK%> auto bond0.257 iface bond0.257 inet static address <%ISP2_IP_ADDRESS%> netmask <%ISP2_NETMASK%>

Bonding

Update the bonding configuration.

With your favorite editor open /etc/network/interfaces and add bond-mode, bond-miimon and bond-updelay parameters to the bond0 stanza:

Contents of /etc/network/interfaces

auto bond0 iface bond0 inet manual bond-slaves eth0 eth1 bond-mode balance-tlb bond-miimon 100 bond-updelay 500 ...

Bring up the new bonding settings:

ifdown bond0 ifup bond0


Physically install

At this point, you're ready to connect the VPN Spoke Node to the network if you haven't already done so. Please set up an 802.1q capable switch with the VLANs listed in AlpineSetup section. Once done, tag all of the VLANs on one port. Connect that port to eth0. Then, connect your first ISP's CPE to a switchport with VLAN 256 untagged.

SSH

Remove password authentication and DNS reverse lookup:

sed -i "s/.PasswordAuthentication yes/PasswordAuthentication no/" /etc/ssh/sshd_config sed -i "s/.UseDNS yes/UseDNS no/" /etc/ssh/sshd_config

Restart ssh:

rc-service sshd restart

NTP server

In order to have attached devices syncing their time agains this host, we need to do some modifications to chrony config.
Add 'allow all' to the end of the '/etc/chrony/chrony.conf' so the file looks something like this:

Contents of /etc/chrony/chrony.conf

server pool.ntp.org initstepslew 10 pool.ntp.org commandkey 10 keyfile /etc/chrony/chrony.keys driftfile /etc/chrony/chrony.drift allow all

Restart chronyd for the changes to take effect

rc-service chronyd restart

Recursive DNS

Install package(s):

apk add -U unbound

With your favorite editor open /etc/unbound/unbound.conf and add the following configuration. If you have a domain that you want unbound to resolve but is internal to your network only, the stub-zone stanza is present:

Contents of /etc/unbound/unbound.conf

server: verbosity: 1 interface: 10.1.0.1 do-ip4: yes do-ip6: no do-udp: yes do-tcp: yes do-daemonize: yes access-control: 10.1.0.0/16 allow access-control: 127.0.0.0/8 allow do-not-query-localhost: no root-hints: "/etc/unbound/root.hints" stub-zone: name: "location1.example.net" stub-addr: 10.1.0.2 stub-zone: name: "example.net" stub-addr: 172.16.255.1 stub-addr: 172.16.255.2 stub-addr: 172.16.255.3 stub-addr: 172.16.255.4 stub-addr: 172.16.255.5 stub-addr: 172.16.255.7 stub-zone: name: "example2.net" stub-addr: 172.16.255.1 stub-addr: 172.16.255.2 stub-addr: 172.16.255.3 stub-addr: 172.16.255.4 stub-addr: 172.16.255.5 stub-addr: 172.16.255.7

Start unbound and start using unbound on this host:

rc-service unbound start rc-update add unbound echo nameserver 10.1.0.1 > /etc/resolv.conf

GRE Tunnel

With your favorite editor open /etc/network/interfaces and add the following:

Contents of /etc/network/interfaces

auto gre1 iface gre1 inet static pre-up ip tunnel add $IFACE mode gre ttl 64 tos inherit key 12.34.56.78 || true address 172.16.1.1 netmask 255.255.0.0 post-down ip tunnel del $IFACE || true

Bring up the new gre1 interface:

ifup gre1

IPSEC

Install package(s):

apk add ipsec-tools

With your favorite editor create /etc/ipsec.conf and set the content to the following:

Contents of /etc/ipsec.conf

spdflush; spdadd 0.0.0.0/0 0.0.0.0/0 gre -P out ipsec esp/transport//require; spdadd 0.0.0.0/0 0.0.0.0/0 gre -P in ipsec esp/transport//require;

Create missing directory:

mkdir /etc/racoon/

Extract your pfx into /etc/racoon, using the filenames ca.pem, cert.pem, and key.pem (see instructions above for command).

With your favorite editor create /etc/racoon/racoon.conf and set the content to the following:

Contents of /etc/racoon/racoon.conf

path certificate "/etc/racoon/"; remote anonymous { exchange_mode main; lifetime time 2 hour; certificate_type x509 "/etc/racoon/cert.pem" "/etc/racoon/key.pem"; ca_type x509 "/etc/racoon/ca.pem"; my_identifier asn1dn; nat_traversal on; script "/etc/opennhrp/racoon-ph1dead.sh" phase1_dead; dpd_delay 120; proposal { encryption_algorithm aes 256; hash_algorithm sha1; authentication_method rsasig; dh_group modp4096; } proposal { encryption_algorithm aes 256; hash_algorithm sha1; authentication_method rsasig; dh_group 2; } } sainfo anonymous { pfs_group 2; lifetime time 2 hour; encryption_algorithm aes 256; authentication_algorithm hmac_sha1; compression_algorithm deflate; }

Edit /etc/conf.d/racoon and unset RACOON_PSK_FILE:

Contents of /etc/conf.d/racoon

... RACOON_PSK_FILE= ...

Start service(s):

rc-service racoon start rc-update add racoon

Next Hop Resolution Protocol (NHRP)

Install package(s):

apk add opennhrp

With your favorite editor open /etc/opennhrp/opennhrp.conf and change the content to the following:

Contents of /etc/opennhrp/opennhrp.conf

interface gre1 dynamic-map 172.16.0.0/16 hub.example.com shortcut redirect non-caching interface bond0.8 shortcut-destination interface bond0.64 shortcut-destination interface bond0.620 shortcut-destination

You must have a DNS A record hub.example.com for each hub node IP address.

With your favorite editor open /etc/opennhrp/opennhrp-script and change the content to the following:

Contents of /etc/opennhrp/opennhrp-script

#!/bin/sh MYAS=$(sed -n 's/router bgp \(\d*\)/\1/p' < /etc/quagga/bgpd.conf) case $1 in interface-up) echo "Interface $NHRP_INTERFACE is up" if [ "$NHRP_INTERFACE" = "gre1" ]; then ip route flush proto 42 dev $NHRP_INTERFACE ip neigh flush dev $NHRP_INTERFACE vtysh -d bgpd \ -c "configure terminal" \ -c "router bgp $MYAS" \ -c "no neighbor core" \ -c "neighbor core peer-group" fi  ;; peer-register)  ;; peer-up) if [ -n "$NHRP_DESTMTU" ]; then ARGS=`ip route get $NHRP_DESTNBMA from $NHRP_SRCNBMA | head -1` ip route add $ARGS proto 42 mtu $NHRP_DESTMTU fi echo "Create link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)" racoonctl establish-sa -w isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA || exit 1 racoonctl establish-sa -w esp inet $NHRP_SRCNBMA $NHRP_DESTNBMA gre || exit 1  ;; peer-down) echo "Delete link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)" racoonctl delete-sa isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA ip route del $NHRP_DESTNBMA src $NHRP_SRCNBMA proto 42  ;; nhs-up) echo "NHS UP $NHRP_DESTADDR" ( flock -x 200 vtysh -d bgpd \ -c "configure terminal" \ -c "router bgp $MYAS" \ -c "neighbor $NHRP_DESTADDR remote-as 65000" \ -c "neighbor $NHRP_DESTADDR peer-group core" \ -c "exit" \ -c "exit" \ -c "clear bgp $NHRP_DESTADDR" ) 200>/var/lock/opennhrp-script.lock  ;; nhs-down) ( flock -x 200 vtysh -d bgpd \ -c "configure terminal" \ -c "router bgp $MYAS" \ -c "no neighbor $NHRP_DESTADDR" ) 200>/var/lock/opennhrp-script.lock  ;; route-up) echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is up" ip route replace $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 via $NHRP_NEXTHOP dev $NHRP_INTERFACE ip route flush cache  ;; route-down) echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is down" ip route del $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 ip route flush cache  ;; esac exit 0

Make it executable and start service(s):

chmod +x /etc/opennhrp/opennhrp-script rc-service opennhrp start rc-update add opennhrp

BGP

Install package(s):

apk add quagga touch /etc/quagga/zebra.conf

With your favorite editor open /etc/quagga/bgpd.conf and change the content to the following (replace strongpassword with a password of your choice and %HUB_GRE_IP% with the Hub node GRE IP address):

  • Add the line neighbor %HUB_GRE_IP% remote-as 65000 for each Hub host you have in your NBMA cloud.

Contents of /etc/quagga/bgpd.conf

password strongpassword enable password strongpassword log syslog access-list 1 remark Command line access authorized IP access-list 1 permit 127.0.0.1 line vty access-class 1 hostname vpnc.example.net router bgp 65001 bgp router-id 172.16.1.1 network 10.1.0.0/16 neighbor %HUB_GRE_IP% remote-as 65000 neighbor %HUB_GRE_IP% remote-as 65000 ...

Start service(s):

rc-service bgpd start rc-update add bgpd

OpenVPN

Install package(s):

echo tun >> /etc/modules modprobe tun apk add openvpn openssl openssl dhparam -out /etc/openvpn/dh1024.pem 1024

Configure openvpn:

Contents of /etc/openvpn/openvpn.conf

dev tun proto udp port 1194 server 10.1.128.0 255.255.255.0 push "route 10.0.0.0 255.0.0.0" push "dhcp-option DNS 10.1.0.1" tls-server ca /etc/openvpn/cacert.pem cert /etc/openvpn/servercert.pem key /etc/openvpn/serverkey.pem crl-verify /etc/openvpn/crl.pem dh /etc/openvpn/dh1024.pem persist-key persist-tun keepalive 10 120 comp-lzo status /var/log/openvpn.status mute 20 verb 3

Start service(s):

rc-service openvpn start rc-update add openvpn

Firewall

Install package(s):

apk add awall

Enable IP forwarding:

sysctl -w net.ipv4.ip_forward=1 sed -i 's/.*net\.ipv4\.ip_forward.*$/net.ipv4.ip_forward = 1/g' /etc/sysctl.conf

With your favorite editor, edit the following files and set their contents as follows:


Contents of /etc/awall/optional/params.json

{ "description": "params", "variable": { "B_IF": "bond0.8", "C_IF": "bond0.64", "DE_IF": "bond0.620", "ISP1_IF": "bond0.256", "ISP2_IF": "bond0.257" } }


Contents of /etc/awall/optional/internet-host.json

{ "description": "Internet host", "import": "params", "zone": { "E": { "iface": [ "$ISP1_IF", "$ISP2_IF" ] }, "ISP1": { "iface": "$ISP1_IF" }, "ISP2": { "iface": "$ISP2_IF" } }, "filter": [ { "in": "E", "service": "ping", "action": "accept", "flow-limit": { "count": 10, "interval": 6 } }, { "in": "E", "out": "_fw", "service": [ "ssh", "https" ], "action": "accept", "conn-limit": { "count": 3, "interval": 60 } }, { "in": "_fw", "out": "E", "service": [ "dns", "http", "ntp" ], "action": "accept" }, { "in": "_fw", "service": [ "ping", "ssh" ], "action": "accept" } ] }


Contents of /etc/awall/optional/openvpn.json

{ "description": "OpenVPN support", "import": "internet-host", "service": { "openvpn": { "proto": "udp", "port": 1194 } }, "filter": [ { "in": "E", "out": "_fw", "service": "openvpn", "action": "accept" } ] }


Contents of /etc/awall/optional/clampmss.json

{ "description": "Deal with ISPs afraid of ICMP", "import": "internet-host", "clamp-mss": [ { "out": "E" } ] }


Contents of /etc/awall/optional/mark.json

{ "description": "Mark traffic based on ISP", "import": [ "params", "internet-host" ], "route-track": [ { "out": "ISP1", "mark": 1 }, { "out": "ISP2", "mark": 2 } ] }


Contents of /etc/awall/optional/dmvpn.json

{ "description": "DMVPN router", "import": "internet-host", "variable": { "A_ADDR": [ "10.0.0.0/8", "172.16.0.0/16" ] }, "zone": { "A": { "addr": "$A_ADDR", "iface": "gre1" } }, "filter": [ { "in": "E", "out": "_fw", "service": "ipsec", "action": "accept" }, { "in": "_fw", "out": "E", "service": "ipsec", "action": "accept" }, { "in": "E", "out": "_fw", "ipsec": "in", "service": "gre", "action": "accept" }, { "in": "_fw", "out": "E", "ipsec": "out", "service": "gre", "action": "accept" }, { "in": "_fw", "out": "A", "service": "bgp", "action": "accept" }, { "in": "A", "out": "_fw", "service": "bgp", "action": "accept"}, { "out": "E", "dest": "$A_ADDR", "action": "reject" } ] }

Contents of /etc/awall/optional/vpnc.json

{ "description": "VPNc", "import": [ "params", "internet-host", "dmvpn" ], "zone": { "B": { "iface": "$B_IF" }, "C": { "iface": "$C_IF" }, "DE": { "iface": "$DE_IF" } }, "policy": [ { "in": "A", "action": "accept" }, { "in": "B", "out": "A", "action": "accept" }, { "in": "C", "out": [ "A", "E" ], "action": "accept" }, { "in": "DE", "out": "E", "action": "accept" }, { "in": "E", "action": "drop" }, { "in": "_fw", "out": "A", "action": "accept" } ], "snat": [ { "out": "E" } ], "filter": [ { "in": "A", "out": "_fw", "service": [ "ping", "ssh", "http", "https" ], "action": "accept" }, { "in": [ "B", "C" ], "out": "_fw", "service": [ "dns", "ntp", "http", "https", "ssh" ], "action": "accept" }, { "in": "_fw", "out": [ "B", "C" ], "service": [ "dns", "ntp" ], "action": "accept" }, { "in": [ "A", "B", "C" ], "out": "_fw", "proto": "icmp", "action": "accept" }, { "out": "DE", "service": [ "ssh", "http", "https", "ping" ], "action": "accept" } ] }

Activate the firewall:

modprobe ip_tables modprobe iptable_nat awall enable clampmss awall enable openvpn awall enable vpnc awall activate -f rc-update add iptables

ISP Failover

Install package(s):

apk add pingu echo -e "1\tisp1">> /etc/iproute2/rt_tables echo -e "2\tisp2">> /etc/iproute2/rt_tables

Configure pingu to monitor our bond0.256 and bond0.257 interfaces in /etc/pingu/pingu.conf. Add the hosts to monitor for ISP failover to /etc/pingu/pingu.conf and bind to primary ISP. We also set the ping timeout to 4 seconds.:

Contents of /etc/pingu/pingu.conf

timeout 4 required 2 retry 11 interface bond0.256 { # route-table must correspond with mark in /etc/awall/optional/mark.json route-table 1 fwmark 1 rule-priority 20000 # google dns ping 8.8.8.8 # opendns ping 208.67.222.222 } interface bond0.257 { # route-table must correspond with mark in /etc/awall/optional/mark.json route-table 2 fwmark 2 rule-priority 20000 }

Make sure we can reach the public IP from our LAN by adding static route rules for our private net(s). Edit /etc/pingu/route-rules:

Contents of /etc/pingu/route-rules

to 10.0.0.0/8 table main prio 1000 to 172.16.0.0/12 table main prio 1000

Start service(s):

rc-service pingu start rc-update add pingu

Now, if both hosts stop responding to pings, ISP-1 will be considered down and all gateways via bond0.256 will be removed from main route table. Note that the gateway will not be removed from the route table '1'. This is so we can continue try ping via bond0.256 so we can detect that the ISP is back online. When ISP starts working again, the gateways will be added back to main route table again.

Commit Configuration

Commit configuration:

lbu ci

Hub Node

We will document only what changes from the Spoke node setup.

Routing Tables

echo -e "42\tnhrp_shortcut\n43\tnhrp_mtu\n44\tquagga\n>> /etc/iproute2/rt_tables

Add the following "up" commands:

Contents of /etc/network/interfaces

auto gre1 ... up ip rule add lookup nhrp_shortcut pref 11000 up ip rule add lookup quagga pref 11001 up ip rule add lookup nhrp_mtu pref 11999

NHRP

With your favorite editor open /etc/opennhrp/opennhrp.conf on Hub 2 and set the content as follows:

Contents of /etc/opennhrp/opennhrp.conf

interface gre1 map %Hub1_GRE_IP%/%MaskBit% hub1.example.org route-table 44 shortcut redirect non-caching

Do the same on Hub 1 adding the data relative to Hub 2.

With your favorite editor open /etc/opennhrp/opennhrp-script and set the content as follows:

#!/bin/sh
case $1 in
interface-up)
	ip route flush proto 42 dev $NHRP_INTERFACE
	ip neigh flush dev $NHRP_INTERFACE
	;;
peer-register)
	CERT=`racoonctl get-cert inet $NHRP_SRCNBMA $NHRP_DESTNBMA | openssl x509 -inform der -text -noout | egrep -o "/OU=[^/]*(/[0-9]+)?" | cut -b 5-`
	if [ -z "`echo "$CERT" | grep "^GRE=$NHRP_DESTADDR"`" ]; then
		logger -t opennhrp-script -p auth.err "GRE registration of $NHRP_DESTADDR to $NHRP_DESTNBMA DENIED"
		exit 1
	fi
	logger -t opennhrp-script -p auth.info "GRE registration of $NHRP_DESTADDR to $NHRP_DESTNBMA authenticated"

	(
		flock -x 200
		
		AS=`echo "$CERT" | grep "^AS=" | cut -b 4-`
		vtysh -d bgpd -c "configure terminal" \
			-c "router bgp 65000" \
			-c "neighbor $NHRP_DESTADDR remote-as $AS" \
			-c "neighbor $NHRP_DESTADDR peer-group leaf" \
			-c "neighbor $NHRP_DESTADDR prefix-list net-$AS-in in"

		SEQ=5
		(echo "$CERT" | grep "^NET=" | cut -b 5-) | while read NET; do
			vtysh -d bgpd -c "configure terminal" \
				-c "ip prefix-list net-$AS-in seq $SEQ permit $NET le 26"
			SEQ=$(($SEQ+5))
		done
	) 200>/var/lock/opennhrp-script.lock
	;;
peer-up)
	echo "Create link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
	racoonctl establish-sa -w isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA || exit 1
	racoonctl establish-sa -w esp inet $NHRP_SRCNBMA $NHRP_DESTNBMA gre || exit 1 

	CERT=`racoonctl get-cert inet $NHRP_SRCNBMA $NHRP_DESTNBMA | openssl x509 -inform der -text -noout | egrep -o "/OU=[^/]*(/[0-9]+)?" | cut -b 5-`
	if [ -z "`echo "$CERT" | grep "^GRE=$NHRP_DESTADDR"`" ]; then
		logger -p daemon.err "GRE mapping of $NHRP_DESTADDR to $NHRP_DESTNBMA DENIED"
		exit 1
	fi

	if [ -n "$NHRP_DESTMTU" ]; then
		ARGS=`ip route get $NHRP_DESTNBMA from $NHRP_SRCNBMA | head -1`
		ip route add $ARGS proto 42 mtu $NHRP_DESTMTU table nhrp_mtu
	fi
	;;
peer-down)
	echo "Delete link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
	if [ "$NHRP_PEER_DOWN_REASON" != "lower-down" ]; then
		racoonctl delete-sa isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA
	fi
	ip route del $NHRP_DESTNBMA src $NHRP_SRCNBMA proto 42 table nhrp_mtu
	;;
route-up)
	echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is up"
	ip route replace $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 via $NHRP_NEXTHOP dev $NHRP_INTERFACE table nhrp_shortcut
	ip route flush cache
	;;
route-down)
	echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is down"
	ip route del $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 table nhrp_shortcut
	ip route flush cache
	;;
esac

exit 0

BGP

With your favorite editor open /etc/quagga/bgpd.conf on Hub 2 and set the content as follows:

Contents of /etc/quagga/bgpd.conf

password zebra enable password zebra log syslog router bgp 65000 bgp router-id %Hub2_GRE_IP% bgp deterministic-med network %GRE_NETWORK%/%MASK_BITS% neighbor hub peer-group neighbor hub next-hop-self neighbor hub route-map CORE-IN in neighbor spoke peer-group neighbor spoke passive neighbor spoke next-hop-self neighbor %Spoke1_GRE_IP% remote-as 65001 neighbor %Spoke1_GRE_IP% peer-group spoke neighbor %Spoke1_GRE_IP% prefix-list net-65001-in in ... ... ... neighbor hub remote-as 65000 neighbor %Hub1_GRE_IP% peer-group core ip prefix-list net-65001-in seq 5 permit 10.1.0.0/16 le 26 ... route-map CORE-IN permit 10 set metric +100

Add the lines neighbor %Spoke1_GRE_IP%... for each spoke node you have. Do the same on Hub 1, changing the relevant data for Hub 2.

Troubleshooting the DMVPN

Broken Path MTU Discovery (PMTUD)

ISPs afraid of ICMP (which is somehow legitimate) often just blindly add no ip unreachables in their router interfaces, effectively creating a blackhole router that breaks PMTUD, since ICMP Type 3 Code 4 packets (Fragmentation Needed) are dropped. PMTUD is needed by ISAKMP that runs on UDP (TCP works because it uses CLAMPMSS).

For technical details see https://packetlife.net/blog/2008/oct/9/disabling-unreachables-breaks-pmtud/

PMTUD could also be broken due to badly configured DSL modem/routers or bugged firmware. Turning off the firewall on modem itself or any VPN passthrough functionality it may help.

You can easily detect which host is the blackhole router by pinging with DF bit set and with packets of standard MTU size, each hop given in your traceroute to destination:

ping -M do -s 1472 %IP%

Note: "-M do" requires GNU ping, present in iputils package

If you don't get a response back (either Echo-Response or Fragmentation-Needed) there's firewall dropping ICMP packets. If it answers to normal ping packets (DF bit cleared), most likely you have hit a blackhole router.

Kernel and NHRP Routing Cache Issues

Todo: ...