Multi ISP
This document describes how to configure your Alpine Linux router with multiple ISPs, with or without failover and load balancing.
Please note: Alpine Linux v2.4.4 or newer is required.
| Interface | Name | IP address | Description |
|---|---|---|---|
| eth0 | ISP1 | DHCP | Internet with static IP address (we will use 1.2.3.4/24 and gw 1.2.3.1 as example) |
| eth1 | DMZ | 192.168.0.1 | Demilitarized zone |
| eth2 | LAN | 192.168.1.1 | Local network |
| eth3 | ISP2 | DHCP | Internet with dynamic IP address. |
Network Interfaces
If there are multiple static IP addresses, or to be exact, if there are multiple default gateways, you must specify a metric value. The gateway with the lowest metric will win.
/etc/network/interfaces:
auto eth0 iface eth0 inet static address 1.2.3.4 netmask 255.255.255.0 gateway 1.2.3.1 metric 200
The DHCP client will automatically add a metric value to the default gateway. It will pick 200 + interface index. Therefore we don't need to worry about that for DHCP.
auto eth3 iface eth0 inet dhcp
For PPP interfaces there is a defaultroute-metric option that can set the metric value.
Shorewall Zones
In /etc/shorewall/zones, create the zones:
INET ipv4 DMZ ipv4 LAN ipv4
Shorewall Interfaces
We need to make shorewall aware of the second ISP interface. In /etc/shorewall/params, add ISP1 under ISP2.
ISP1_IF=eth0 ISP2_IF=eth3 DMZ_IF=eth1 LAN_IF=eth2
Add it to /etc/shorewall/interfaces:
INET $ISP1_IF detect dhcp INET $ISP2_IF detect dhcp DMZ $DMZ_IF detect dhcp LAN $LAN_IF detect dhcp
The dhcp option is not really needed unless dhcp is actually used. We recommend leaving the dhcp option on in case the ISP changes to DHCP.
Shorewall Masquerade
We must masquerade the internet addresses so that every ISP gets a correct source address. This is done in /etc/shorewall/masq.
$ISP1_IF 0.0.0.0/0 $ISP2_IF 0.0.0.0/0
Shorewall Policy
We disallow routing between the 2 ISP's by adding a drop policy in /etc/shorewall/policy.
... INET INET DROP ...
Shorewall Providers
The /etc/shorewall/providers defines the 2 ISP's and allows firewall packet marking (the firewall can mark certain packages which can later be used for the routing decision). Shorewall will also let the firewall mark the incomming connections in order to keep track of what gateway the response should go via. This is needed in case you do DNAT of incomming traffic on the secondary ISP. Without the connection tracking the outgoing traffic would always go via the primary ISP.
We do not want shorewall to mess with the routing tables (which we needed it to do with older versions of pingu). We can tell shorewall not to flush the routing tables that pingu will manage by setting DUPLICATE column to '-', GATEWAY column to 'none' and leaving the COPY column empty.
In this example eth0 is attatched to ISP1, eth3 is attatched to ISP2.
#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY ISP1 1 1 - eth0 none track,optional,loose ISP2 2 2 - eth3 none track,optional,loose
After shorewall has restarted it should be possible to see the route rules for the firewall marks using the command ip rule:
... 10001: from all fwmark 0x1 lookup ISP1 10002: from all fwmark 0x2 lookup ISP2 ...
Shorewall Route Rules
We also use /etc/shorewall/route_rules to set up some static route rules. This is to force local traffic to ignore the alternate route tables for the ISP's.
#SOURCE DEST PROVIDER PRIORITY
# prevent local traffic go via providers' route tables - 192.168.0.1/24 main 1000 - 192.168.1.1/24 main 1000
Make Shorewall not touch default gateway
Shorewall will store the default gateway(s) on startup and try restore on stopping. Unfortunally it does it badly, restulting in a default gateway getting lost on stop. Additionally, the default gateway might have changed during runtime, due to dhcp lease renewal. We need configure shorewall to not do anything with the default gateways at all and let pingu handle this.
Add the following to /etc/shorewall/shorewall.conf:
RESTORE_DEFAULT_ROUTE=No
Policy Routing with Pingu
We need to make sure that where ISP1's IP address is used as source address, the ISP1's default gateway must be used. Otherwise we might send packets with ISP1 IP source address via ISP2. The response would come through the ISP2's interfaces and firewall would block.
Since IP address and default gateway might change (DHCP lease renewal) we need a daemon that monitors the source address of the gateways via the interfaces. This is what pingu does.
apk add pingu
We configure pingu to monitor our eth0 and eth3 interfaces in /etc/pingu/pingu.conf:
interface eth0 {
# route-table must correspond with NUMBER column in /etc/shorewall/providers
route-table 1
# the rule-priority must be a higher number than the priority in /etc/shorewall/route_rules
rule-priority 20000
}
interface eth3 {
# route-table must correspond with NUMBER column in /etc/shorewall/providers
route-table 2
rule-priority 20000
}
When pingu is started you should be able to see the routing rules with
ip rule
... 20000: from 1.2.3.4 lookup ISP1 20000: from 192.168.254.4 lookup ISP2 ...
and see the ISP route tables with
ip route show table 1
default via 1.2.3.1 dev eth0 metric 200 1.2.3.0/24 dev eth0 proto kernel scope link metric 200
Internet Failover with Pingu
To do failover we need a way to detect if an ISP is still or not. We configure some hosts that pingu will ping for us at regular intervals. When a host stops to respond we consider that host offline. We can bind a ping host to an interface in pingu and when enough hosts stops to respond, pingu will remove that gateway from main route table and a gateway will lower metric will take over. We normally want ping more than one host since a single host might down without it meaning that the rest of internet is down.
We also want pick hosts that is unlikely to change. DNS servers are good in this aspect. They seldomly change since it would require all clients to do reconfiguration. In this example we will use google's DNS and opendns hosts.
Add the hosts to /etc/pingu/pingu.conf and bind to primary ISP. We also set the ping timeout to 2.5 seconds.
...
# Ping responses that takes more than 2.5 seconds are considered lost. timeout 2.5
# ping google dns via ISP1
host 8.8.8.8 {
interval 60
bind-interface eth0
}
# ping opendns via ISP1
host 208.67.222.222 {
interval 60
bind-interface eth0
}
Now, if both hosts stops to respond to pings, ISP-1 will be considered down and all gateways via eth0 will be removed from main route table. Note that the gateway will not be removed from the route table '1'. This is so we can continue try ping via eth0 so we can detect that the ISP is back online. When ISP starts working again, the gateways will be added back to main route table again.
Load Balancing
It is possible to give the impression of load balancing by using multipath routes. It will not let a http file download use both ISPs for double download speed but it will let some users surf via ISP1 and others via ISP2, resulting that bandwith usage is spread over both ISPs.
Note that balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider. It also means that the source IP might change so websites that requires login (webmail, banks etc) will break if they check if the clients ip is consistent. Banking sites often do this. It is recommended to avoid this feature if possible.
To enable load balance, simply add the 'load-balance' keyword to all the interfaces that should be balanced in /etc/pingu/pingu.conf:
interface eth0 {
...
load-balance
}
interface eth3 {
...
load-balance
}
...