Multi ISP

This document describes how to configure your Alpine Linux router with multiple ISPs, with or without failover and load balancing.

Please note: Alpine Linux v2.4.4 or newer is required.

Interface	Name	IP address	Description
eth0	ISP1	DHCP	Internet with static IP address (we will use 1.2.3.4/24 and gw 1.2.3.1 as example)
eth1	DMZ	192.168.0.1	Demilitarized zone
eth2	LAN	192.168.1.1	Local network
eth3	ISP2	DHCP	Internet with dynamic IP address.

Network Interfaces

If there are multiple static IP addresses, or to be exact, if there are multiple default gateways, you must specify a metric value. The gateway with the lowest metric will win.

/etc/network/interfaces:

auto eth0
iface eth0 inet static
  address 1.2.3.4
  netmask 255.255.255.0
  gateway 1.2.3.1
  metric 200

The DHCP client will automatically add a metric value to the default gateway. It will pick 200 + interface index. Therefore we don't need to worry about that for DHCP.

auto eth3
iface eth0 inet dhcp

For PPP interfaces there is a defaultroute-metric option that can set the metric value.

Shorewall Zones

In /etc/shorewall/zones, create the zones:

INET    ipv4
DMZ     ipv4
LAN     ipv4

Shorewall Interfaces

We need to make shorewall aware of the second ISP interface. In /etc/shorewall/params, add ISP1 under ISP2.

ISP1_IF=eth0
ISP2_IF=eth3
DMZ_IF=eth1
LAN_IF=eth2

Add it to /etc/shorewall/interfaces:

INET	$ISP1_IF	detect	dhcp
INET	$ISP2_IF	detect	dhcp
DMZ	$DMZ_IF	        detect	dhcp
LAN	$LAN_IF	        detect	dhcp

The dhcp option is not really needed unless dhcp is actually used. We recommend leaving the dhcp option on in case the ISP changes to DHCP.

Shorewall Masquerade

We must masquerade the internet addresses so that every ISP gets a correct source address. This is done in /etc/shorewall/masq.

$ISP1_IF	0.0.0.0/0
$ISP2_IF	0.0.0.0/0

Shorewall Policy

We disallow routing between the 2 ISP's by adding a drop policy in /etc/shorewall/policy.

...
INET    INET    DROP
...

Shorewall Providers

The /etc/shorewall/providers defines the 2 ISP's and allows firewall packet marking (the firewall can mark certain packages which can later be used for the routing decision). Shorewall will also let the firewall mark the incomming connections in order to keep track of what gateway the response should go via. This is needed in case you do DNAT of incomming traffic on the secondary ISP. Without the connection tracking the outgoing traffic would always go via the primary ISP.

We do not want shorewall to mess with the routing tables (which we needed it to do with older versions of pingu). We can tell shorewall not to flush the routing tables that pingu will manage by setting DUPLICATE column to '-', GATEWAY column to 'none' and leaving the COPY column empty.

In this example eth0 is attatched to ISP1, eth3 is attatched to ISP2.

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE    GATEWAY     OPTIONS                 COPY
ISP1	1       1       -               eth0         none        track,optional,loose
ISP2	2       2       -               eth3         none        track,optional,loose

After shorewall has restarted it should be possible to see the route rules for the firewall marks using the command ip rule:

...
10001:	from all fwmark 0x1 lookup ISP1 
10002:	from all fwmark 0x2 lookup ISP2 
...

Shorewall Route Rules

We also use /etc/shorewall/route_rules to set up some static route rules. This is to force local traffic to ignore the alternate route tables for the ISP's.

#SOURCE			DEST			PROVIDER	PRIORITY

# prevent local traffic go via providers' route tables
-			192.168.0.1/24		main		1000
-			192.168.1.1/24		main		1000

Make Shorewall not touch default gateway

Shorewall will store the default gateway(s) on startup and try restore on stopping. Unfortunally it does it badly, restulting in a default gateway getting lost on stop. Additionally, the default gateway might have changed during runtime, due to dhcp lease renewal. We need configure shorewall to not do anything with the default gateways at all and let pingu handle this.

Add the following to /etc/shorewall/shorewall.conf:

RESTORE_DEFAULT_ROUTE=No

Policy Routing with Pingu

We need to make sure that where ISP1's IP address is used as source address, the ISP1's default gateway must be used. Otherwise we might send packets with ISP1 IP source address via ISP2. The response would come through the ISP2's interfaces and firewall would block.

Since IP address and default gateway might change (DHCP lease renewal) we need a daemon that monitors the source address of the gateways via the interfaces. This is what pingu does.

apk add pingu

We configure pingu to monitor our eth0 and eth3 interfaces in /etc/pingu/pingu.conf:

interface eth0 { 
  # route-table must correspond with NUMBER column in /etc/shorewall/providers
  route-table 1
  # the rule-priority must be a higher number than the priority in /etc/shorewall/route_rules
  rule-priority 20000
}

interface eth3 {
  # route-table must correspond with NUMBER column in /etc/shorewall/providers
  route-table 2
  rule-priority 20000
}

When pingu is started you should be able to see the routing rules with

ip rule

...
20000:	from 1.2.3.4 lookup ISP1 
20000:	from 192.168.254.4 lookup ISP2 
...

and see the ISP route tables with

ip route show table 1

default via 1.2.3.1 dev eth0  metric 200
1.2.3.0/24 dev eth0  proto kernel  scope link  metric 200 

Internet Failover with Pingu

To do failover we need a way to detect if an ISP is still or not. We configure some hosts that pingu will ping for us at regular intervals. When a host stops to respond we consider that host offline. We can bind a ping host to an interface in pingu and when enough hosts stops to respond, pingu will remove that gateway from main route table and a gateway will lower metric will take over. We normally want ping more than one host since a single host might down without it meaning that the rest of internet is down.

We also want pick hosts that is unlikely to change. DNS servers are good in this aspect. They seldomly change since it would require all clients to do reconfiguration. In this example we will use google's DNS and opendns hosts.

Add the hosts to /etc/pingu/pingu.conf and bind to primary ISP. We also set the ping timeout to 2.5 seconds.

...

# Ping responses that takes more than 2.5 seconds are considered lost.
timeout 2.5

# ping google dns via ISP1
host 8.8.8.8 {
  interval 60
  bind-interface eth0
}

# ping opendns via ISP1
host 208.67.222.222 {
  interval 60
  bind-interface eth0
}

Now, if both hosts stops to respond to pings, ISP-1 will be considered down and all gateways via eth0 will be removed from main route table. Note that the gateway will not be removed from the route table '1'. This is so we can continue try ping via eth0 so we can detect that the ISP is back online. When ISP starts working again, the gateways will be added back to main route table again.

Load Balancing

It is possible to give the impression of load balancing by using multipath routes. It will not let a http file download use both ISPs for double download speed but it will let some users surf via ISP1 and others via ISP2, resulting that bandwith usage is spread over both ISPs.

Note that balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider. It also means that the source IP might change so websites that requires login (webmail, banks etc) will break if they check if the clients ip is consistent. Banking sites often do this. It is recommended to avoid this feature if possible.

To enable load balance, simply add the 'load-balance' keyword to all the interfaces that should be balanced in /etc/pingu/pingu.conf:

interface eth0 {
  ...
  load-balance
}

interface eth3 {
  ...
  load-balance
}

...