Let's assume that a firewall is connected via two separate Ethernet interfaces to two different ISPs.^[1] as in the following diagram.

Each of these providers is described in an entry in the file /etc/shorewall/providers.

Entries in /etc/shorewall/providers can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in /etc/shorewall/mangle and /etc/shorewall/rtrules can be used to direct particular outgoing connections to one ISP or the other. Use of /etc/shorewall/mangle (or /etc/shorewall/tcrules) is not required for /etc/shorewall/providers to work, but in most cases, you must select a unique MARK value for each provider so Shorewall can set up the correct marking rules for you.

When you use the track option in /etc/shorewall/providers, connections from the Internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall.

Shorewall will set up the routing and will update the /etc/iproute2/rt_tables to include the table names and numbers of the tables that it adds.

The /etc/shorewall/providers file can also be used in other routing scenarios. See the Squid documentation for an example.

Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value.

Adding another entry in the providers file simply creates an alternate routing table for you (see the LARTC Howto). The table will usually contain two routes:

Note that the first route is omitted if "-" is specified as the GATEWAY; in that case, the default route does not specify a gateway (point-to-point link).

If the DUPLICATE column is non-empty, then routes from the table named in that column are copied into the new table. By default, all routes (except default routes) are copied. The set of routes copied can be restricted using the COPY column which lists the interfaces whose routes you want copied. You will generally want to include all local interfaces in this list. You should exclude the loopback interface (lo) and any interfaces that do not have an IP configuration. You should also omit interfaces like tun interfaces that are created dynamically. Traffic to networks handled by those interfaces should be routed through the main table using entries in /etc/shorewall/rtrules (see Example 2 below) or by using USE_DEFAULT_RT=Yes.

In addition:

That's all that these entries do. You still have to follow the principle stated in the Shorewall Routing documentation:

The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic with the provider's MARK value in /etc/shorewall/mangle and you must do that marking in the PREROUTING chain; or, you must provide the appropriate rules in /etc/shorewall/rtrules.

Shorewall itself provides no mechanism for dealing with provider links that are in the up state but not responsive. If you want transparent failover when a link is unresponsive, you must configure all provider interfaces as optional (shorewall-interfaces(5)) then install and configure LSM.

Shorewall-init provides for handling links that go hard down and are later brought back up.

If you masquerade a local network, you will need to add masquerade rules for both external interfaces. Referring to the diagram above, if each of the interfaces has only a single IP address and you have no systems with public IP addresses behind your firewall, then I suggest the following simple entries:

#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27

If you have a public subnet (for example 206.124.146.176/30) behind your firewall, then use exclusion:

#INTERFACE       SOURCE               ADDRESS
eth0             !206.124.146.176/29  206.124.146.176
eth1             0.0.0.0/0            130.252.99.27

Note that exclusion is only used on the interface corresponding to internal subnetwork.

If you have multiple IP addresses on one of your interfaces, you can use a similar technique -- simplY exclude the smallest network that contains all of those addresses from being masqueraded.

One problem that often arises with Multi-ISP configuration is 'Martians'. If you set ROUTE_FILTER=Yes in /etc/shorewall/shorewall.conf or if your Internet interfaces are configured with the routefilter option in /etc/shorewall/interfaces (remember that if you set that option, you should also select logmartians), then things may not work correctly and you will see messages like this:

Feb  9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 
Feb  9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00

The above message is somewhat awkwardly phrased. The source IP in this incoming packet was 64.86.88.116 and the destination IP address was 206.124.146.176. Another gotcha is that the incoming packet has already had the destination IP address changed for DNAT or because the original outgoing connection was altered by an entry in /etc/shorewall/masq (SNAT or Masquerade). So the destination IP address (206.124.146.176) may not have been the destination IP address in the packet as it was initially received.

There a couple of common causes for these problems:

If all else fails, remove the routefilter option from your external interfaces. If you do this, you may wish to add rules to log and drop packets from the Internet that have source addresses in your local networks. For example, if the local LAN in the above diagram is 192.168.1.0/24, then you would add this rule:

#ACTION          SOURCE                     DEST
DROP:info        net:192.168.1.0/24         all

Be sure the above rule is added before any other rules with net in the SOURCE column.

This section describes the legacy method of configuring multiple uplinks. It is deprecated in favor of the USE_DEFAULT_RT=Yes configuration described below.

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers as follows.

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2

Other configuration files go something like this:

/etc/shorewall/interfaces:

#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …

/etc/shorewall/policy:

#SOURCE    DESTINATION    POLICY     LIMIT:BURST
net        net            DROP

/etc/shorewall/masq:

#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27

This continues the example in the preceding section.

Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. If you are running Shorewall 4.6.0 or later, you would make this entry in /etc/shorewall/mangle.

#ACTION         SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
MARK(2):P       <local network> 0.0.0.0/0       tcp     25

Note that traffic from the firewall itself must be handled in a different rule:

#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
MARK(2)         $FW             0.0.0.0/0       tcp     25

If you are running a Shorewall version earlier than 4.6.0, the above rules in /etc/shorewall/tcrules would be:

#ACTION         SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2:P             <local network> 0.0.0.0/0       tcp     25

And for traffic from the firewall:

#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2               $FW             0.0.0.0/0       tcp     25

Shorewall provides considerable flexibility for port forwarding in a multi-ISP environment.

Normal port forwarding rules such as the following will forward from both providers.

/etc/shorewall/rules:

#ACTION        SOURCE             DEST              PROTO     DEST PORT(S)     SOURCE       ORIGINAL
#                                                                              PORTS(S)     DEST
DNAT           net                loc:192.168.1.3   tcp       25

Continuing the above example, to forward only connection requests from ISP 1, you can either:

When there are more than two providers, you need to extend the two-provider case in the expected way:

If we extend the above example to add eth3 with IP address 16.105.78.4 with gateway 16.105.78.254, then:

/etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2
ISP3    3       3       main            eth3            16.105.78.254   track,balance    eth2

/etc/shorewall/masq:

#INTERFACE       SUBNET            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27
eth3             0.0.0.0/0         16.105.78.4

The rtrules file allows assigning certain traffic to a particular provider just as entries in the mangle file. The difference between the two files is that entries in rtrules are independent of Netfilter.

gateway:~ # ip rule ls
0:      from all lookup local                <=== Local (to the firewall) IP addresses
10001:  from all fwmark 0x1 lookup Blarg     <=== This and the next rule are generated by the
10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
20000:  from 206.124.146.176 lookup Blarg    <=== This and the next rule are generated unless
20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
32766:  from all lookup main                 <=== This is the routing table shown by 'iproute -n'
32767:  from all lookup default              <=== This table is usually empty
gateway:~ #

#SOURCE            DEST      PROVIDER        PRIORITY
eth1               -         Comcast         1000

gateway:~ # ip rule ls
0:      from all lookup local
1000:   from all iif eth1 lookup Comcast
10001:  from all fwmark 0x1 lookup Blarg
10002:  from all fwmark 0x2 lookup Comcast
20000:  from 206.124.146.176 lookup Blarg
20256:  from 24.12.22.33 lookup Comcast
32766:  from all lookup main
32767:  from all lookup default
gateway:~ #

#SOURCE                 DEST            PROVIDER        PRIORITY
-                       10.8.0.0/24     main            1000

As noted above, separate entries in /etc/shorewall/mangle are required for traffic originating from the firewall.

Experience has shown that in some cases, problems occur with applications running on the firewall itself. This is especially true when you have specified routefilter on your external interfaces in /etc/shorewall/interfaces (see above). When this happens, it is suggested that you have the application use specific local IP addresses rather than 0.

Examples:

Note that some traffic originating on the firewall doesn't have a SOURCE IP address before routing. At least one Shorewall user reports that an entry in /etc/shorewall/rtrules with 'lo' in the SOURCE column seems to be the most reliable way to direct such traffic to a particular ISP.

Example:

#SOURCE     DEST      PROVIDER        PRIORITY
lo          -         shorewall       1000

Beginning with Shorewall 4.4.15, additional routes can be added to the provider routing tables using the /etc/shorewall/routes file.

The columns in the file are as follows.

Assume the following entry in /etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
Comcast      1     -    xxx             eth2            ....     

The following table gives some example entries in the file and the ip route command which results.

#PROVIDER     DEST             GATEWAY         DEVICE        |              Generated Command
Comcast       172.20.1.0/24    -               eth0          | ip -4 route add 172.20.1.0/24 dev eth0 table 1
Comcast       192.168.4.0/24   172.20.1.1                    | ip -4 route add 192.168.1.0/24 via 172.20.1.1 table 1
Comcast       192.168.4.0/24                                 | ip -4 route add 192.168.4.0/24 dev eth2 table 1 

Null routing is a type of routing which discards a given packet instead of directing it through a specific predefined route. Generally speaking, there are 3 different types of Null routing as indicated below:

To look at the various routing tables, you must use the ip utility. To see the entire routing configuration (including rules), the command is shorewall show routing. To look at an individual provider's table use ip route ls table provider where provider can be either the provider name or number.

Example:

lillycat:- #ip route ls
144.77.167.142 dev ppp0  proto kernel  scope link  src 144.177.121.199
71.190.227.208 dev ppp1  proto kernel  scope link  src 71.24.88.151
192.168.7.254 dev eth1  scope link  src 192.168.7.1
192.168.7.253 dev eth1  scope link  src 192.168.7.1
192.168.7.0/24 dev eth1  proto kernel  scope link  src 192.168.7.1
192.168.5.0/24 via 192.168.4.2 dev eth0
192.168.4.0/24 dev eth0  proto kernel  scope link  src 192.168.4.223
192.168.1.0/24 via 192.168.4.222 dev eth0
default
        nexthop dev ppp1 weight 2
        nexthop dev ppp0 weight 1
lillycat: #ip route ls table 1
144.77.167.142 dev ppp0  proto kernel  scope link  src 144.177.121.199 
192.168.5.0/24 via 192.168.4.2 dev eth0 
192.168.4.0/24 dev eth0  proto kernel  scope link  src 192.168.4.223 
192.168.1.0/24 via 192.168.4.222 dev eth0 
default dev ppp0  scope link 
lillycat: #

USE_DEFAULT_RT is an option in shorewall.conf (5).

One of the drawbacks of the Multi-ISP support as described in the preceding sections is that changes to the main table made by applications are not added to the individual provider tables. This makes route rules such as described in one of the examples above necessary.

USE_DEFAULT_RT=Yes works around that problem by passing packets through the main table first rather than last. This has a number of implications:

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers as follows.

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       -               eth0            206.124.146.254 track            -
ISP2    2       2       -               eth1            130.252.99.254  track            -  

The remainder of the example is the same.

Although 'balance' is automatically assumed when USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider except when you explicitly direct it to use the other provider via shorewall-rtrules (5) or shorewall-mangle (5).

Example (send all traffic through the 'shorewall' provider unless otherwise directed).

/etc/shorewall/providers:

#NAME    NUMBER MARK DUPLICATE INTERFACE GATEWAY       OPTIONS
linksys       1    1    -        wlan0   172.20.1.1    track,balance=1,optional
shorewall     2    2    -        eth0    192.168.1.254 track,balance=2,optional

/etc/shorewall/rtrules:

#SOURCE     DEST      PROVIDER        PRIORITY
-           -         shorewall       11999

Tuomo Soini describes the following issue when using USE_DEFAULT_RT=Yes.

He has a /27 network (let.s call it 70.90.191.0/27) from his primary ISP and his secondary ISP supplies him with a dynamic IP address on the 91.156.0.0/19 network. From the output of shorewall show routing:

999:    from all lookup main 
10000:  from all fwmark 0x100 lookup ISP1 
10001:  from all fwmark 0x200 lookup ISP2

Note that the main routing table is consulted prior to the marks for his two provlders. When clients in the large /19 network connected to his /27 (through ISP1), the responses were routed out of the ISP2 interface because the main routing table included a route to the /19.

The solution was to add an additional entry to rtrules:

#SOURCE             DEST           PROVIDER       PRIORITY
70.90.191.0/27      91.156.0.0/19  ISP1           900

With this additional entry, the routing rules are as below and traffic from the /27 is returned via ISP1.

900:    from 70.90.191.0/27 to 91.156.0.0/19 lookup ISP1 
999:    from all lookup main 
10000:  from all fwmark 0x100 lookup ISP1 
10001:  from all fwmark 0x200 lookup ISP2

# Config file for dhcpcd. Note that you have to edit the interface
# name below, or duplicate the configuration for different interfaces.
# If you are editing this file just to get DNS servers set by DHCP,
# then you should consider installing the resolvconf package instead.

case ${INTERFACE} in
eth0) 

# Uncomment this to allow dhcpcd to set the DNS servers in /etc/resolv.conf
# If you are using resolvconf then you can leave this commented out.
#SET_DNS='yes'

# Uncomment this to allow dhcpcd to set hostname of the host to the
# hostname option supplied by DHCP server.
#SET_HOSTNAME='yes'

# Uncomment this to allow dhcpcd to set the NTP servers in /etc/ntp.conf
#SET_NTP='yes'

# Uncomment this to allow dhcpcd to set the YP servers in /etc/yp.conf
#SET_YP='yes'

# Add other options here, see man 8 dhcpcd-bin for details.
OPTIONS=(--nogateway --nodns --nontp --script /etc/shorewall/dhcpcd.sh)
;;

# Add other interfaces here
*)
;;

esac

cat <<EOF > /var/lib/shorewall/eth0.info
ETH0_GATEWAY=$SW_ETH0_GATEWAY
ETH0_ADDRESS=$SW_ETH0_ADDRESS
EOF

#!/bin/sh

if [ $2 != down ]; then
    if [ -f /var/lib/dhcpcd/dhcpcd-eth0.info ]; then
        . /var/lib/dhcpcd/dhcpcd-eth0.info
    else
        logger -p daemon.err "/var/lib/dhcpcd/dhcpcd-eth0.info does not exist!"
        exit 1
    fi

    logger -p daemon.info "DHCP-assigned address/gateway for eth0 is $IPADDR/$GATEWAYS"

    [ -f /var/lib/shorewall/eth0.info ] && . /var/lib/shorewall/eth0.info
    
    if [ "$GATEWAYS" != "$ETH0_GATEWAY" -o "$IPADDR" != "$ETH0_ADDRESS" ]; then
        logger -p daemon.info "eth0 IP configuration changed - restarting lsm and Shorewall"
        killall lsm
        /sbin/shorewall restart
    fi
fi

BOOTPROTO=dhcp
PERSISTENT_DHCLIENT=yes
PEERDNS=no
PEERNTP=no
DEFROUTE=no
DHCLIENTARGS="-nc"
DEVICE=eth2
ONBOOT=yes

Beginning with Shorewall 4.5.0, an alternative to the balance=weight option in shorewall-providers (5) is available in the form of a PROBABILITY column in shorewall-mangle(5) (shorewall-tcrules) (5). This feature requires the Statistic Match capability in your iptables and kernel.

This method works when there are multiple links to the same ISP where both links have the same default gateway.

The key features of this method are:

Here's an example that sends 1/3 of the connections through provider ComcastC and the rest through ComastB.

/etc/shorewall/shorewall.conf:

MARK_IN_FORWARD_CHAIN=No
...
USE_DEFAULT_RT=Yes
...
TC_BITS=0
PROVIDER_BITS=2
PROVIDER_OFFSET=16
MASK_BITS=8
ZONE_BITS=4

/etc/shorewall/providers:

#NAME    NUMBER MARK DUPLICATE  INTERFACE GATEWAY       OPTIONS
ComcastB 1      -    -          eth1      70.90.191.126 loose,balance,load=0.66666667
ComcastC 2      -    -          eth0      detect        loose,fallback,load=0.33333333

/etc/shorewall/rtrules:

#SOURCE             DEST  PROVIDER  PRIORITY
70.90.191.120/29    -     ComcastB  1000
&eth0               -     ComcastC  1000

There is an option (LSM) available for monitoring the status of provider links and taking action when a failure occurs. LSM assumes that each provider has a unique nexthop gateway.

You specify the optional option in /etc/shorewall/interfaces:

#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          optional         
net      eth1         detect          optional

EXT_IF=eth0
COM_IF=eth1

local status=0
#
# Read the status file (if any) created by /etc/lsm/script
#
[ -f ${VARDIR}/${1}.status ] && status=$(cat ${VARDIR}/${1}.status)

return $status

###############################################################################
# Create /etc/lsm/shorewall.conf
# Remove the current interface status files
# Start lsm
###############################################################################
start_lsm() {
   #
   # Kill any existing lsm process(es)
   #
   killall lsm 2> /dev/null
   #
   # Create the Shorewall-specific part of the LSM configuration. This file is
   # included by /etc/lsm/lsm.conf
   #
   # Avvanta has a static gateway while Comcast's is dynamic
   #
   cat <<EOF > /etc/lsm/shorewall.conf
connection {
    name=Avvanta
    checkip=206.124.146.254
    device=$EXT_IF
    ttl=2
}

connection {
    name=Comcast
    checkip=${SW_ETH0_GATEWAY:-71.231.152.1}
    device=$COM_IF
    ttl=1
}
EOF
   #
   # Since LSM assumes that interfaces start in the 'up' state, remove any
   # existing status files that might have an interface in the down state
   #
   rm -f /var/lib/shorewall/*.status
   #
   # Run LSM -- by default, it forks into the background
   #
   /usr/sbin/lsm /etc/lsm/lsm.conf >> /var/log/lsm
}

##################################################################################
# [re]start lsm if this is a 'start' command or if lsm isn't running
##################################################################################
if [ "$COMMAND" = start -o -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
    start_lsm
fi

##################################################################################
# Start lsm if it isn't running
##################################################################################
if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
   start_lsm
fi

#
# Defaults for the connection entries
#
defaults {
  name=defaults
  checkip=127.0.0.1
  eventscript=/etc/lsm/script
  max_packet_loss=20
  max_successive_pkts_lost=7
  min_packet_loss=5
  min_successive_pkts_rcvd=10
  interval_ms=2000
  timeout_ms=2000
  warn_email=you@yourdomain.com
  check_arp=0
  sourceip=
  ttl=0
}

include /etc/lsm/shorewall.conf

#!/bin/sh
#
# (C) 2009 Mika Ilmaranta <ilmis@nullnet.fi>
# (C) 2009 Tom Eastep <teastep@shorewall.net>
#
# License: GPLv2
#

STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}

if [ -f /usr/share/shorewall-lite/lib.base ]; then
    VARDIR=/var/lib/shorewall-lite
    STATEDIR=/etc/shorewall-lite
    TOOL=/sbin/shorewall-lite
else
    VARDIR=/var/lib/shorewall
    STATEDIR=/etc/shorewall
    TOOL=/sbin/shorewall
fi

[ -f ${STATEDIR}/vardir ] && . ${STATEDIR}/vardir

cat <<EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}

Hi,

Connection ${NAME} is now ${STATE}.

Following parameters were passed:
newstate     = ${STATE}
name         = ${NAME}
checkip      = ${CHECKIP}
device       = ${DEVICE}
warn_email   = ${WARN_EMAIL}

Packet counters:
replied      = ${REPLIED} packets replied
waiting      = ${WAITING} packets waiting for reply
timeout      = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late   = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd    = ${CONS_RCVD} consecutively received replies in sequence
cons_wait    = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss    = ${CONS_MISS} consecutive packets that have timed out
avg_rtt      = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this

Your LSM Daemon

EOM

if [ ${STATE} = up ]; then
# echo 0 > ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier
  ${VARDIR}/firewall enable ${DEVICE}
else
#  echo 1 > ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier
   ${VARDIR}/firewall disable ${DEVICE}
fi

$TOOL show routing >> /var/log/lsm

exit 0

#EOF

#!/bin/sh
#
# (C) 2009 Mika Ilmaranta <ilmis@nullnet.fi>
# (C) 2009 Tom Eastep <teastep@shorewall.net>
#
# License: GPLv2
#

STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}

if [ -f /usr/share/shorewall-lite/lib.base ]; then
    VARDIR=/var/lib/shorewall-lite
    STATEDIR=/etc/shorewall-lite
    TOOL=/sbin/shorewall-lite
else
    VARDIR=/var/lib/shorewall
    STATEDIR=/etc/shorewall
    TOOL=/sbin/shorewall
fi

[ -f ${STATEDIR}/vardir ] && . ${STATEDIR}/vardir

cat <<EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}

Hi,

Connection ${NAME} is now ${STATE}.

Following parameters were passed:
newstate     = ${STATE}
name         = ${NAME}
checkip      = ${CHECKIP}
device       = ${DEVICE}
warn_email   = ${WARN_EMAIL}

Packet counters:
replied      = ${REPLIED} packets replied
waiting      = ${WAITING} packets waiting for reply
timeout      = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late   = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd    = ${CONS_RCVD} consecutively received replies in sequence
cons_wait    = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss    = ${CONS_MISS} consecutive packets that have timed out
avg_rtt      = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this

Your LSM Daemon

EOM

# Uncomment the next two lines if you are running Shorewall 4.4.x or earlier

# [ ${STATE} = up ] && state=0 || state=1
# echo $state > ${VARDIR}/${DEVICE}.status

$TOOL restart -f >> /var/log/lsm 2>&1

$TOOL show routing >> /var/log/lsm

exit 0

#EOF

Shared interface support has the following characteristics:

Taken together, b. and h. effectively preclude using this technique with dynamic IP addresses.

Example:

This is our home network circa fall 2008. We have two Internet providers:

Because the old Compaq™ Presario™ that I use for a firewall only has three PCI slots and no onboard Ethernet, it doesn't have enough Ethernet controllers to support both providers. So I use a Linksys WRT300n pre-N router as a gateway to Comcast. Note that because the Comcast IP address is dynamic, I could not share a single firewall interface between the two providers directly.

On my personal laptop (ursa), I have 9 virtual machines running various Linux distributions. It is the Shorewall configuration on ursa that I will describe here.

Below is a diagram of our network:

The local wired network in my office is connected to both gateways and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast gateway has local IP address 172.20.1.1 while the Avvanta gateway has local IP address 172.20.1.254. Ursa's eth0 interface has a single IP address (172.20.1.130).

This configuration uses USE_DEFAULT_RT=Yes in shorewall.conf (see above).

Here is the providers file:

#NAME          NUMBER   MARK DUPLICATE INTERFACE            GATEWAY      OPTIONS                            COPY
comcast        1        1    -         eth0:172.20.1.130    172.20.1.1   track,loose,balance,optional
avvanta        2        2    -         eth0:172.20.1.130    172.20.1.254 track,optional,loose
wireless       3        3    -         wlan0                172.20.1.1   track,optional

Several things to note:

Here is the rtrules file:

#SOURCE                 DEST                    PROVIDER        PRIORITY
-                       206.124.146.176/31      avvanta         1000
-                       206.124.146.178/31      avvanta         1000
-                       206.124.146.180/32      avvanta         1000

Those rules direct traffic to the five static Avvanta IP addresses (only two are currently used) through the avvanta provider.

Here is the mangle file (MARK_IN_FORWARD_CHAIN=No in shorewall.conf):

#ACTION               SOURCE          DEST            PROTO   DEST            SOURCE  USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
#                                                             PORT(S)         PORT(S) 
MARK(2)               $FW             0.0.0.0/0       tcp     21
MARK(2)               $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
MARK(2)               $FW             0.0.0.0/0       tcp     119

If you are still using a tcrules file, you should consider switching to using a mangle file (shorewall update -t will do that for you). Here are the equivalent tcrules entries:

#MARK           SOURCE          DEST            PROTO   PORT(S)         CLIENT  USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
#                                                                       PORT(S) 
2               $FW             0.0.0.0/0       tcp     21
2               $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
2               $FW             0.0.0.0/0       tcp     119

These rules:

The same rules converted to use the mangle file are:

#ACTION         SOURCE          DEST            PROTO   PORT(S)         CLIENT  USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
#                                                                       PORT(S) 
MARK(2)         $FW             0.0.0.0/0       tcp     21
MARK(2)         $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
MARK(2)         $FW             0.0.0.0/0       tcp     119

The remaining files are for a rather standard two-interface config with a bridge as the local interface.

zones:

#ZONE   IPSEC   OPTIONS                 IN                      OUT
#       ONLY                            OPTIONS                 OPTIONS
fw      firewall
net     ipv4
kvm     ipv4

policy:

net             net             NONE
fw              net             ACCEPT
fw              kvm             ACCEPT
kvm             all             ACCEPT
net             all             DROP            info
all             all             REJECT          info

interfaces:

#ZONE    INTERFACE      BROADCAST       OPTIONS                 GATEWAY
#
net     eth0            detect          dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore
net     wlan0           detect          dhcp,tcpflags,routefilter,blacklist,logmartians,optional
kvm     br0             detect          routeback       #Virtual Machines

masq:

#INTERFACE              SUBNET          ADDRESS         PROTO   PORT(S) IPSEC
eth0                    192.168.0.0/24
wlan0                   192.168.0.0/24

The Business Gateway manages a gigabit local network with address 10.0.1.1/24. So The firewall is given address 10.0.1.11/24 and the gateway is configured to route the public IP block via that address. The gateway's firewall is only enabled for the 10.0.1.0/24 network.

Because the business network is faster and more reliable, the configuration favors sending local network traffic via that uplink rather than the consumer line.

Here are the key entries in /etc/shorewall/params:

LOG=NFLOG

INT_IF=eth2
TUN_IF=tun+
COMB_IF=eth1
COMC_IF=eth0

STATISTICAL=
PROXY=
FALLBACK=
PROXYDMZ=
SQUID2=

The last five variables are used to configure the firewall differently to exercise various Shorewall features. Their use requires Shorewall 4.5.2 or later.

Here are the key entries in /etc/shorewall/shorewall.conf:

###############################################################################
#                       F I R E W A L L   O P T I O N S
###############################################################################

...

ACCOUNTING_TABLE=mangle

...

AUTOMAKE=Yes

BLACKLISTNEWONLY=Yes

...

EXPAND_POLICIES=No

EXPORTMODULES=Yes

FASTACCEPT=No

..

KEEP_RT_TABLES=Yes #This is necessary when both IPv4 and IPv6 Multi-ISP are used

LEGACY_FASTSTART=Yes

LOAD_HELPERS_ONLY=Yes

...

MARK_IN_FORWARD_CHAIN=No

MODULE_SUFFIX=ko

MULTICAST=No

MUTEX_TIMEOUT=60

NULL_ROUTE_RFC1918=Yes

OPTIMIZE=31

OPTIMIZE_ACCOUNTING=No

REQUIRE_INTERFACE=No

RESTORE_DEFAULT_ROUTE=No

RETAIN_ALIASES=No

ROUTE_FILTER=No

SAVE_IPSETS=

TC_ENABLED=No

TC_EXPERT=No

TC_PRIOMAP="2 3 3 3 2 3 1 1 2 2 2 2 2 2 2 2"

TRACK_PROVIDERS=Yes

USE_DEFAULT_RT=Yes

USE_PHYSICAL_NAMES=Yes

ZONE2ZONE=-

################################################################################
#                       P A C K E T  M A R K  L A Y O U T
################################################################################

TC_BITS=8

PROVIDER_BITS=2

PROVIDER_OFFSET=16

MASK_BITS=8

ZONE_BITS=0

I use USE_DEFAULT_RT=Yes and since there are only two providers, two provider bits are all that are required.

Here is /etc/shorewall/zones:

fw              firewall
loc             ip           #Local Zone
net             ip           #Internet
smc:net         ip           #10.0.1.0/24
vpn             ip           #OpenVPN clients
dmz             ip           #LXC Containers

/etc/shorewall/interfaces:

#ZONE  INTERFACE   OPTIONS
loc    INT_IF           dhcp,physical=$INT_IF,ignore=1,wait=5,routefilter,nets=172.20.1.0/24,routeback
net    COMB_IF          optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMB_IF,upnp,nosmurfs,tcpflags
net    COMC_IF          optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMC_IF,upnp,nosmurfs,tcpflags,dhcp
vpn    TUN_IF+          physical=tun+,ignore=1
dmz    br0              routeback,proxyarp=1,required,wait=30

/etc/shorewall/hosts:

#ZONE   HOST(S)                                 OPTIONS
smc     COMB_IF:10.1.10.0/24
smc     COMC_IF:10.0.0.0/24

/etc/shorewall/providers:

#NAME             NUMBER   MARK    DUPLICATE  INTERFACE	GATEWAY         OPTIONS               COPY
?if $FALLBACK
ComcastB          1        0x10000 -          COMB_IF     70.90.191.126 loose,fallback
ComcastC          2        0x20000 -          COMC_IF     detect        loose,fallback
?elsif $STATISTICAL
ComcastB          1        0x10000 -          COMB_IF     70.90.191.126 loose,load=0.66666667
ComcastC          2        0x20000 -          COMC_IF     detect        loose,load=0.33333333
?else
ComcastB          1        0x10000 -          COMB_IF     70.90.191.126 loose,balance=2
ComcastC          2        0x20000 -          COMC_IF     detect        loose,balance
?endif
?if $PROXY && ! $SQUID2
Squid             3        -       -          lo          -             tproxy 
?endif

Notice that in the current balance mode, as in the STATISTICAL mode, the business line is favored 2:1 over the consumer line.

Here is /etc/shorewall/rtrules:

#SOURCE             DEST             PROVIDER  PRIORITY
70.90.191.121       -                ComcastB  1000
70.90.191.123       -                ComcastB  1000
&COMC_IF            -                ComcastC  1000
br0                 -                ComcastB  11000
172.20.1.191        -                ComcastB  1000

For reference, this configuration generates these routing rules:

root@gateway:~# ip rule ls
0:      from all lookup local 
1:      from all fwmark 0x80000/0x80000 lookup TProxy 
999:    from all lookup main 
1000:   from 70.90.191.121 lookup ComcastB 
1000:   from 70.90.191.123 lookup ComcastB 
1000:   from 172.20.1.191 lookup ComcastB 
1000:   from 10.0.0.4 lookup ComcastC 
10000:  from all fwmark 0x10000/0x30000 lookup ComcastB 
10001:  from all fwmark 0x20000/0x30000 lookup ComcastC 
11000:  from all iif br0 lookup ComcastB 
32765:  from all lookup balance 
32767:  from all lookup default 
root@gateway:~# 

/etc/shorewall/mangle is not used to support Multi-ISP:

#MARK                           SOURCE        DEST          PROTO  DEST    SOURCE  
#                                                                  PORT(S) PORT(S)
FORMAT 2
TTL(+1):P                       INT_IF        -
SAME:P                          INT_IF        -             tcp    80,443
?if $PROXY && ! $SQUID2
   DIVERT                       COMB_IF       -             tcp    -       80
   DIVERT                       COMC_IF       -             tcp    -       80
   DIVERT                       br0           172.20.1.0/24 tcp    -       80
   TPROXY(3129,172.20.1.254)    INT_IF        -             tcp    80
   ?if $PROXYDMZ
      TPROXY(3129,172.20.1.254) br0           -             tcp    80
   ?endif
?endif

The IPv6 configuration has two separate sub-nets, both services through 6in4 tunnels from Hurricane Electric. They are both configured through the Business IPv4 uplink. I originally had the sit2 tunnel configured through the consumer uplink but Comcast (Xfinity) decided to start blocking HE IPv6 tunnels on their consumer network, preferring their own 6to4 IPv6 solution.

One HE tunnel handles the servers and one tunnel handles the local network.

Here are the key entries in /etc/shorewall6/shorewall6.conf:

###############################################################################
#                      F I R E W A L L  O P T I O N S
###############################################################################

...

FASTACCEPT=No

FORWARD_CLEAR_MARK=Yes

IMPLICIT_CONTINUE=No

IP_FORWARDING=Keep

KEEP_RT_TABLES=Yes #Required when both IPv4 and IPv6 Multi-ISP are used

...

TRACK_PROVIDERS=No

USE_DEFAULT_RT=Yes

ZONE2ZONE=-

...

################################################################################
#                      P A C K E T  M A R K  L A Y O U T
################################################################################

TC_BITS=8

PROVIDER_BITS=8

PROVIDER_OFFSET=8

MASK_BITS=8

ZONE_BITS=0

Here is /etc/shorewall6/zones:

#ZONE   TYPE    OPTIONS
fw      firewall
net     ipv6
loc     ipv6
dmz     ipv6

/etc/shorewall/interfaces:

#ZONE   INTERFACE       OPTIONS
net     sit1            forward=1,sfilter=2001:470:b:227::40/124,optional
net     sit2            forward=1,sfilter=2001:470:b:227::40/124,optional
net     sit3            forward=1,sfilter=2001:470:b:227::40/124,optional
loc     eth2            forward=1
dmz     br0             routeback,forward=1,required

/etc/shorewall/providers:

#NAME   NUMBER    MARK    DUPLICATE   INTERFACE GATEWAY                OPTIONS            COPY
LOC     4        0x100    -           sit2      -                      track,balance,loose
DMZ     5        0x200    -           sit1      -                      track,fallback,loose
6to4    6        0x300    -           sit3      ::192.88.99.1          track,fallback,loose

Notice that the provider numbers are disjoint from those in the IPv4 configuration. This allows for unique provider names in /etc/iproute2/rt_tables:

#
# reserved values
#
255     local
254     main
253     default
250     balance
0       unspec
#
# local
#
1       ComcastB
2       ComcastC
3       TProxy
4       LOC
5       DMZ
6       6to4

The /etc/shorewall6/rtrules file is straight-forward:

#SOURCE                DEST                     PROVIDER             PRIORITY
2001:470:B:227::1/64   ::/0                     DMZ                  11000
2001:470:B:787::1/64   ::/0                     LOC                  11000
2002:465a:bf79::1/64   ::/0                     6to4                 11000

This results in the following routing rules:

root@gateway:~# ip -6 rule ls
0:     from all lookup local 
999:   from all lookup main 
10003: from all fwmark 0x100/0xff00 lookup LOC 
10004: from all fwmark 0x200/0xff00 lookup DMZ 
10005: from all fwmark 0x300/0xff00 lookup 6to4 
11000: from 2001:470:b:787::1/64 lookup LOC 
11000: from 2001:470:b:227::1/64 lookup DMZ 
11000: from 2002:465a:bf79::1/64 lookup 6to4 
32765: from all lookup balance 
32767: from all lookup default 
root@gateway:~#