Servers in Small Site Multi-homing
by Ivan Pepelnjak
The IP Corner articles Small Site Multi-homing and Redundant Small Site Multi-homing have generated significant feedback, most of it focused on the question “Can I deploy public servers in this design”. The short answer is “You don’t want to” and in most cases, you’d be far better off buying a decent hosting service (and you should use an external Domain Name System server anyway). Highly reliable hosting services cost around $10/month, and one would hope that the time you’d have to spend configuring and troubleshooting the solution described in this article is worth significantly more than that.
However, if you want to deploy local mail server within your LAN (which is a very reasonable proposition) or you have a special application that simply cannot be hosted anywhere else and you’re willing to accept less-than-perfect reliability, this article will describe the challenges you’ll face and give you several design and deployment guidelines.
Note
Even if you’re not interested in deploying public servers in a small site, the Network Address Translation (NAT) issues described in the article will give you valuable insight that you could use in your network design/implementation projects.
Initial Design Summary
Throughout the article we’ll work with a design in which a single small-office router is connected to two Internet Service Providers (ISPs) as displayed in Figure 1. A server (inside server) is connected to the LAN and should be reachable from the public Internet through both ISPs.
Figure 1
IP addressing in small multi-homed site
Static IP addresses belonging to the ISP’s address space are assigned to both upstream links and a single IP address is available on each link. Throughout the article, the IP addresses from Table 1 will be used in router configurations and printouts.
Table 1
IP address allocation in sample network
|
Interface
|
IP address
|
|
Serial 0/1/0 (primary link)
|
192.168.2.1
|
|
Serial 0/2/0 (backup link)
|
172.16.1.1
|
|
LAN interface
|
10.0.0.0/24
|
Reliable static routing is used to select the primary/backup link for the outgoing traffic and route map-based Network Address Translation (NAT) is configured to perform address translation based on selected outgoing interface (Figure 2).
Figure 2
Static routing and address translation
Note
The detailed description of this design and the associated router configurations can be found in the article.
Design and Deployment Challenges
When you want to deploy a redundantly-connected inside server offering public services on the multihomed small site, you have to solve these challenges:
The inside server should be reachable through both public IP addresses configured on the upstream links.
The external client application accessing the inside server should be given all public IP addresses (or the public IP address that is currently reachable) and should try to reach the inside server on all available outside IP addresses.
If an external client reaches the inside server through one of the upstream links, the return traffic has to flow over the same link to ensure that the correct NAT translations are used for the return traffic (see Figure 3).
Figure 3
Return traffic flow
Redundant Server Addresses
To make the inside server reachable through both public interfaces, you have to configure static NAT entries redirecting a single TCP or UDP port on the outside interface to the inside server with the ip inside source static router configuration commands detailed in the Network Layer Configuration section.
Redundant Application Access
This task is easy to accomplish if the application you’re using accepts multiple IP addresses for the server host and is willing to try each one of them. Common applications meeting this requirement are VPN clients (that can try to connect to multiple VPN concentrators) and Internet mail delivery using Simple Mail Transfer Protocol (SMTP) based on Mail Exchange (MX) records in Domain Name System (DNS). Unfortunately, common applications (including web browsers), do not support such a model and require special DNS techniques detailed in the Application Layer Configuration section.
Return Traffic Flow
If you want to ensure that the return packets sent from the inside server to an external client always uses the same upstream link as the incoming packet, you have to deploy policy-based routing (PBR) on the gateway router. PBR should force the packets coming from the inside server to one or the other upstream link regardless of the current state of the IP routing table on the gateway router.
It’s almost impossible for the PBR mechanism to recognize the correct outbound link to use based solely on the return packet. You could try to use different IP precedence settings (relying on the fact that the return packets from a server have to use the same IP precedence settings as the incoming requests), but it’s more reliable to use two server IP addresses to achieve the separation (Figure 4):
Destination address of packets received through the primary link is translated into server’s primary IP address;
Destination address of packets received through the backup link is translated into server’s secondary IP address;
The policy routing of outbound packets matches the source IP address and forces the packet onto the primary link if the source IP address matches server’s primary IP address (and similarly for the backup link).
Figure 4
Policy routing from the inside server to an external client
Note
You should configure two extra IP addresses on the server in addition to the LAN address to ensure that the packets sent from the applications on the server are not policy routed and can reach Internet destinations even if one of the upstream links is unavailable.
Network-Layer Configuration
The network-layer configuration should start on the server side: you have to create two loopback interfaces on the server (only the Windows-specific configuration is covered in this section) and assign them static IP addresses.
Windows Server Configuration
If you’re using a Windows-based server, follow the instructions in the . After both interfaces have been created, open the Network connections control panel, rename both loopback interfaces to Loopback_x (solely for documentation purposes) and assign static IP addresses to them.
Note
Windows XP does not allow you to configure a /32 subnet mask on the loopback interface; you have to use a /30 mask.
The final network configuration displayed in the control panel should resemble the Figure 5 and the ipconfig command should display all the interfaces addresses as shown in Listing 1.
Figure 5
Network connections on the inside Windows server
Listing 1
IP interface configuration on the Windows server
Server$ ipconfig
Windows IP Configuration
Ethernet adapter Loopback_2:
Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 10.0.1.5
Subnet Mask . . . . . . . . . . . : 255.255.255.252
Default Gateway . . . . . . . . . :
Ethernet adapter Loopback_1:
Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 10.0.1.1
Subnet Mask . . . . . . . . . . . : 255.255.255.252
Default Gateway . . . . . . . . . :
Ethernet adapter LAB:
Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 10.0.0.10
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.0.0.1
The IP routing table on the server should contain entries for all directly connected subnets as well as the default route toward the gateway router (Listing 2).
Listing 2
Routing table on the inside server
Server$ netstat -rn
Route Table
=======================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...02 00 4c 4f 4f 50 ...... Microsoft Loopback Adapter #2
0x3 ...02 00 4c 4f 4f 50 ...... Microsoft Loopback Adapter
0x10006 ...00 0c 29 a4 c7 ef ...... AMD PCNET Family PCI Ethernet Adapter - Packet Scheduler Miniport
=======================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 10.0.0.1 10.0.0.10 1
10.0.0.0 255.255.255.0 10.0.0.10 10.0.0.10 10
10.0.0.10 255.255.255.255 127.0.0.1 127.0.0.1 10
10.0.1.0 255.255.255.252 10.0.1.1 10.0.1.1 30
10.0.1.1 255.255.255.255 127.0.0.1 127.0.0.1 30
10.0.1.4 255.255.255.252 10.0.1.5 10.0.1.5 30
10.0.1.5 255.255.255.255 127.0.0.1 127.0.0.1 30
10.255.255.255 255.255.255.255 10.0.0.10 10.0.0.10 10
10.255.255.255 255.255.255.255 10.0.1.1 10.0.1.1 30
10.255.255.255 255.255.255.255 10.0.1.5 10.0.1.5 30
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 240.0.0.0 10.0.0.10 10.0.0.10 10
224.0.0.0 240.0.0.0 10.0.1.1 10.0.1.1 30
224.0.0.0 240.0.0.0 10.0.1.5 10.0.1.5 30
255.255.255.255 255.255.255.255 10.0.0.10 10.0.0.10 1
255.255.255.255 255.255.255.255 10.0.1.1 10.0.1.1 1
255.255.255.255 255.255.255.255 10.0.1.5 10.0.1.5 1
Default Gateway: 10.0.0.1
=======================================================================
Persistent Routes:
None
Router Configuration
On the router’s side, you should configure a static route for the address space used by the server’s loopbacks and multiple ip nat inside source static commands to translate outside public IP addresses into multiple private IP addresses configured on the server (Listing 3). You could even define multiple inside servers (or multiple applications running on the inside server), each one of them operating on a different public TCP or UDP port(s).
Note
The reference to interface names (not the IP addresses) in the ip nat statements makes the router configuration more resilient to potential changes made by the ISPs.
Listing 3
Static routes and static NAT entries configured on the gateway router
ip route 10.0.1.0 255.255.255.248 10.0.0.10
!
ip nat inside source static tcp 10.0.1.1 80 interface Serial0/1/0 80
ip nat inside source static tcp 10.0.1.5 80 interface Serial0/2/0 80
The policy routing needed for the return traffic from the server’s loopback interfaces, requires two access lists (each one matching a single source IP address) and a route map which forces the return traffic to one or the other upstream link (Listing 4). Both route-map entries include a fallback interface (null 0) ensuring that the return packets are dropped if the link through which they should be sent is not available.
Listing 4
Policy routing configuration
interface FastEthernet0/0
ip address 10.0.0.1 255.255.255.0
ip nat inside
ip policy route-map LocalPolicy
!
ip access-list extended From_10.0.1.1
permit ip host 10.0.1.1 any
ip access-list extended From_10.0.1.5
permit ip host 10.0.1.5 any
!
route-map LocalPolicy permit 10
match ip address From_10.0.1.1
set interface Serial0/1/0 Null0
!
route-map LocalPolicy permit 30
match ip address From_10.0.1.5
set interface Serial0/2/0 Null0
After the network-layer configuration has been completed, you can try accessing your inside server through both public IP addresses. If you want to offer HTTP (web) services on the inside server, you should use IP addresses in URLs (http://172.16.1.1/ and http://192.168.2.1/ in our scenario) to ensure the correct IP address is targeted during the connectivity tests.
Application Layer Configuration
Some applications accept multiple IP addresses for the server host (through manual configuration or via DNS responses) and are willing to try each one of them. Common applications meeting this requirement are VPN clients (that can try to connect to multiple VPN concentrators) and Internet mail delivery using Simple Mail Transfer Protocol (SMTP) based on Mail Exchange (MX) records in Domain Name System (DNS). For example, to deploy a mail server on the inside server, you just need to enter multiple MX records in the DNS zone for your domain (Listing 5).
Listing 5
DNS zone defining mail servers for the example.com domain
@ IN SOA example.com. test.example.com. (
2008050501; serial number YYMMDDNN
28800 ; Refresh
7200 ; Retry
864000 ; Expire
86400 ; Min TTL
)
NSns.my-primary-isp.invalid.
NSns.my-backup-isp.invalid.
$ORIGIN example.com.
MXmail-gw-1.example.com.
MXmail-gw-2.example.com.
mail-gw-1A192.168.2.1
mail-gw-2A172.16.1.1
Unfortunately, common applications (including web browsers), do not support such a model. These applications would select the first IP address returned in the DNS reply (or round-robin between them), try it and report a failure if the address is not reachable.
Technical Detail
Web browsers are even more problematic; they have to use to prevent the users from spoofing attacks.
Multiple techniques could be used to fix this problem; one of the most straightforward ones is the conversion of the server’s host name (A record) into a DNS zone:
The public DNS data for your inside server’s host name contains no address information (A records) but solely the NS records pointing to your public IP addresses. The DNS zone file for www.example.com is displayed in Listing 6.
Listing 6
DNS records for www.example.com
@ IN SOA example.com. test.example.com. (
2008050501; serial number
28800 ; Refresh
7200 ; Retry
864000 ; Expire
86400 ; Min TTL
)
NSns.primary-isp.invalid.
$ORIGIN example.com.
wwwNSwww-ns1
NSwww-ns2
www-ns1A192.168.2.1
www-ns2A172.16.1.1
Two DNS views are configured on the gateway router, each one of them having a different IP address for the server’s public host name (Listing 7)
Listing 7
DNS views configured on the gateway router
ip dns view Primary
ip dns view Secondary
!
ip dns view-list Primary
view Primary 1
ip dns view-list Secondary
view Secondary 1
!
ip host view Primary www.example.com 192.168.1.2
ip host view Secondary www.example.com 172.16.1.1
A DNS server is started on the router. Public interfaces are placed into different DNS view-groups (Listing 8)
Listing 8
DNS server and DNS view group configurations on the gateway router
ip dns server
!
interface Serial0/1/0
description *** Primary link ***
ip address 192.168.2.1 255.255.255.248
ip dns view-group Primary
ip nat outside
!
interface Serial0/2/0
description *** Backup link ***
ip address 172.16.1.1 255.255.255.252
ip dns view-group Secondary
ip nat outside
Whenever a client tries to access the inside server using the host name www.example.com, its local DNS server selects one of the name servers serving the www.example.com domain and asks about the IP address belonging to www.example.com. If the selected name server is not reachable (due to the uplink being unavailable), the DNS server automatically resends the query to the alternate name server (the other public IP address configured on the router). Due to the DNS views and groups configured on the router, it always replies to the DNS queries for www.example.com with the IP address of the interface through which the query has been received, ensuring that the application client always gets a working IP address. You can easily test this functionality with the program available in the (it’s also available on Windows platforms). The sample results for www.example.com are displayed in the Listing 9.
Listing 9
Dig results for www.example.com
; <<>> DiG 9.4.1 <<>> www.example.com.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1837
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2
;; QUESTION SECTION:
;www.example.com. IN A
;; ANSWER SECTION:
www.example.com. 10 IN A 192.168.1.2
;; AUTHORITY SECTION:
www.example.com. 86400 IN NS www-ns2.example.com.
www.example.com. 86400 IN NS www-ns1.example.com.
;; ADDITIONAL SECTION:
www-ns1.example.com. 86400 IN A 192.168.2.1
www-ns2.example.com. 86400 IN A 172.16.1.1
;; Query time: 60 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue May 06 10:05:45 2008
;; MSG SIZE rcvd: 125
Summary
If you’re willing to accept a complex design and less-than-perfect reliability, you could deploy publicly accessible inside servers on a multihomed small side without having your own public IP address space or running Border Gateway Protocol (BGP) with your ISPs. However, due to the inherent complexity of the solution, you’re advised to use it only for mail servers (where the DNS MX records solve most of the application-layer issues) or special applications that you cannot offload to an external hosting solution. In most cases, you should consider using an external hosting service.
To make an inside server redundantly available through a set of public IP addresses, you have to perform the following tasks:
Configure multiple IP addresses on the server;
Configure static or dynamic routing between the gateway router and the server;
Configure multiple static NAT entries on the gateway router;
Configure policy routing for return (server-to-client) traffic on the gateway router.
If you’re deploying an application where the clients are not willing to try connecting to multiple server’s IP addresses, you should use advanced DNS techniques:
Transform the server’s hostname into a DNS zone served by the gateway router through both public IP addresses;
Configure two DNS views on the router. Each DNS view should have a different IP address associated to the server’s hostname.
Configure the DNS server on the router, placing each of the outside interfaces into a different DNS view.
More to explore: