Different linux distribution vendors put their networking configuration files in different places in the filesystem. Here is a brief summary of the locations of the IP networking configuration information under a few common linux distributions along with links to further documentation.

The format of the networking configuration files differs significantly from distribution to distribution, yet the tools used by these scripts are the same. This documentation will focus on these tools and how they instruct the kernel to alter interface and route information. Consult the distribution's documentation for questions of file format and order of operation.

For the remainder of this document, many examples refer to machines in a hypothetical network. Refer to the example network description for the network map and addressing scheme.

Assuming an already configured machine named tristan, let's look at the IP addressing and routing table. Next we'll examine how the machine communicates with computers (hosts) on the locally reachable network. We'll then send packets through our default gateway to other networks. After learning what a default route is, we'll look at a static route.

One of the first things to learn about a machine attached to an IP network is its IP address. We'll begin by looking at a machine named tristan on the main desktop network (192.168.99.0/24).

The machine tristan is alive on IP 192.168.99.35 and has been properly configured by the system administrator. By examining the route and ifconfig output we can learn a good deal about the network to which tristan is connected ^[1].

[root@tristan]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:80:C8:F8:4A:51  
          inet addr:192.168.99.35  Bcast:192.168.99.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27849718 errors:1 dropped:0 overruns:0 frame:0
          TX packets:29968044 errors:5 dropped:0 overruns:2 carrier:3
          collisions:0 txqueuelen:100 
          RX bytes:943447653 (899.7 Mb)  TX bytes:2599122310 (2478.7 Mb)
          Interrupt:9 Base address:0x1000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:7028982 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7028982 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1206918001 (1151.0 Mb)  TX bytes:1206918001 (1151.0 Mb)

[root@tristan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.99.254  0.0.0.0         UG    0      0        0 eth0
      

For the moment, ignore the loopback interface (lo) and concentrate on the Ethernet interface. Examine the output of the ifconfig command. We can learn a great deal about the IP network to which we are connected simply by reading the ifconfig output. For a thorough discussion of ifconfig, see Section C.1, “ifconfig”.

The IP address active on tristan is 192.168.99.35. This means that any IP packets created by tristan will have a source address of 192.168.99.35. Similarly any packet received by tristan will have the destination address of 192.168.99.35. When creating an outbound packet tristan will set the destination address to the server's IP. This gives the remote host and the networking devices in between these hosts enough information to carry packets between the two devices.

Because tristan will advertise that it accepts packets with a destination address of 192.168.99.35, any frames (packets) appearing on the Ethernet bound for 192.168.99.35 will reach tristan. The process of communicating the ownership of an IP address is called ARP. Read Section 2.1.1, “Overview of Address Resolution Protocol” for a complete discussion of this process.

This is fundamental to IP networking. It is fundamental that a host be able to generate and receive packets on an IP address assigned to it. This IP address is a unique identifier for the machine on the network to which it is connected.

Common traffic to and from machines today is unicast IP traffic. Unicast traffic is essentially a conversation between two hosts. Though there may be routers between them, the two hosts are carrying on a private conversation. Examples of common unicast traffic are protocols such as HTTP (web), SMTP (sending mail), POP3 (fetching mail), IRC (chat), SSH (secure shell), and LDAP (directory access). To participate in any of these kinds of traffic, tristan will send and receive packets on 192.168.99.35.

In contrast to unicast traffic, there is another common IP networking technique called broadcasting. Broadcast traffic is a way of addressing all hosts in a given network range with a single destination IP address. To continue the analogy of the unicast conversation, a broadcast is more like shouting in a room. Occasionally, network administrators will refer to broadcast techniques and broadcasting as "chatty network traffic".

Broadcast techniques are used at the Ethernet layer and the IP layer, so the cautious person talks about Ethernet broadcasts or IP broadcast. Refer to Section 2.1.1, “Overview of Address Resolution Protocol”, for more information on a common use of broadcast Ethernet frames.

IP Broadcast techniques can be used to share information with all partners on a network or to discover characteristics of other members of a network. SMB (Server Message Block) as implemented by Microsoft products and the samba package makes extensive use of broadcasting techniques for discovery and information sharing. Dynamic Host Configuration Protocol (DHCP) also makes use of broadcasting techniques to manage IP addressing.

The IP broadcast address is, usually, correctly derived from the IP address and network mask although it can be easily be set explicitly to a different address. Because the broadcast address is used for autodiscovery (e.g, SMB under some protocols, an incorrect broadcast address can inhibit a machine's ability to participate in networked communication ^[2].

The netmask on the interface should match the netmask in the routing table for the locally connected network. Typically, the route and the IP interface definition are calculated from the same configuration data so they should match perfectly.

If you are at all confused about how to address a network or how to read either the traditional notation or the CIDR notation for network addressing, see one of the CIDR/netmask references in Section I.1.3, “General IP Networking Resources”.

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
      

[root@tristan]# ping -c 1 -n 192.168.99.254
PING 192.168.99.254 (192.168.99.254) from 192.168.99.35 : 56(84) bytes of data.

--- 192.168.99.254 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 192.168.99.254 (192.168.99.254) from 192.168.99.35 : 56(84) bytes of data.
64 bytes from 192.168.99.254: icmp_seq=0 ttl=255 time=238 usec

--- 192.168.99.254 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/mdev = 0.238/0.238/0.238/0.000 ms
        

1.2.2. Sending Packets to Unknown Networks Through the Default Gateway

In Section 1.2.1, “Sending Packets to the Local Network”, we verified that hosts connected to the same local network can reach each other and, importantly, the default gateway. Now, let's see what happens to packets which have a destination address outside the locally connected network.

Assuming that the network administrator allows ping packets from the desktop network into the public network, ping can be invoked with the record route option to show the path the packet travels from tristan to wan-gw and back.

Example 1.3. Testing reachability of non-local hosts

[root@tristan]# ping -R -c 1 -n 205.254.211.254 PING 205.254.211.254 (205.254.211.254) from 192.168.99.35 : 56(84) bytes of data. --- 205.254.211.254 ping statistics --- 1 packets transmitted, 0 packets received, 100% packet loss PING 205.254.211.254 (205.254.211.254) from 192.168.99.35 : 56(84) bytes of data. 64 bytes from 205.254.211.254: icmp_seq=0 ttl=255 time=238 usec RR: 192.168.99.35 205.254.211.179 205.254.211.254 205.254.211.254 192.168.99.254 192.168.99.35 --- 192.168.99.254 ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max/mdev = 0.238/0.238/0.238/0.000 ms

	As the packet passes through the IP stack on tristan, before hitting the Ethernet, tristan adds its IP to the list of IPs in the option field in the header.
	This is masq-gw's public IP address.
	Our intended destination! (Anybody know why there are two entries in the record route output?)
	This is masq-gw's private IP address.
	And finally, tristan will add its IP to the option field in the header of the IP packet just before the packet reaches the calling ping program.

By testing reachability of the local network 192.168.99.0/24 and an IP address outside our local network, we have verified the basic elements of IP connectivity.

To summarize this section, we have:

• identified the IP address, network address and netmask in use on tristan using the tools ifconfig and route

• verified that tristan can reach its default gateway

• tested that packets bound for destinations outside our local network reach the intended destination and return

[root@tristan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.98.0    192.168.99.1    255.255.255.0   UG    0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.99.254  0.0.0.0         UG    0      0        0 eth0
        

This section introduces changing the IP address on an interface, changing the default gateway, and adding and removing a static route. With the knowledge of ifconfig and route output it's a small step to learn how to change IP configuration with these same tools.

[root@morgan]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:80:C8:F8:4A:53  
          inet addr:192.168.98.82  Bcast:192.168.98.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:9 Base address:0x5000 

[root@morgan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.98.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.98.254  0.0.0.0         UG    0      0        0 eth0
        

[root@morgan]# ifconfig eth0 down
        

[root@morgan]# ifconfig eth0 192.168.99.14 netmask 255.255.255.0 up
[root@morgan]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:80:C8:F8:4A:53  
          inet addr:192.168.99.14  Bcast:192.168.99.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:9 Base address:0x5000 

        

[root@morgan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
[root@morgan]# route add default gw 192.168.99.254
[root@morgan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.99.254  0.0.0.0         UG    0      0        0 eth0
        

[root@morgan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.99.254  0.0.0.0         UG    0      0        0 eth0
[root@morgan]# route add -net 192.168.98.0 netmask 255.255.255.0 gw 192.168.99.1
[root@morgan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.98.0    192.168.99.1    255.255.255.0   UG    0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.99.254  0.0.0.0         UG    0      0        0 eth0
        

[root@morgan]# route del -net 192.168.98.0 netmask 255.255.255.0 gw 192.168.99.1
[root@morgan]# route add -net 192.168.98.42 netmask 255.255.255.255 gw 192.168.99.1
[root@morgan]# route add -host 192.168.98.42 gw 192.168.99.1
SIOCADDRT: File exists
[root@morgan]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.98.42   192.168.99.1    255.255.255.255 UGH   0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.99.254  0.0.0.0         UG    0      0        0 eth0
        

This chapter has introduced the simplest uses of ifconfig and route to view and alter the IP configuration of a host. To reiterate the minimum requirements to create an IP network between two machines:

This concludes the tour of basic host networking and IP layer configuration as well as some basic tools available to the linux user. For further documentation on these tools, other tips, tricks, and more advanced content, keep reading!

Address Resolution Protocol (ARP) hovers in the shadows of most networks. Because of its simplicity, by comparison to higher layer protocols, ARP rarely intrudes upon the network administrator's routine. All modern IP-capable operating systems provide support for ARP. The uncommon alternative to ARP is static link-layer-to-IP mappings.

ARP defines the exchanges between network interfaces connected to an Ethernet media segment in order to map an IP address to a link layer address on demand. Link layer addresses are hardware addresses (although they are not immutable) on Ethernet cards and IP addresses are logical addresses assigned to machines attached to the Ethernet. Subsequently in this chapter, link layer addresses may be known by many different names: Ethernet addresses, Media Access Control (MAC) addresses, and even hardware addresses. Disputably, the correct term from the kernel's perspective is "link layer address" because this address can be changed (on many Ethernet cards) via command line tools. Nevertheless, these terms are not realistically distinct and can be used interchangeably.

2.1.1. Overview of Address Resolution Protocol

Address Resolution Protocol (ARP) exists solely to glue together the IP and Ethernet networking layers. Since networking hardware such as switches, hubs, and bridges operate on Ethernet frames, they are unaware of the higher layer data carried by these frames ^[9]. Similarly, IP layer devices, operating on IP packets need to be able to transmit their IP data on Ethernets. ARP defines the conversation by which IP capable hosts can exchange mappings of their Ethernet and IP addressing.

ARP is used to locate the Ethernet address associated with a desired IP address. When a machine has a packet bound for another IP on a locally connected Ethernet network, it will send a broadcast Ethernet frame containing an ARP request onto the Ethernet. All machines with the same Ethernet broadcast address will receive this packet ^[10]. If a machine receives the ARP request and it hosts the IP requested, it will respond with the link layer address on which it will receive packets for that IP address. N.B., the arp_filter sysctl will alter this behaviour somewhat.

Once the requestor receives the response packet, it associates the MAC address and the IP address. This information is stored in the arp cache. The arp cache can be manipulated with the ip neighbor and arp commands. To learn how and when to manipulate the arp cache, see Section B.1, “arp”.

In Example 1.2, we used ping to test reachability of masq-gw. Using a packet sniffer to capture the sequence of packets on the Ethernet as a result of tristan's attempt to ping, provides an example of ARP in flagrante delicto. Consult the example network map for a visual representation of the network layout in which this traffic occurs.

This is an archetypal conversation between two computers exchanging relevant hardware addressing in order that they can pass IP packets, and is comprised of two Ethernet frames.

Example 2.1. ARP conversation captured with tcpdump ^[11]

[root@masq-gw]# tcpdump -ennqti eth0 \( arp or icmp \) tcpdump: listening on eth0 0:80:c8:f8:4a:51 ff:ff:ff:ff:ff:ff 42: arp who-has 192.168.99.254 tell 192.168.99.35 0:80:c8:f8:5c:73 0:80:c8:f8:4a:51 60: arp reply 192.168.99.254 is-at 0:80:c8:f8:5c:73 0:80:c8:f8:4a:51 0:80:c8:f8:5c:73 98: 192.168.99.35 > 192.168.99.254: icmp: echo request (DF) 0:80:c8:f8:5c:73 0:80:c8:f8:4a:51 98: 192.168.99.254 > 192.168.99.35: icmp: echo reply

	This broadcast Ethernet frame, identifiable by the destination Ethernet address with all bits set (ff:ff:ff:ff:ff:ff) contains an ARP request from tristan for IP address 192.168.99.254. The request includes the source link layer address and the IP address of the requestor, which provides enough information for the owner of the IP address to reply with its link layer address.
	The ARP reply from masq-gw includes its link layer address and declaration of ownership of the requested IP address. Note that the ARP reply is a unicast response to a broadcast request. The payload of the ARP reply contains the link layer address mapping. The machine which initiated the ARP request (tristan) now has enough information to encapsulate an IP packet in an Ethernet frame and forward it to the link layer address of the recipient (00:80:c8:f8:5c:73).
	The final two packets in Example 2.1 display the link layer header and the encapsulated ICMP packets exchanged between these two hosts. Examining the ARP cache on each of these hosts would reveal entries on each host for the other host's link layer address.

This example is the commonest example of ARP traffic on an Ethernet. In summary, an ARP request is transmitted in a broadcast Ethernet frame. The ARP reply is a unicast response, containing the desired information, sent to the requestor's link layer address.

An even rarer usage of ARP is gratuitous ARP, where a machine announces its ownership of an IP address on a media segment. The arping utility can generate these gratuitous ARP frames. Linux kernels will respect gratuitous ARP frames ^[12].

Example 2.2. Gratuitous ARP reply frames

[root@tristan]# arping -q -c 3 -A -I eth0 192.168.99.35 [root@masq-gw]# tcpdump -c 3 -nni eth2 arp tcpdump: listening on eth2 06:02:50.626330 arp reply 192.168.99.35 is-at 0:80:c8:f8:4a:51 (0:80:c8:f8:4a:51) 06:02:51.622727 arp reply 192.168.99.35 is-at 0:80:c8:f8:4a:51 (0:80:c8:f8:4a:51) 06:02:52.620954 arp reply 192.168.99.35 is-at 0:80:c8:f8:4a:51 (0:80:c8:f8:4a:51)

The frames generated in Example 2.2 are ARP replies to a question never asked. This sort of ARP is common in failover solutions and also for nefarious sorts of purposes, such as ettercap.

Unsolicited ARP request frames, on the other hand, are broadcast ARP requests initiated by a host owning an IP address.

Example 2.3. Unsolicited ARP request frames

[root@tristan]# arping -q -c 3 -U -I eth0 192.168.99.35 [root@masq-gw]# tcpdump -c 3 -nni eth2 arp tcpdump: listening on eth2 06:28:23.172068 arp who-has 192.168.99.35 (ff:ff:ff:ff:ff:ff) tell 192.168.99.35 06:28:24.167290 arp who-has 192.168.99.35 (ff:ff:ff:ff:ff:ff) tell 192.168.99.35 06:28:25.167250 arp who-has 192.168.99.35 (ff:ff:ff:ff:ff:ff) tell 192.168.99.35 [root@masq-gw]# ip neigh show

These two uses of arping can help diagnose Ethernet and ARP problems--particularly hosts replying for addresses which do not belong to them.

To avoid IP address collisions on dynamic networks (where hosts are turning on and off, connecting and disconnecting and otherwise changing IP addresses) duplicate address detection becomes important. Fortunately, arping provides this functionality as well. A startup script could include the arping utility in duplicate address detection mode to select between IP addresses or methods of acquiring an IP address.

Example 2.4. Duplicate Address Detection with ARP

[root@tristan]# arping -D -I eth0 192.168.99.147; echo $? ARPING 192.168.99.47 from 0.0.0.0 eth0 Unicast reply from 192.168.99.47 [00:80:C8:E8:1E:FC] for 192.168.99.47 [00:80:C8:E8:1E:FC] 0.702ms Sent 1 probes (1 broadcast(s)) Received 1 response(s) 1 [root@tristan]# tcpdump -eqtnni eth2 arp tcpdump: listening on eth2 0:80:c8:f8:4a:51 ff:ff:ff:ff:ff:ff 60: arp who-has 192.168.99.147 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0 0:80:c8:e8:1e:fc 0:80:c8:f8:4a:51 42: arp reply 192.168.99.147 is-at 0:80:c8:e8:1e:fc (0:80:c8:e8:1e:fc) [root@masq-gw]# ip neigh show

Address Resolution Protocol, which provides a method to connect physical network addresses with logical network addresses is a key element to the deployment of IP on Ethernet networks.

2.1.2. The ARP cache

In simplest terms, an ARP cache is a stored mapping of IP addresses with link layer addresses. An ARP cache obviates the need for an ARP request/reply conversation for each IP packet exchanged. Naturally, this efficiency comes with a price. Each host maintains its own ARP cache, which can become outdated when a host is replaced, or an IP address moves from one host to another. The ARP cache is also known as the neighbor table.

To display the ARP cache, the venerable and cross-platform arp admirably dispatches its duty. As with many of the iproute2 tools, more information is available via ip neighbor than with arp. Example 2.5 below illustrates the differences in the output between the output of these two different tools.

Example 2.5. ARP cache listings with arp and ip neighbor

[root@tristan]# arp -na ? (192.168.99.7) at 00:80:C8:E8:1E:FC [ether] on eth0 ? (192.168.99.254) at 00:80:C8:F8:5C:73 [ether] on eth0 [root@tristan]# ip neighbor show 192.168.99.7 dev eth0 lladdr 00:80:c8:e8:1e:fc nud reachable 192.168.99.254 dev eth0 lladdr 00:80:c8:f8:5c:73 nud reachable

A major difference between the information reported by ip neighbor and arp is the state of the proxy ARP table. The only way to list permanently advertised entries in the neighbor table (proxy ARP entries) is with the arp.

Entries in the ARP cache are periodically and automatically verified unless continually used. Along with net/ipv4/neigh/$DEV/gc_stale_time, there are a number of other parameters in net/ipv4/neigh/$DEV which control the expiration of entries in the ARP cache.

When a host is down or disconnected from the Ethernet, there is a period of time during which other hosts may have an ARP cache entry for the disconnected host. Any other machine may display a neighbor table with the link layer address of the recently disconnected host. Because there is a recently known-good link layer address on which the IP was reachable, the entry will abide. At gc_stale_time the state of the entry will change, reflecting the need to verify the reachability of the link layer address. When the disconnected host fails to respond ARP requests, the neighbor table entry will be marked as incomplete

Here are a the possible states for entries in the neighbor table.

Table 2.1. Active ARP cache entry states

ARP cache entry state	meaning	action if used
permanent	never expires; never verified	reset use counter
noarp	normal expiration; never verified	reset use counter
reachable	normal expiration	reset use counter
stale	still usable; needs verification	reset use counter; change state to delay
delay	schedule ARP request; needs verification	reset use counter
probe	sending ARP request	reset use counter
incomplete	first ARP request sent	send ARP request
failed	no response received	send ARP request

To resume, a host (192.168.99.7) in tristan's ARP cache on the example network has just been disconnected. There are a series of events which will occur as tristan's ARP cache entry for 192.168.99.7 expires and gets scheduled for verification. Imagine that the following commands are run to capture each of these states immediately before state change.

Example 2.6. ARP cache timeout

[root@tristan]# ip neighbor show 192.168.99.7 192.168.99.7 dev eth0 lladdr 00:80:c8:e8:1e:fc nud reachable [root@tristan]# ip neighbor show 192.168.99.7 192.168.99.7 dev eth0 lladdr 00:80:c8:e8:1e:fc nud stale [root@tristan]# ip neighbor show 192.168.99.7 192.168.99.7 dev eth0 lladdr 00:80:c8:e8:1e:fc nud delay [root@tristan]# ip neighbor show 192.168.99.7 192.168.99.7 dev eth0 lladdr 00:80:c8:e8:1e:fc nud probe [root@tristan]# ip neighbor show 192.168.99.7 192.168.99.7 dev eth0 nud incomplete

	Before the entry has expired for 192.168.99.7, but after the host has been disconnected from the network. During this time, tristan will continue to send out Ethernet frames with the destination frame address set to the link layer address according to this entry.
	It has been gc_stale_time seconds since the entry has been verified, so the state has changed to stale.
	This entry in the neighbor table has been requested. Because the entry was in a stale state, the link layer address was used, but now the kernel needs to verify the accuracy of the address. The kernel will soon send an ARP request for the destination IP address.
	The kernel is actively performing address resolution for the entry. It will send a total of ucast_solicit frames to the last known link layer address to attempt to verify reachability of the address. Failing this, it will send mcast_solicit broadcast frames before altering the ARP cache state and returning an error to any higher layer services.
	After all attempts to reach the destination address have failed, the entry will appear in the neighbor table in this state.

The remaining neighbor table flags are visible when initial ARP requests are made. If no ARP cache entry exists for a requested destination IP, the kernel will generate mcast_solicit ARP requests until receiving an answer. During this discovery period, the ARP cache entry will be listed in an incomplete state. If the lookup does not succeed after the specified number of ARP requests, the ARP cache entry will be listed in a failed state. If the lookup does succeed, the kernel enters the response into the ARP cache and resets the confirmation and update timers.

After receipt of a corresponding ARP reply, the kernel enters the response into the ARP cache and resets the confirmation and update timers.

For machines not using a static mapping for link layer and IP addresses, ARP provides on demand mappings. The remainder of this section will cover the methods available under linux to control the address resolution protocol.

2.1.4. The ARP Flux Problem

When a linux box is connected to a network segment with multiple network cards, a potential problem with the link layer address to IP address mapping can occur. The machine may respond to ARP requests from both Ethernet interfaces. On the machine creating the ARP request, these multiple answers can cause confusion, or worse yet, non-deterministic population of the ARP cache. Known as ARP flux ^[13], this can lead to the possibly puzzling effect that an IP migrates non-deterministically through multiple link layer addresses. It's important to understand that ARP flux typically only affects hosts which have multiple physical connections to the same medium or broadcast domain.

This is a simple illustration of the problem in a network where a server has two Ethernet adapters connected to the same media segment. They need not have IP addresses in the same IP network for the ARP reply to be generated by each interface. Note the first two replies received in response to the ARP broadcast request. These replies arrive from conflicting link layer addresses in response to this request. Also notice the greater time required for the sending and receiving hosts to process the broadcast ARP request frames than the unicast frames which follow (probes two and three).

Example 2.7. ARP flux

[root@real-client]# arping -I eth0 -c 3 10.10.20.67 ARPING 10.10.20.67 from 10.10.20.33 eth0 Unicast reply from 10.10.20.67 [00:80:C8:7E:71:D4] 11.298ms Unicast reply from 10.10.20.67 [00:80:C8:E8:1E:FC] 12.077ms Unicast reply from 10.10.20.67 [00:80:C8:E8:1E:FC] 1.542ms Unicast reply from 10.10.20.67 [00:80:C8:E8:1E:FC] 1.547ms Sent 3 probes (1 broadcast(s)) Received 4 response(s)

There are four solutions to this problem. The common solution for kernel 2.4 harnesses the arp_filter sysctl, while the common solution for kernel 2.2 takes advantage of the hidden sysctl. These two solutions alter the behaviour of ARP on a per interface basis and only if the functionality has been enabled.

Alternate solutions which provide much greater control of ARP (possibly documented here at a later date) include Julian Anastasov's ip arp tool and his noarp route flag. While these tools were conceived in the course of the Linux Virtual Server project, they have practical application outside this realm.

2.1.4.1. ARP flux prevention with arp_filter

One method for preventing ARP flux involves the use of net/ipv4/conf/$DEV/arp_filter. In short, the use of arp_filter causes the recipient (in the case below, real-server) to perform a route lookup to determine the interface through which to send the reply, instead of the default behaviour (shown above), replying from all Ethernet interfaces which receive the request.

The arp_filter solution can have unintended effects if the only route to the destination is through one of the network cards. In Example 2.8, real-client will demonstrate this. This instructive example should highlight the shortcomings of the arp_filter solution in very complex networks where finer-grained control is required.

In general, the arp_filter solution sufficiently solves the ARP flux problem. First, hosts do not generate ARP requests for networks to which they do not have a direct route (see Section 4.2, “Routing to Locally Connected Networks”) and second, when such a route exists, the host normally chooses a source address in the same network as the destination. So, the arp_filter solution is a good general solution, but does not adequately address the occasional need for more control over ARP requests and replies.

Example 2.8. Correction of ARP flux with conf/$DEV/arp_filter

[root@real-server]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter [root@real-server]# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_filter [root@real-server]# echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_filter [root@real-server]# ip address show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:80:c8:e8:1e:fc brd ff:ff:ff:ff:ff:ff inet 10.10.20.67/24 scope global eth0 [root@real-server]# ip address show dev eth1 3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:80:c8:7e:71:d4 brd ff:ff:ff:ff:ff:ff inet 192.168.100.1/24 brd 192.168.100.255 scope global eth1 [root@real-client]# arping -I eth0 -c 3 10.10.20.67 ARPING 10.10.20.67 from 10.10.20.33 eth0 Unicast reply from 10.10.20.67 [00:80:C8:E8:1E:FC] 0.882ms Unicast reply from 10.10.20.67 [00:80:C8:E8:1E:FC] 1.221ms Unicast reply from 10.10.20.67 [00:80:C8:E8:1E:FC] 1.487ms Sent 3 probes (1 broadcast(s)) Received 3 response(s) [root@real-client]# arping -I eth0 -c 3 192.168.100.1 ARPING 192.168.100.1 from 10.10.20.33 eth0 Unicast reply from 192.168.100.1 [00:80:C8:E8:1E:FC] 0.877ms Unicast reply from 192.168.100.1 [00:80:C8:E8:1E:FC] 1.517ms Unicast reply from 192.168.100.1 [00:80:C8:E8:1E:FC] 1.661ms Sent 3 probes (1 broadcast(s)) Received 3 response(s) [root@real-client]# ip neighbor del 192.168.100.1 dev eth0 [root@real-client]# ip address add 192.168.100.2/24 brd + dev eth0 [root@real-client]# arping -I eth0 -c 3 192.168.100.1 ARPING 192.168.100.1 from 192.168.100.2 eth0 Unicast reply from 192.168.100.1 [00:80:C8:7E:71:D4] 0.804ms Unicast reply from 192.168.100.1 [00:80:C8:7E:71:D4] 1.381ms Unicast reply from 192.168.100.1 [00:80:C8:7E:71:D4] 2.487ms Sent 3 probes (1 broadcast(s)) Received 3 response(s)

	Set the sysctl variables to enable the arp_filter functionality. After this, you might expect that ARP replies for 10.10.20.67 would only advertise the link layer address on eth0 (00:80:c8:e8:1e:fc).
	Here is the expected behaviour. Only one reply comes in for the IP 10.10.20.67 after the arp_filter sysctl has been enabled. The reply originates from the interface on real-server which actually hosts the IP address. Note that the source address on the ARP queries is 10.10.20.33, and that the ARP query causes real-server to perform a route lookup on 10.10.20.33 to choose an interface from which to send the reply.
	Here, real-client requests the link layer address of the host 192.168.100.1, but the source IP on the request packet (chosen according to the rules for source address selection) is 10.10.20.33. When real-server looks up a route to this destination, it chooses its eth0, and replies with the link layer address of its eth0. Conventional networking needs should not run afoul of this oddity of the arp_filter ARP flux prevention technique.
	Remove the entry in the neighbor table before testing again.
	By adding an IP address in the same network as the intended destination (which would be rather common where multiple IP networks share the same medium or broadcast domain), the kernel can now select a different source address for the ARP request packets.
	Note the source address of the ARP queries is now 192.168.100.2. When real-server performs a route lookup for the 192.168.100.0/24 destination, the chosen path is through eth1. The ARP reply packets now have the correct link layer address.

In general, the arp_filter solution should suffice, but this knowledge can be key in determining whether or not an alternate solution, such as an ARP filtering solution are necessary.

2.1.4.2. ARP flux prevention with hidden

The ARP flux problem can also be combatted with a kernel patch by Julian Anastasov, which was incorporated into the 2.2.14+ kernel series, but never into the 2.4+ kernel series. Therefore, the functionality may not be available in all kernels.

The sysctl net/ipv4/conf/$DEV/hidden toggles the generation of ARP replies for requested IPs. It marks an interface and all of its IP addresses invisible to other interfaces for the purpose of ARP requests. When an ARP request arrives on any interface, the kernel tests to see if the IP address is locally hosted anywhere on the machine. If the IP is found on any interface, the kernel will generate a reply.

Since this is not always desirable, the hidden sysctl can be employed. This prevents the kernel from finding the IP address when testing to see what IP addresses are locally hosted. The kernel can always find IPs hosted on the interface on which the packet arrived, but it cannot find addresses which are hidden.

As shown in Example 2.9, not only can ARP flux be corrected, but sensitive information about the IP addresses available on a linux box can be safeguarded ^[14]. This makes the hidden sysctl useful for preventing unwanted IP disclosure via ARP on multi-homed hosts, in addition to preventing ARP flux on hosts connected to the same network medium.

Example 2.9. Correction of ARP flux with net/$DEV/hidden

[root@real-client]# arping -I eth0 -c 1 172.19.22.254 ARPING 172.19.22.254 from 172.19.22.2 eth0 Unicast reply from 172.19.22.254 [00:60:F5:08:8A:2D] 0.704ms Unicast reply from 172.19.22.254 [00:60:F5:08:8A:2E] 0.844ms Unicast reply from 172.19.22.254 [00:60:F5:08:8A:2F] 0.918ms Unicast reply from 172.19.22.254 [00:60:F5:08:8A:2C] 0.974ms Sent 1 probes (1 broadcast(s)) Received 4 response(s) [root@real-server]# for i in all eth2 eth3 eth4 eth5 ; do > echo 1 > /proc/sys/net/ipv4/conf/$i/hidden > done [root@real-client]# arping -I eth0 -c 2 172.19.22.254 ARPING 172.19.22.254 from 172.19.22.2 eth0 Unicast reply from 172.19.22.254 [00:60:F5:08:8A:2D] 0.710ms Unicast reply from 172.19.22.254 [00:60:F5:08:8A:2D] 0.624ms Sent 2 probes (1 broadcast(s)) Received 2 response(s)

These are two examples of methods to prevent ARP flux. Other alternatives for correcting this problem are documented in Section 2.1.6, “ARP filtering”, where much more sophisticated tools are available for manipulation and control over the ARP functions of linux.

Virtual LANs are a way to take a single switch and subdivide it into logical media segments. A single switch port in a VLAN-capable switch can carry packets from multiple virtual LANs and linux can understand the format of these Ethernet frames. For more on this, see the linux 802.1q VLAN implementation site.

Kernels in the late 2.4 series have support for VLAN incorporated into the stock release. The vconfig tool, however needs to be compiled against the kernel source in order to provide userland configurability of the kernel support for VLANs.

There are a few items of note which may prevent quick adoption of VLAN support under linux. Ben McKeegan wrote a good summary of the MTU/MRU issues involved with VLANs and 10/100 Ethernet. Gigabit Ethernet drivers are not hamstrung with this problem. Consider using gigabit Ethernet cards from the outset to avoid these potential problems.

[root@real-router]# vconfig add eth0 7
[root@real-router]# ip addr add dev eth0.7 192.168.30.254/24 brd +
[root@real-router]# ip link set dev eth0.7 up
       

Each interface defined using the vconfig utility takes its name from the base device to which it has been bound, and appends the VLAN tag ID, as shown in Example 2.10.

This documentation is sparse. Visit the main site and the VLAN mailing list archives.

Networking vendors have long offered a functionality for aggregating bandwidth across multiple physical links to a switch. This allows a machine (frequently a server) to treat multiple physical connections to switch units as a single logical link. The standard moniker for this technology is IEEE 802.3ad, although it is known by the common names of trunking, port trunking and link aggregation. The conventional use of bonding under linux is an implementation of this link aggregation.

A separate use of the same driver allows the kernel to present a single logical interface for two physical links to two separate switches. Only one link is used at any given time. By using media independent interface signal failure to detect when a switch or link becomes unusable, the kernel can, transparently to userspace and application layer services, fail to the backup physical connection. Though not common, the failure of switches, network interfaces, and cables can cause outages. As a component of high availability planning, these bonding techniques can help reduce the number of single points of failure.

For more information on bonding, see the Documentation/networking/bonding.txt from the linux source code tree.

[root@real-server root]# modprobe  bonding
[root@real-server root]# ip addr add 192.168.100.33/24 brd + dev bond0
[root@real-server root]# ip link set dev bond0 up
[root@real-server root]# ifenslave  bond0 eth2 eth3
master has no hw address assigned; getting one from slave!
The interface eth2 is up, shutting it down it to enslave it.
The interface eth3 is up, shutting it down it to enslave it.
[root@real-server root]# ifenslave  bond0 eth2 eth3
[root@real-server root]# cat /proc/net/bond0/info
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 0

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
        

[root@real-server root]# modprobe bonding mode=1 miimon=100 downdelay=200 updelay=200
[root@real-server root]# ip link set dev bond0 addr 00:80:c8:e7:ab:5c
[root@real-server root]# ip addr add 192.168.100.33/24 brd + dev bond0
[root@real-server root]# ip link set dev bond0 up
[root@real-server root]# ifenslave  bond0 eth2 eth3
The interface eth2 is up, shutting it down it to enslave it.
The interface eth3 is up, shutting it down it to enslave it.
[root@real-server root]# ip link show eth2 ; ip link show eth3 ; ip link show bond0
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 100
  link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,NOARP,SLAVE,DEBUG,AUTOMEDIA,PORTSEL,NOTRAILERS,UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 100
  link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff
58: bond0: <BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue
  link/ether 00:80:c8:e7:ab:5c brd ff:ff:ff:ff:ff:ff
        

There is a Bridge and Netfilter HOWTO which illustrates the use of a bridge as a firewall.

Yes, Virginia, it can be done.

In order to take advantage of ebtables the machine needs to be running as a bridge. (Accurate, nicht wahr?)

If you believe in really scary stuff, you can run the bridging code with netfilter, so you can manipulate IP packets transparently on your bridge. For more on this, see the documentation of bridging and firewalling. The firewall and bridge architecture is part of the development branch of the kernel 2.5 series.

The design of IP routing allows for very simple route definitions for small networks, while not hindering the flexibility of routing in complex environments. A key concept in IP routing is the ability to define what addresses are locally reachable as opposed to not directly known destinations. Every IP capable host knows about at least three classes of destination: itself, locally connected computers and everywhere else.

Most fully-featured IP-aware networked operating systems (all unix-like operating systems with IP stacks, modern Macintoshes, and modern Windows) include support for the loopback device and IP. This is an IP and range configured on the host machine itself which allows the machine to talk to itself. Linux systems can communicate over IP on any locally configured IP address, whether on the loopback device or not. This is the first class of destinations: locally hosted addresses.

The second class of IP addresses are addresses in the locally connected network segment. Each machine with a connection to an IP network can reach a subset of the entire IP address space on its directly connected network interface.

All other hosts or destination IPs fall into a third range. Any IP which is not on the machine itself or locally reachable (i.e. connected to the same media segment) is only reachable through an IP routing device. This routing device must have an IP address in a locally reachable IP address range.

All IP networking is a permutation of these three fundamental concepts of reachability. This list summarizes the three possible classifications for reachability of destination IP addresses from any single source machine.

As a practical description of the above, this partial diagram of the example network shows two machines connected to 192.168.99.0/24. On tristan the IP addresses 127.0.0.1 (loopback--not pictured) and 192.168.99.35 are considered locally hosted IP addresses. The directly reachable IP addresses fall inside the 192.168.99.0/24 network. Any other destination addresses are only reachable through a gateway, probably masq-gw.

Example 4.1. Classes of IP addresses

Before examining the routing system in more detail, there are some terms to identify and define. These terms are general IP networking terms and should be familiar to users who have used IP on other operating systems and networking equipment.

These definitions are common to IP networking in general, and are understood by all in the IP networking community. For less terse introductory material on matters of IP network addressing in general, see Section I.1.3, “General IP Networking Resources”.

As is apparent from the interdependencies amongst the above definitions, each term defines a separate part of the concept of the relationships between an IP address and its network. A good IP calculator can assist in mastering these IP fundamentals.

[user@workstation]$ ipcalc -n 12.7.149.0/26

Address:   12.7.149.0            00001100.00000111.10010101.00 000000
Netmask:   255.255.255.192 = 26  11111111.11111111.11111111.11 000000
Wildcard:  0.0.0.63              00000000.00000000.00000000.00 111111
=>
Network:   12.7.149.0/26         00001100.00000111.10010101.00 000000 (Class A)
Broadcast: 12.7.149.63           00001100.00000111.10010101.00 111111
HostMin:   12.7.149.1            00001100.00000111.10010101.00 000001
HostMax:   12.7.149.62           00001100.00000111.10010101.00 111110
Hosts/Net: 62 
      

A tool similar to the one shown in Example 4.2 can assist in visualizing the relationships among IP addressing concepts.

Subequently, this chapter will introduce some concrete examples of routing in a real network. The example network illustrates this network and all of the addresses involved.

Any IP network is defined by two sets of numbers: network address and netmask. By convention, there are two ways to represent these two numbers. Netmask notation is the convention and tradition in IP networking although the more succinct CIDR notation is gaining popularity.

In the example network, isolde has IP address 192.168.100.17. In CIDR notation, isolde's address is 192.168.100.17/24, and in traditional netmask notation, 192.168.100.17/255.255.255.0. Any of the IP calculators, confirms that the first usable IP address is 192.168.100.1 and the last usable IP address is 192.168.100.254. Importantly, the IP network address, 192.168.100.0/24, is reachable through the directly connected Ethernet interface (refer to classification 2). Therefore, isolde should be able to reach any IP address in this range directly on the locally connected Ethernet segment.

Below is the routing table for isolde, first shown with the conventional route -n output ^[15] and then with the ip route show ^[16] command. Each of these tools conveys the same routing table and operates on the same kernel routing table. For more on the routing table displayed in Example 4.3, consult Section 4.8.3, “The Main Routing Table”.

[root@isolde]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         192.168.100.254 0.0.0.0         UG    0      0        0 eth0
[root@isolde]# ip route show
192.168.100.0/24 dev eth0  scope link 
127.0.0.0/8 dev lo  scope link 
default via 192.168.100.254 dev eth0
      

In the above example, the locally reachable destination is 192.168.100.0/255.255.255.0 which can also be written 192.168.100.0/24 as in ip route show. In classful networking terms, the network to which isolde is directly connected is called a class C sized network.

When a process on isolde needs to send a packet to another machine on the locally connected network, packets will be sent from 192.168.100.17 (isolde's IP). The kernel will consult the routing table to determine the route and the source address to use when sending this packet. Assuming the destination is 192.168.100.32, the kernel will find that 192.168.100.32 falls inside the IP address range 192.168.100.0/24 and will select this route for the outbound packet. For further details on source address selection, see Section 4.6, “Source Address Selection”. The source address on the outbound packet conveys vital information to the host receiving the packet. In order for the packet to be able to return, isolde has to use an IP address that is locally available, 192.168.100.32 has to have a route to isolde and neither host must block the packet.

The packet will be sent to the locally connected network segment directly, because isolde interprets from the routing table that 192.168.100.32 is directly reachable through the physical network connection on eth0.

Occasionally, a machine will be directly connected to two different IP networks on the same device. The routing table will show that both networks are reachable through the same physical device. For more on this topic, see Section 9.2, “Multiple IP Networks on one Ethernet Segment”. Similarly, multi-homed hosts will have routes for all locally connected networks through the locally-connected network interface. For more on this sort of configuration, see Section 9.6, “Multihomed Hosts”.

This covers the classification of IP destinations which are available on a locally connected network. This highlights the importance of an accurate netmask and network address. The next section will cover IP ranges which are neither locally hosted nor fall in the range of the locally reachable networks. These destinations must be reached through a router.

By comparison to the total number of publicly accessible hosts on the Internet there is an almost insignificant number of hosts inside any locally reachable network. This means that the majority of potential destinations are only available via a router.

Any machine which will accept and forward packets between two networks is a router. Every router is at least dual-homed; one NIC connects to one network, and a second NIC connects to another network. Machines connected to either network learn by a routing protocol or are statically configured to pass traffic for the other network to the router.

For tristan, there are two different paths out of 192.168.99.0/24. One path has another leaf network, 192.168.98.0/24, and the other path has many networks, including the Internet. The routing table on tristan should then contain two different routes out of the network. One destination 192.168.98.0/24 will be reachable through 192.168.99.1. So, if tristan has a packet with a destination IP address in the range of the branch office network, it will choose to send the packet directly to isdn-router.

The default route is another way to say the route for destination 0/0. This is the most general possible route. It is the catch-all route. If no more specific route exists in a routing table, a default route will be used. Many servers and workstations are connected to leaf networks with only one router, hence Example 4.3 shows a very common sort of routing table. There's a route for localhost, for the locally connected IP network, and a default route.

For Internet-connected hosts, the default route is customarily set to the IP of the locally reachable router which has a path to the Internet. Each router in turn has a default gateway pointing to another Internet-connected router until the packet is handed off to an Internet Service Provider's network.

Operating as a router allows a linux machine to accept packets on one interface and transmit them on another. This is the nature of a router. The process of accepting and transmitting IP packets is known as forwarding. IP forwarding is a requirement for many of the networking techniques identified here. Stateless NAT and firewalling, transparent proxying and masquerading all require the support of IP forwarding in order to function correctly.

The sysctl net/ipv4/ip_forward toggles the IP forwarding functionality on a linux box. Note that setting this sysctl alters other routing-related sysctl entries, so it is wise to set this first, and then alter other entries. Frequently, an administrator will forget this simple and crucial detail when configuring a new machine to operate as a router only to be frustrated at the simple error.

The sysctl net/ipv4/conf/$DEV/forward defaults to the value of net/ipv4/ip_forward, but can be independently modified. In order to allow forwarding of packets between two interfaces while prohibiting such behaviour on a third interface, this sysctl can be employed.

Crucial to the proper ability of hosts to exchange IP packets is the correct selection of a route to the destination. The rules for the selection of route path are traditionally made on a hop-by-hop basis ^[17] based solely upon the destination address of the packet. Linux behaves as a conventional routing device in this way, but can also provide a more flexible capability. Routes can be chosen and prioritized based on other packet characteristics.

The route selection algorithm under linux has been generalized to enable the powerful latter scenario without complicating the overwhelmingly common case of the former scenario.

if packet.routeCacheLookupKey in routeCache :
    route = routeCache[ packet.routeCacheLookupKey ]
else
    for rule in rpdb :
        if packet.rpdbLookupKey in rule :
            routeTable = rule[ lookupTable ]
            if packet.routeLookupKey in routeTable :
                route = route_table[ packet.routeLookup_key ]
          

[root@isolde]# ip rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup 253
        

The selection of the correct source address is key to correct communication between hosts with multiple IP addresses. If a host chooses an address from a private network to communicate with a public Internet host, it is likely that the return half of the communication will never arrive.

The initial source address for an outbound packet is chosen in according to the following series of rules. The application can request a particular IP ^[19], the kernel will use the src hint from the chosen route path ^[20], or, lacking this hint, the kernel will choose the first address configured on the interface which falls in the same network as the destination address or the nexthop router.

The following list recapitulates the manner by which the kernel determines what the source address of an outbound packet.

Also refer to this excerpt from the iproute2 command reference.

The routing cache is also known as the forwarding information base (FIB). This term may be familiar to users of other routing systems.

The routing cache stores recently used routing entries in a fast and convenient hash lookup table, and is consulted before the routing tables. If the kernel finds a matching entry during route cache lookup, it will forward the packet immediately and stop traversing the routing tables.

Because the routing cache is maintained by the kernel separately from the routing tables, manipulating the routing tables may not have an immediate effect on the kernel's choice of path for a given packet. To avoid a non-deterministic lag between the time that a new route is entered into the kernel routing tables and the time that a new lookup in those route tables is performed, use ip route flush cache. Once the route cache has been emptied, new route lookups (if not by a packet, then manually with ip route get) will result in a new lookup to the kernel routing tables.

The following is a listing of the hash lookup keys in the routing cache and a description of each key. Compare this list with the elements identified in Table 4.1.

The following attributes may be stored for each entry in the routing cache.

Collectively the hash keys uniquely identify routes in the forwarding information base (routing cache) and each entry provides attributes of the route.

Linux kernel 2.2 and 2.4 support multiple routing tables ^[21]. Beyond the two commonly used routing tables (the local and main routing tables), the kernel supports up to 252 additional routing tables.

The multiple routing table system provides a flexible infrastructure on top of which to implement policy routing. By allowing multiple traditional routing tables (keyed primarily to destination address) to be combined with the routing policy database (RPDB) (keyed primarily to source address), the kernel supports a well-known and well-understood interface while simultaneously expanding and extending its routing capabilities. Each routing table still operates in the traditional and expected fashion. Linux simply allows you to choose from a number of routing tables, and to traverse routing tables in a user-definable sequence until a matching route is found.

Any given routing table can contain an arbitrary number of entries, each of which is keyed on the following characteristics (cf. Table 4.1)

For practical purposes, this means that (even) a single routing table can contain multiple routes to the same destination if the ToS differs on each route or if the route applies to a different interface ^[22].

Kernels supporting multiple routing tables refer to routing tables by unique integer slots between 0 and 255 ^[23]. The two routing tables normally employed are table 255, the local routing table, and table 254, the main routing table. For examples of using multiple routing tables, see Chapter 9, Advanced IP Management, in particular, Example 10.1, Example 10.3 and Example 10.4. Also be sure to read Section 10.3, “Using the Routing Policy Database and Multiple Routing Tables” and Section 4.9, “Routing Policy Database (RPDB)”.

The ip route and ip rule commands have built in support for the special tables main and local. Any other routing tables can be referred to by number or an administratively maintained mapping file, /etc/iproute2/rt_tables.

The format of this file is extraordinarily simple. Each line represents one mapping of an arbitrary string to an integer. Comments are allowed.

Example 4.6. Typical content of /etc/iproute2/rt_tables

# # reserved values # 255 local 254 main 253 default 0 unspec # # local # 1 inr.ruhep

	The local table is a special routing table maintained by the kernel. Users can remove entries from the local routing table at their own risk. Users cannot add entries to the local routing table. The file /etc/iproute2/rt_tables need not exist, as the iproute2 tools have a hard-coded entry for the local table.
	The main routing table is the table operated upon by route and, when not otherwise specified, by ip route. The file /etc/iproute2/rt_tables need not exist, as the iproute2 tools have a hard-coded entry for the main table.
	The default routing table is another special routing table, but WHY is it special!?!
	Operating on the unspec routing table appears to operate on all routing tables simultaneously. Is this true!? What does that imply?
	This is an example indicating that table 1 is known by the name inr.ruhep. Any references to table inr.ruhep in an ip rule or ip route will substitue the value 1 for the word inr.ruhep.

The routing table manipulated by the conventional route command is the main routing table. Additionally, the use of both ip address and ifconfig will cause the kernel to alter the local routing table (and usually the main routing table). For further documentation on how to manipulate the other routing tables, see the command description of ip route.

[root@real-server]# ip address show dev eth1
6: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
   link/ether 00:80:c8:e8:1e:fc brd ff:ff:ff:ff:ff:ff
   inet 10.10.20.89/24 brd 10.10.20.255 scope global eth1
[root@real-server]# ip route show dev eth1
10.10.20.0/24  proto kernel  scope link  src 10.10.20.89
[root@real-server]# ip route show dev eth1 table local
broadcast 10.10.20.0  proto kernel  scope link  src 10.10.20.89 
broadcast 10.10.20.255  proto kernel  scope link  src 10.10.20.89
local 10.10.20.89  proto kernel  scope host  src 10.10.20.89
[root@real-server]# ip address add 192.168.254.254/24 brd + dev eth1
[root@real-server]# ip address show dev eth1
6: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
   link/ether 00:80:c8:e8:1e:fc brd ff:ff:ff:ff:ff:ff
   inet 10.10.20.89/24 brd 10.10.20.255 scope global eth1
   inet 192.168.254.254/24 brd 192.168.254.255 scope global eth1
[root@real-server]# ip route show dev eth1
10.10.20.0/24  proto kernel  scope link  src 10.10.20.89 
192.168.254.0/24  proto kernel  scope link  src 192.168.254.254
[root@real-server]# ip route show dev eth1 table local
broadcast 10.10.20.0  proto kernel  scope link  src 10.10.20.89 
broadcast 192.168.254.0  proto kernel  scope link  src 192.168.254.254          
broadcast 10.10.20.255  proto kernel  scope link  src 10.10.20.89               
local 192.168.254.254  proto kernel  scope host  src 192.168.254.254            
local 10.10.20.89  proto kernel  scope host  src 10.10.20.89                    
broadcast 192.168.254.255  proto kernel  scope link  src 192.168.254.254
        

The routing policy database (RPDB) controls the order in which the kernel searches through the routing tables. Each rule has a priority, and rules are examined sequentially from rule 0 through rule 32767.

When a new packet arrives for routing (assuming the routing cache is empty), the kernel begins at the highest priority rule in the RPDB--rule 0. The kernel iterates over each rule in turn until the packet to be routed matches a rule. When this happens the kernel follows the instructions in that rule. Typically, this causes the kernel to perform a route lookup in a specified routing table. If a matching route is found in the routing table, the kernel uses that route. If no such route is found, the kernel returns to traverse the RPDB again, until every option has been exhausted.

The priority-based rule system provides a flexible way to define routes while taking advantage of the traditional routing table concept. For a complete picture of the entire route selection process including the RPDB, see the section on routing selection.

There are a number of different rule types available for use in the routing policy database. These rule types have a striking similarity to the route types available for route entries.

The routing policy database provides the core of functionality around which the policy routing and advanced routing features can be built.

ICMP is a very important part of the communication between hosts on IP networks. Used by routers and endpoints (clients and servers) ICMP communicates error conditions in networks and provides a means for endpoints to receive information about a network path or requested connection.

One of the commonest uses of ICMP by the administrator of a network is the use of ping to detect the state of a machine in the network. There are other types of ICMP which are used for other inter-computer communication. One other common type of ICMP is the ICMP returned by a router or host which is not accepting connections. Essentially, the host returns the ICMP as a polite method of saying “Go away.”.

[root@tristan]# echo test | nc 192.168.98.82 22
[root@tristan]# tcpdump -nneqti eth0
0:80:c8:f8:4a:51 0:80:c8:f8:5c:71 74: 192.168.99.35.54510 > 192.168.98.82.22: tcp 0 (DF)
0:80:c8:f8:5c:71 0:80:c8:f8:4a:51 102: 192.168.99.254 > 192.168.99.35: icmp: redirect 192.168.98.82 to host 192.168.99.1 [tos 0xc0] 
0:80:c8:f8:5c:71 0:c0:7b:45:6a:39 74: 192.168.99.35.54510 > 192.168.98.82.22: tcp 0 (DF)
        

Network address translation (NAT) is a technique of transparently mapping an IP address or range to another IP address or range. Any routing device situated between two endpoints can perform this transformation of the packet. Network designers must however take one key element under consideration when laying out a network with NAT in mind. The router(s) performing NAT must have an opportunity to rewrite the packet upon entry to the network and upon exit from the network ^[29].

Because network address translation manipulates the addressing of a packet, the NAT transformation becomes a passive but critical part of the conversation between hosts exchanging packets. NAT is by necessity transparent to the application layer endpoints and operates on any type of IP packet. There are some application and even network layer protocols which will break as a result of this rewriting. Consult Section 5.2, “Application Layer Protocols with Embedded Network Information” for a discussion of these cases.

Here are a few common reasons to consider NAT along with potential NAT solution candidates shown in parentheses.

These are the commonest reasons to consider and implement NAT. Other niche applications of NAT, notably as part of load balancing systems, exist although this chapter will concentrate on the use of NAT to hide, isolate or renumber networks. It will also cover inbound connections, leaving the discussion of many-to-one NAT, SNAT and masquerading for Chapter 6, Masquerading and Source Network Address Translation.

One motivator for deploying NAT in a network is the benefit of virtualizing the network. By isolating services provided in one network from changes in other networks, the effects of such changes can be minimized. The disadvantage of virtualizing the network in this way is the increased reliance on the NAT device.

Providing inbound services via NAT can be accomplished in several different ways. Two common techniques are to use iproute2 NAT and netfilter DNAT. Less common (and possibly less desirable) one can use port redirection tools. Depending on which tool is employed, different characteristics of a packet can trigger the address transformation.

The simplest form of NAT under linux is iproute2 NAT. This type of NAT requires two matching commands, one to cause the kernel to rewrite the inbound packets (ip route add nat $NATIP via $REAL) and one to rewrite the outbound packets (ip rule add from $REAL nat $NATIP). The router configured in this fashion will retain no state for connections. It will simply transform any packets passing through. By contrast, netfilter is capable of retaining state on connections passing through the router and selecting packets more granularly than is possible with only iproute2 tools.

Before the advent of the netfilter engine in the linux kernel, there were several tools available to administer NAT, DNAT and PAT. These tools were not included in many distributions and weren't adopted broadly in the community. Although you may find references to ipmasqadm, ipnatadm and ipportfw across the Internet in older documentation, these tools have been superseded in functionality and widespread deployment by the netfilter engine and its userspace partner, iptables.

The netfilter engine provides a more flexible language for selection of packets to be transformed than that provided by the iproute2 suite and kernel routing functionality. Additionally, any NAT services provided by the netfilter engine come with the labor-saving and resource-consuming connection tracking mechanism. DNAT translates the address on an inbound packet and creates an entry in the connection tracking state table. For even modest machines, the connection tracking resource consumption should not be problematic.

Netfilter DNAT allows the user to select packets based on characteristics such as destination port. This blurs the distinction between network address translation and port address translation. NAT always transforms the layer 3 contents of a packet. Port redirection operates at layer 4. From a practical perspective, there is little difference between a port redirection and a netfilter DNAT which has selected a single port. The manner in which the packet and contents are retransmitted, however, is tremendously different.

One other less common technique for furnishing inbound services is the use of port redirection. Although there are higher layer tools which can perform transparent application layer proxying (e.g. Squid), these are outside the scope of this documentation.

There are a number of IP addresses involved in any NAT transformations or connection states. The following list identifies these names and the convention used to describe each IP address. Beware that the prevalance of NAT to publish services on the Internet via public IP addresses has lead to the server/client lingo common in discussions of NAT.

The above terms will be used below and in general discussions of NAT.

Network address translation is beautifully invisible when it works, but has adverse effects on some protocols. Some network applications, e.g., FTP, SNMP, H323, LDAP, IRC, make use of embedded IP information in the application layer protocol or data stream. Since the 2.0.x kernel series (which is not covered here), linux has supported modules which inspect and manipulate packet contents on particular types of packets when used with NAT or masquerading.

FTP is the classic example. Within the FTP control channel (usually established to destination port tcp/21) the client and the server exchange IP address and port information. If the network address translation device doesn't manipulate this data, the FTP server will not be able to contact the client to provide the data.

Passive mode FTP provides the possibility for a network layer which requires only outbound TCP connections. This results in a more NAT friendly and firewall friendly protocol, because the connections are initiated from the client.

Not only are there network applications which break when NAT is involved but also network layer protocols. IPSec is a standards-based network-layer security protocol commonly used in VPNs and IPv6 networks. There are many different ways to use IPSec, but, when used in AH (Authentication Header) mode, NAT will break IPSec functionality.

This underscores the importance of determining if NAT is the best solution for the problem. There are kernel modules to help handle many (though not all) of the application layer protocol when using NAT, but some protocols, such as IPSec in AH mode simply cannot be used with NAT.

Stateless NAT, occasionally maligned as dumb NAT ^[30], is the simplest form of NAT. It involves rewriting addresses passing through a routing device: inbound packets will undergo destination address rewriting and outbound packets will undergo source address rewriting. The iproute2 suite of tools provides the two commands required to configure the kernel to perform stateless NAT. This section will cover only stateless NAT, which can only be accomplished under linux with the iproute2 tools, although it can be simulated with netfilter.

Creating an iproute2 NAT mapping has the side effect of causing the kernel to answer ARP requests for the NAT IP. For more detail on ARP filtering, suppression and conditional ARP, see Chapter 2, Ethernet. This can be considered, alternatively, a benefit or a misfeature of the kernel support for NAT. The nat entry in the local routing table causes the kernel to reply for ARP requests to the NAT IP. Conversely, netfilter DNAT makes no ARP entry or provision for neighbor advertisement.

Whether or not it is using a packet filter, a linux machine can perform NAT using the iproute2 suite of tools. This chapter will document the use of iproute2 tools for NAT with a simple example and an explanation of the required commands, then an example of using NAT with the RPDB and using NAT with a packet filter.

NAT with iproute2 can be used in conjunction with the routing policy database (cf. RPDB) to support conditional NAT, e.g. only perform NAT if the source IP falls within a certain range. See Section 5.3.3, “Conditional Stateless NAT”.

5.3.1. Stateless NAT Packet Capture and Introduction

Assume that example company in example network wants to provide SMTP service on a public IP (205.254.211.0/24) but plans to move to a different IP addressing space in the near future. Network address translation can assist example company prepare for the move. The administrator will select an IP on the internal network (192.168.100.0/24) and configure the router to accept and translate packets for the publicly reachable IP into the private IP.

Example 5.1.  Stateless NAT Packet Capture ^[31]

[root@masq-gw]# tcpdump -qnn 19:30:17.824853 eth1 < 64.70.12.210.35131 > 205.254.211.17.25: tcp 0 (DF) 19:30:17.824976 eth0 > 64.70.12.210.35131 > 192.168.100.17.25: tcp 0 (DF) 19:30:17.825400 eth0 < 192.168.100.17.25 > 64.70.12.210.35131: tcp 0 (DF) 19:30:17.825568 eth1 > 205.254.211.17.25 > 64.70.12.210.35131: tcp 0 (DF)

	The first packet comes in on eth1, masq-gw's outside interface. The packet is addressed to the NAT IP, 205.254.211.17 on tcp/25. This is the IP/port pair on which which our service runs. This is a snapshot of the packet before it has been handled by the NAT code.
	The next line is the "same" packet leaving eth0, masq-gw's inside interface, bound for the internal network. The NAT code has substituted the real IP of the server, 192.168.100.17. This rewriting is handled by the nat entry in the local routing table (ip route). See also Example 5.2.
	The SMTP server then sends a return packet which arrives on eth0. This is the packet before the NAT code on masq-gw has rewritten the outbound packet. This rewriting is handled by the RPDB entry (ip rule). See also Example 5.2.
	Finally, the return packet is transmitted on eth1 after having been rewritten. The source IP address on the packet is now the public IP on which the service is published.

5.3.2. Stateless NAT Practicum

There are only a few commands which are required to enable stateless NAT on a linux routing device. The commands below will configure the host masq-gw (see Section A.1, “Example Network Map and General Notes” and Section A.2, “Example Network Addressing Charts”) as shown above in Example 5.1.

Example 5.2. Basic commands to create a stateless NAT

[root@masq-gw]# ip route add nat 205.254.211.17 via 192.168.100.17 [root@masq-gw]# ip rule add nat 205.254.211.17 from 192.168.100.17 [root@masq-gw]# ip route flush cache [root@masq-gw]# ip route show table all | grep ^nat nat 205.254.211.17 via 192.168.100.17 table local scope host [root@masq-gw]# ip rule show 0: from all lookup local 32765: from 192.168.100.17 lookup main map-to 205.254.211.17 32766: from all lookup main 32767: from all lookup 253

	This command tells the kernel to perform network address translation on any packet bound for 205.254.211.17. The parameter via tells the NAT code to rewrite the packet bound for 205.254.211.17 with the new destination address 192.168.100.17. Note, that this only handles inbound packets; that is, packets whose destination address contains 205.254.211.17.
	This command enters the corresponding rule for the outbound traffic into the RPDB (kernel 2.2 and up). This rule will cause the kernel rewrite any packet from 192.168.100.17 with the specified source address (205.254.211.17). Any packet originating from 192.168.100.17 which passes through this router will trigger this rule. In short, this command rewrites the source address of outbound packets so that they appear to originate from the NAT IP.
	The kernel maintains a routing cache to handle routing decisions more quickly (Section 4.7, “Routing Cache”). After making changes to the routing tables on a system, it is good practice to empty the routing cache with ip route flush cache. Once the cache is empty, the kernel is guaranteed to consult the routing tables again instead of the routing cache.
	These two commands allow the user to inspect the routing policy database and the local routing table to determine if the NAT routes and rules were added correctly.

[root@masq-gw]# ip rule add to 192.168.99.0/24 from 192.168.100.17
[root@masq-gw]# ip route flush cache
[root@masq-gw]# ip rule show
0:      from all lookup local 
32764:  from 192.168.100.17 to 192.168.99.0/24 lookup main 
32765:  from 192.168.100.17 lookup main map-to 205.254.211.17 
32766:  from all lookup main 
32767:  from all lookup 253
          

Because NAT rewrites the packet as it passes through the IP stack, packet filtering can become complex. With attentiveness to the addressing of the packet at each stage in its journey through the packet filtering code, you can ease the burden of writing a packet filter.

All of the below requirements can be deduced from an understanding of NAT and the path a packet takes through the kernel. Consult also the ipchains packet path as illustrated in the ipchains HOWTO to understand the packet path when using ipchains. Keep in mind when viewing the ASCII diagram that stateless NAT will always occur in the routing stage. Also consult the kernel packet traveling diagram for a good picture of a 2.4 kernel packet path.

Table 5.1 identifies the IP addresses on a packet traversing each of the input, forward and output chains in an ipchains installation.

A firewall implementing a tight policy (deny all, selectively allow) will require a large number of individual rules to allow the NAT packets to traverse the firewall packet filter. Assuming the configuration detailed in Example 5.1, the following set of chains is required and will restrict access to only port 25 ^[32].

[root@masq-gw]# ipchains -I input  -i eth1 -p tcp -l -y -s 0/0 1024:65535    -d 205.254.211.17 25 -j ACCEPT
[root@masq-gw]# ipchains -I input  -i eth1 -p tcp  ! -y -s 0/0 1024:65535    -d 205.254.211.17 25 -j ACCEPT
[root@masq-gw]# ipchains -I forward        -p tcp       -s 0/0 1024:65535    -d 192.168.100.17 25 -j ACCEPT
[root@masq-gw]# ipchains -I output -i eth0 -p tcp       -s 0/0 1024:65535    -d 192.168.100.17 25 -j ACCEPT
[root@masq-gw]# ipchains -I input  -i eth0 -p tcp  ! -y -s 192.168.100.17 25 -d 0/0 1024:65535    -j ACCEPT
[root@masq-gw]# ipchains -I forward        -p tcp       -s 205.254.211.17 25 -d 0/0 1024:65535    -j ACCEPT
[root@masq-gw]# ipchains -I output -i eth1 -p tcp       -s 205.254.211.17 25 -d 0/0 1024:65535    -j ACCEPT
[root@masq-gw]# for icmptype in \
> destination-unreachable source-quench time-exceeded parameter-problem; do
> ipchains -I input  -i eth1 -p icmp -s 0/0 $icmptype            -d 205.254.211.17 -j ACCEPT
> ipchains -I forward        -p icmp -s 0/0 $icmptype            -d 192.168.100.17 -j ACCEPT
> ipchains -I output -i eth0 -p icmp -s 0/0 $icmptype            -d 192.168.100.17 -j ACCEPT
> ipchains -I input  -i eth0 -p icmp -s 192.168.100.17 $icmptype -d 0/0            -j ACCEPT
> ipchains -I forward        -p icmp -s 205.254.211.17 $icmptype -d 0/0            -j ACCEPT
> ipchains -I output -i eth1 -p icmp -s 205.254.211.17 $icmptype -d 0/0            -j ACCEPT
> done
      

Please note that the formatting of the commands is simply for display purposes, and to allow for easier reading of a complex set of commands. The above set of rules is 31 individual chains. This is most certainly a complex set of rules. For further details on how to use ipchains please see the ipchains HOWTO. The salient detail you should notice from the above set of rules is the difference between the IPs used in the input and forward chains. Since packets are rewritten by the stateless NAT code in the routing stage, the transformation of the packet will by complete before the forward chain is traversed.

The first two lines cover all inbound TCP packets, the first line as a special case of the second, indicating (-l) that we want to log the packet. After successfully traversing the input chain, the packet is routed, at which point the destination address of the packet has changed. Now, we need to forward the packet from the public source address to the private (or real) internal IP address. Finally, we need to allow the packet out on the internal interface.

The next set of rules handles all of the TCP return packets. On the input rule, we are careful to match only non-SYN packets from our internal server bound for the world. Once again, the packet is rewritten during the routing stage. Now in the forward chain, the packet's source IP is the public IP of the service. Finally, we need to let the packet out on our external interface.

The next series of lines are required ICMP rules to prevent network traffic from breaking terribly. These types of ICMP, particularly destination unreachable (ICMP 3) and source quench (ICMP 4) help to ensure that TCP sessions run with optimized characteristics.

These rules are the minimum set of ipchains rules needed to support a NAT'd TCP service. This concludes our discussion of publishing a service to the world with iproute2 based NAT and protecting the service with ipchains. As you can see, the complexity of supporting NAT with iproute2 can be substantial, which is why we'll examine the benefits of inbound NAT (DNAT) with netfilter in the next section.

Destination NAT with netfilter is commonly used to publish a service from an internal RFC 1918 network to a publicly accessible IP. To enable DNAT, at least one iptables command is required. The connection tracking mechanism of netfilter will ensure that subsequent packets exchanged in either direction (which can be identified as part of the existing DNAT connection) are also transformed.

In a devilishly subtle difference, netfilter DNAT does not cause the kernel to answer ARP requests for the NAT IP, where iproute2 NAT automatically begins answering ARP requests for the NAT IP.

[root@real-server]# iptables -t nat -A PREROUTING -d 10.10.20.99 -j DNAT --to-destination 10.10.14.2
      

In this example, all packets arriving on the router with a destination of 10.10.20.99 will depart from the router with a destination of 10.10.14.2.

[root@real-server]# iptables -t nat -A PREROUTING -p tcp -d 10.10.20.99 --dport 80 -j DNAT --to-destination 10.10.14.2
      

Full network address translation, as performed with iproute2 can be simulated with both netfilter SNAT and DNAT, with the potential benefit (and attendent resource consumption) of connection tracking.

[root@real-server]# iptables -t nat -A PREROUTING -d 205.254.211.17 -j DNAT --to-destination 192.168.100.17
[root@real-server]# iptables -t nat -A POSTROUTING -s 192.168.100.17 -j SNAT --to-destination 205.254.211.17
      

Port address translation (hereafter PAT) provides a similar functionality to NAT, but is a more specific tool. PAT forwards requests for a particular IP and port pair to another IP port pair. This feature is commonly used on publicly connected hosts to make an internal service available to a larger network.

PAT will break in strange and wonderful ways if there is an alternate route between the two hosts connected by the port address translation.

PAT has one important benefit over NAT (with the iproute2 tools). Let's assume that you have only five public IP addresses for which you have paid dearly. Additionally, let's assume that you want to run services on standard ports. You had hoped to connect four SMTP servers, two SSH servers and five HTTP servers. If you had wanted to accomplish this with NAT, you'd need more IP space.

Packet filtering refers to the technique of conditionally allowing or denying packets entering or exiting a network or host based on the characteristics of that packet. There are two fundamental types of packet filters. A static packet filter is a set of rules against which every packet is checked, and allowed or denied.

A dynamic packet filter keeps track of the connections currently passing the firewall. This is usually described as a stateful or dynamic packet filtering engine. Netfilter provides the capability for linux (2.4+) to operate as a stateful packet filtering device.

The network layer portion of a firewall solution, packet filtering is one part of a good security stance. As the embodiment and manifestation of an organizational security policy for network layer traffic, the packet filter restricts traffic flows between networks and hosts. There is tremendous value from a security perspective in enforcing these traffic flows, instead of allowing arbitrary traffic flow.

The use of packet filtering to enforce these traffic flows is not restricted to routers and firewalls alone. Standalone servers and workstations can use these very same tools to protect themselves. There are a couple of common approaches to packet filtering. Generally, network security professionals subscribe to the notion that the filtering policy should deny or drop all traffic and selectively allow desired traffic. An alternate, more open, policy suggests allowing everything, selectively blocking undesirable traffic.

The languages used in most packet filtering tools for describing IP packets allow for a great deal of specifity when identifying traffic. This specifity enables an administrator a great deal of flexibility for protecting resources and limiting traffic flows.

Stateless packet filters. (cf. iptables connection tracking), cf. state vs. stateless discussion.

Use of ICMP, when to block ICMP; tunneling through lax packet filters with ICMP (trinoo, ICMPchat).

spoofed source addresses, xref binding non-local addresses

confounded application layer protocols like FTP, H323

DoS on connection tracking packet filters

DoS on rate limiters ?

minimum ICMP required to meet the networking needs; xref PMTU discussion

source quench

parameter problem

inbound destination unreachable

outbound destination unreachable fragmentation needed

optional: echo request and echo reply

optional: outbound destination unreachable

optional: time exceeded

packet filtering engine in kernel 2.2 (skip history, adequately documented elsewhere)

packet filtering engine as part of netfilter in kernel 2.4, backwards compatible support for ipchains

differences between the packet traversal in ipchains and iptables. link to Stef Coene's KPTD (kernel 2.4). Anybody know of a link to a KPTD for kernel 2.2?

the three builtin chains, input, output, forward

policy per chain, see targets

jumping from chain to chain, -j $TARGET; wher TARGET=chain

the big picture; how chains are traversed

targets (other than chains) ACCEPT, DENY, REJECT....

selecting on interface

Host protection in the past was typically performed with application layer checks on the originating IP or hostname. This was (and still is) frequently accomplished with libwrap, which verifies whether or not to allow a connection based on the contents of the system wide configuration files /etc/hosts.allow and /etc/hosts.deny.

Host protection is one part of protecting a host, by preventing inbound packets from reaching higher layers. This is no substitute for tight application layer security. Strong network and host-level packet filters mitigate a host's exposure when it is connected to a network.

[root@masq-gw]# iptables -I FORWARD -p tcp -d 209.10.26.51 --dport 22 -j REJECT
[root@tristan]# ssh 209.10.26.51
ssh: connect to address 209.10.26.51 port 22: Connection refused
[root@masq-gw]# tcpdump -nnq -i eth2
tcpdump: listening on eth2
22:16:59.111947 192.168.99.35.51991 > 209.10.26.51.22: tcp 0 (DF)
22:16:59.112270 192.168.99.254 > 192.168.99.35: icmp: 209.10.26.51 tcp port 22 unreachable (DF) [tos 0xc0]
      

The use of linux packet filtering features is mature and well-documented in many places throughout the Internet. One of the most thorough introductions to the use of iptables has been collected by Oskar Andreasson at his Iptables tutorial. For further reference material on the use of iptables consult this resource.

For those continuing to use ipchains the ipchains HOWTO courtesy of TLDP provides an introduction to the world of ipchains.

For kernel 2.4, understanding the sequence of packet mangling, filtering and network address translation is key. The kernel packet traveling diagram provides a visual representation of the path a packet takes through the kernel. Here you will see the netfilter hooks, traffic control, and routing stages. A similar picture of kernel 2.4's packet path is available in a single page PDF entitled Linux Kernel 2.4 Packet handling.

See also Section I.1.8, “ipchains Resources” and Section I.1.7, “Netfilter Resources” in the appendices for a more complete set of references and links.

ARP flux. /proc/sys/net/ipv4/conf/all/hidden Nothing here for now. Refer to Section 2.1.4, “The ARP Flux Problem”.

Media share; IP overlay; compare VLANS; consider bridging; consider migrating from one IP space to another (vrrpd, anybody?).

Proxy ARP is a technique for splitting an IP network into two separate segments. Hosts on one segment can only reach hosts in the other segment through the router performing proxy ARP. If a router sits between two parts of an IP network and is not running bridging software, then routes to hosts in each segment and proxy ARP are required on the router to allow each half of the network to communicate with the other half.

Occasionally, this technique is incorrectly called proxy ARP bridging. An Ethernet bridge operates on frames and a router operates on packets. The proxy ARP router should have routes to all hosts on both segments. Once the router can reach all locally connected destinations via the correct interfaces, you can begin to configure the proxy ARP functionality.

Although proxy ARP complicates a network, a great advantage of proxy ARP technique is the greater control over IP connections between hosts.

There are two primary proxy ARP techniques. With the 2.4 kernel, it is possible to use the sysctl net/ipv4/conf/all/proxy_arp to perform proxy ARP. Alternatively, manual population of the ARP table reaches the same end.

The key part of the correct functioning of proxy ARP in a network is that the host breaking a network into two parts has correct routes for all destinations in both halves of the network. If the host which has interfaces in both networks does not have an accurate routing table, IP packets will get dropped on the routing device.

One common method of breaking a network in two involves making a very small stub subnet at one end or the other of the IP range. This small subnet (maybe as small as a /30 network, with two usable IPs) makes an excellent sequestered location for a host which requires more protection or even, a generally untrusted host which shouldn't have complete access to the Ethernet to which the other machines connect.

For a practical example of this, see the relationship between the service-router, masq-gw and isolde in the network map. isolde and service-router share the same IP network, 192.168.100.0/24. If either has a packet for the other, it will generate an ARP request which should be answered by masq-gw. Naturally, masq-gw has its routes configured in such a way that both hosts are reachable from it. Thus, the packet will successfully pass through masq-gw.

Let's examine what the sequence of events is by which the packet will reach service-router from isolde. In this example, isolde will send an echo request packet to service-router. Please also refer to Section B.1, “arp” for examples and command lines to create a proxy ARP configuration.

Where possible, a simplified network is easier to maintain, but occasionally, this sort of trickery is necessary. This is an excellent way to insert a firewall into the middle of a network. The firewall, naturally, has to have its routes set properly, and proxy ARP entries will be required for routers.

Now, here's a short script and configuration file which can be run as a SysVInit style script. This script provides a great deal of control over the ARP table directly so may be preferable in some cases to an alternate solution outlined below. This proxy-arp script reads the following configuration file. Each is commented heavily so it should be clear how to use them.

This chapter discussed how to break a network in twain with proxy ARP techniques. For another explanation of the same concepts, read the Proxy ARP Subnet mini-HOWTO. Available in most (all?) 2.4 kernels is built-in capability for Proxy ARP. This is documented in deeper detail above. Consider familiarizing yourself with the methods of suppressing and controling ARP through Julian Anastasov's work.

Don't forget to add something here about multiple IPs bound to loopback; and refer to Julian's work. FIXME

Assume a machine has multiple connections to the same Ethernet segment, and has individual IPs bound to each interface. A peculiar feature of linux is its willingness to respond to ARP requests for any IP bound to any interface. This can lead to ARP flux, a situation where a given IP is sometimes accessed on one MAC address and sometimes another.

/proc/sys/net/ipv4/conf/all/hidden; consider arp suppression issues.

Consider ARP suppression issues. Leakage of sensitive (IP addressing) information from other interfaces.

FIXME!! Don't forget to note that iproute2 NAT and binding to non-local IPs do not play well together. I disagree with this. Binding to a non-local socket, which was possible under kernel 2.2 with when the kernel was compiled with CONFIG_IP_TRANSPROXY, is available under kernel 2.4 via the /proc IP sysctl interface. If you wish to be able to bind to non-local sockets:

# echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind

Thanks go to Oskar Andreasson for his IP sysctl tutorial page. If using sysctl to allow binding to non-local IP doesn't solve your problem, then see if netfilter NAT can be used to solve this class of problem. Some people view the technique of binding to non-local IPs as spoofing, and indeed, it can be used for nefarious purposes, if an attacker controls a machine on the route between a target and a victim.

One of the most difficult aspects of working with the advanced routing features of linux is gaining an understanding the sequence of events as a packet traverses the kernel space. It is, in fact, the key knowledge needed to grasp the potential of advanced routing scenarios and to troubleshoot successfully when things don't go as planned.

If you are reading this for the first time, stop now and go visit and study the kernel packet traveling diagram and the kernel packet handling diagram now. These represent two different efforts to describe the order in which different networking subsystems inside the linux kernel have an opportunity to inspect, manipulate and redirect a packet. Understanding this sequence of events is key to harnessing the power of linux networking.

Now, let's examine some of the different commands you can use to manipulate packets at each of these stages. The list below describes the sequence of events for a packet bound for a non-local destination.

The above describes the sequence of events for packets passing through the linux routing device. Let's look at a similar descriptions of the paths that packets bound for local destinations take through the kernel.

Naturally, packets need to go out from the machine as well, so let's look at the path for outbound packets which were locally generated.

Understanding and practically applying the knowledge of how and when to harness the routing features of linux is a matter of experience. The below is a set of examples for how to use the RPDB and multiple routing tables to solve different types of problems. These are but a few simple examples which allude to the flexibility and power available with the complex policy routing system under linux.

The questions summarized in this section should rightly be entered into the FAQ, since they are FAQs on the LARTC list.

There are many places where a linux based router/masquerading device can assist in managing multiple Internet connections. We'll outline here some of the more common setups involving multiple Internet connections and how to manage them with iptables, ipchains, and iproute2. One of the first distinctions you can make when planning how to use multiple Internet connections is what inbound services you expect to host and how you want to split traffic over the multiple links.

In the discussion and examples below, I'll address the issues involved with two separate uplinks to two different providers. I assume the following:

Additionally, I'll restrict my comments to statically assigned public IP address ranges unless I mention (in particular) dynamically allocated addresses.

In the following sections we'll look at the use of multiple Internet connections first in terms of outbound traffic only, then in terms of inbound traffic only. After that, we'll look at using multiple Internet connections for handling both inbound and outbound services.

[root@masq-gw]# ip route show table main
192.168.100.0/30 dev eth3  scope link
67.17.28.0/28 dev eth4  scope link
205.254.211.0/24 dev eth1  scope link
192.168.100.0/24 dev eth0  scope link
192.168.99.0/24 dev eth0  scope link
192.168.98.0/24 via 192.168.99.1 dev eth0
10.38.0.0/16 via 192.168.100.1 dev eth3
127.0.0.0/8 dev lo  scope link 
default via 205.254.211.254 dev eth1
[root@masq-gw]# ip route flush table 4
[root@masq-gw]# ip route show table main | grep -Ev ^default \
>   | while read ROUTE ; do
>     ip route add table 4 $ROUTE
> done
[root@masq-gw]# ip route add table 4 default via 67.17.28.14
[root@masq-gw]# ip route show table 4
192.168.100.0/30 dev eth3  scope link
67.17.28.0/28 dev eth4  scope link
205.254.211.0/24 dev eth1  scope link
192.168.100.0/24 dev eth0  scope link
192.168.99.0/24 dev eth0  scope link
192.168.98.0/24 via 192.168.99.1 dev eth0
10.38.0.0/16 via 192.168.100.1 dev eth3
127.0.0.0/8 dev lo  scope link 
default via 67.17.28.14 dev eth4
        

[root@masq-gw]# iptables -t mangle -A PREROUTING -p tcp --dport 80 -s 192.168.99.0/24 -j MARK --set-mark 4
[root@masq-gw]# iptables -t mangle -A PREROUTING -p tcp --dport 443 -s 192.168.99.0/24 -j MARK --set-mark 4
[root@masq-gw]# iptables -t mangle -nvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source                destination         
    0     0 MARK       tcp  --  *      *       192.168.99.0/24       0.0.0.0/0          tcp dpt:80 MARK set 0x4 
    0     0 MARK       tcp  --  *      *       192.168.99.0/24       0.0.0.0/0          tcp dpt:443 MARK set 0x4 

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
  pkts bytes target     prot opt in     out     source               destination
[root@masq-gw]# iptables -t nat -A POSTROUTING -o eth4 -j SNAT --to-source 67.17.28.12
[root@masq-gw]# iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 205.254.211.179
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 SNAT       all  --  *      eth4    0.0.0.0/0            0.0.0.0/0          to:67.17.28.12
    0     0 SNAT       all  --  *      eth1    0.0.0.0/0            0.0.0.0/0          to:205.254.211.179

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
        

[root@masq-gw]# ip rule add fwmark 4 table 4
[root@masq-gw]# ip rule show
0:      from all lookup local 
32765:  from all fwmark        4 lookup 4 
32766:  from all lookup main 
32767:  from all lookup 253
[root@masq-gw]# ip route flush cache
        

[root@masq-gw]# ip route add nat 67.17.28.10 via 192.168.100.10
[root@masq-gw]# ip rule add nat 67.17.28.10 from 192.168.100.10 table 4
[root@masq-gw]# ip route add nat 205.254.211.17 via 192.168.100.17
[root@masq-gw]# ip rule add nat 205.254.211.17 from 192.168.100.17
[root@masq-gw]# ip rule show
0:      from all lookup local 
32765:  from 192.168.100.17 lookup main map-to 205.254.211.17
32765:  from 192.168.100.10 lookup 4 map-to 67.17.28.10
32766:  from all lookup main 
32767:  from all lookup 253
[root@masq-gw]# ip route show table local | grep ^nat
nat 205.254.211.17 via 192.168.100.17  scope host 
nat 67.17.28.10 via 192.168.100.10  scope host
        

The proxy ARP script was written before the kernel supported proxy ARP natively. If you simply want proxy ARP to work, then you need only enable it in your 2.4 kernel. If you require more control than afforded by the kernel proxy ARP functionality and you wish to recompile iproute2 and your kernel, you can use the iproute2 extension, ip arp. Otherwise, you might try this script.

#! /bin/sh -
#
# proxy-arp	Set proxy-arp settings in arp cache
#
# chkconfig: 2345 15 85
# description: using the arp command line utility, populate the arp
#              cache with IP addresses for hosts on different media
#              which share IP space.
#  
# Copyright (c)2002 SecurePipe, Inc. - http://www.securepipe.com/
# 
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation, 
# Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
# -- written initially during 1998
#    2002-08-14; Martin A. Brown <mabrown@securepipe.com>
#      - cleaned up and commented extensively
#      - joined the process parsimony bandwagon, and eliminated
#        many unnecessary calls to ifconfig and awk
#

gripe () {  echo "$@" >&2;               }
abort () {  gripe "Fatal: $@"; exit 1;   }

CONFIG=${CONFIG:-/etc/proxy-arp.conf}
[ -r "$CONFIG" ] || abort $CONFIG is not readable

case "$1" in
  start)
        # -- create proxy arp settings according to
        #    table in the config file
        #
        grep -Ev '^#|^$' $CONFIG | {
          while read INTERFACE IPADDR ; do
            [ -z "$INTERFACE" -o -z "$IPADDR" ] && continue
            arp -s $IPADDR -i $INTERFACE -D $INTERFACE pub
          done
        }
	;;
  stop)
	# -- clear the cache for any entries in the
        #    configuration file
        #
        grep -Ev '^#|^$' /etc/proxy-arp.conf | {
          while read INTERFACE IPADDR ; do
            [ -z "$INTERFACE" -o -z "$IPADDR" ] && continue
            arp -d $IPADDR -i $INTERFACE
	  done
        }
	;;
  status)
        arp -an | grep -i perm
	;;
  restart)
	$0 stop
	$0 start
	;;
  *)
	echo "Usage: proxy-arp {start|stop|restart}"
	exit 1
esac

exit 0
#   
# - end of proxy-arp


      

#  
# Proxy ARP configuration file
#
# -- This is the proxy-arp configuration file.  A sysV init script
#    (proxy-arp) reads this configuration file and creates the
#    required arp table entries.
#
# Copyright (c)2002 SecurePipe, Inc. - http://www.securepipe.com/
# 
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation, 
# Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
#
# -- file was created during 1998
#    2002-08-15; Martin A. Brown <mabrown@securepipe.com>
#      - format unchanged
#      - added comments
#
# -- field descriptions:
#    field 1   this field contains the ethernet interface on which
#              to advertise reachability of an IP.
#    field 2   this field contains the IP address for which to advertise
#
# -- notes
#
#    - white space, lines beginning with a comment and blank lines are ignored
#
# -- examples
#
#    - each example is commented with an English description of the
#      resulting configuration
#    - followed by a pseudo shellcode description of how to understand
#      what will happen
#
# -- example #1; advertise for 10.10.15.175 on eth1
#
# eth1 10.10.15.175
#
# for any arp request on eth1; do
#   if requested address is 10.10.15.175; then
#     answer arp request with our ethernet address from eth1 (so
#       that the reqeustor sends IP packets to us)
#   fi
# done
#
# -- example #1; advertise for 172.30.48.10 on eth0
#
# eth0 172.30.48.10
#
# for any arp request on eth1; do
#   if requested address is 10.10.15.175; then
#     answer arp request with our ethernet address from eth1 (so
#       that the reqeustor sends IP packets to us)
#   fi
# done
#
# -- add your own configuration here


# -- end /etc/proxy-arp.conf
#   

      

The script will remove all NAT route entries and then all RPDB entries, other than the three default entries and anything saying "iif lo". It will then populate the RPDB and create NAT route entries according to the configuration file. Use this script with caution if you have customized your RPDB.

#! /bin/sh -
#
# nat;  start and stop network address translations using iproute2 tools
#
# chkconfig: 345 45 55
# description: iproute2 tools allow for sophisticated routing, network
#              address translation, and policy based routing.  This script
#              generalizes static NAT mappings and exceptions.
#  
# Copyright (c)2002 SecurePipe, Inc. - http://www.securepipe.com/
# 
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation, 
# Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
#
# -- written initially, 2002-03-02; -MAB
#    2002-08-14; Martin A. Brown <mabrown@securepipe.com>
#      - cleaned up and commented the code a bit
#      - altered the script to provide support for NAT from user-specified
#        networks instead of assuming that anything from 0/0 should be
#        translated
#    2002-08-30; Martin A. Brown <mabrown@securepipe.com>
#      - add configuration setting to flush all NAT rules and routes before
#        installing new rules and routes
#      - add a ./nat flush option
#    2003-01-31; Matthew Callaway <matt@securepipe.com>
#      - add validation routines
#    2003-02-05; Martin A. Brown <mabrown@securepipe.com>
#      - oversight identified by Shawn Balestracci; not all NAT rules
#        were flushed--we were looking only for map-to, not the exclude
#        rules as well

gripe () {  echo "$@" >&2;               }
abort () {  gripe "Fatal: $@"; exit 1;   }

CONFIG=${CONFIG:-/etc/sysconfig/static-nat}
[ -r "$CONFIG" ] || abort $CONFIG is not readable

function isIP () {
  # -- this function validates a variable as a valid IP address or CIDR network
  #
  VAR=$1

  echo ${VAR} | grep -Eq \
    "[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}(|[[:digit:]]{1,2})"

  [ $? -eq 0 ] && return 0
  return 1
}

function isINT () {
  # -- this function validates a variable as a valid integer
  #
  VAR=$1

  echo ${VAR} | grep -Eq \
    "[[:digit:]]{1,}"

  [ $? -eq 0 ] && return 0
  return 1
}

function validate () {
  grep -Ev '^#|^$' $CONFIG | while read NET NAT REAL NPRIO RPRIO EXCLUDE ; do
    # Fields 5 and 6 are optional
    if [ -z "$NET" -o -z "$NAT" -o -z "$REAL" -o -z "$NPRIO" ]; then
       echo Syntax error: Missing field: $NET $NAT $REAL $NPRIO $RPRIO $EXCLUDE
       exit 1
    fi
    if [ -n "$RPRIO" -a -z "$EXCLUDE" ]; then
       echo Syntax error: $NET $NAT $REAL $NPRIO $RPRIO $EXCLUDE
       echo Field 6 must be used with field 5
       exit 1
    fi
    for ITEM in $NET $NAT $REAL $EXCLUDE ; do
      isIP $ITEM
      if [ $? -ne 0 ]; then 
         echo "In line:"
         echo $NET $NAT $REAL $NPRIO $RPRIO $EXCLUDE
         echo $ITEM is not a valid IP or CIDR block
         exit 1
      fi
    done
    for ITEM in $NPRIO $RPRIO; do
      isINT $ITEM
      if [ $? -ne 0 ]; then 
         echo "In line:"
         echo $NET $NAT $REAL $NPRIO $RPRIO $EXCLUDE
         echo $ITEM is not an integer
         exit 1
      fi
    done
  done
}

function flush () {
  # -- this function should remove all NAT rules and routes
  #
  # -- remove all of the rules, except the three builtins and any IPSec
  #    rule; -MAB;
  #
  ip rule show | grep -Ev '^(0|32766|32767):|iif lo' \
    | while read PRIO NATRULE; do
    ip rule del prio ${PRIO%%:*} $( echo $NATRULE | sed 's|all|0/0|' )
  done
  # -- remove all of the rules
  #
  ip route show table local  | grep ^nat | while read NATROUTE; do
    ip route del $NATROUTE
  done
  ip route flush cache;
}

function nat () {
  grep -Ev '^#|^$' $CONFIG | while read NET NAT REAL NPRIO RPRIO EXCLUDE ; do

    # <-- set up the route for the NAT IP to turn it into the real IP
    #
    ip route add from $NET nat $NAT via ${REAL%%/*}
    [ "$?" -eq "0" ] || \
      gripe cmd failed: ip route add nat $NAT via ${REAL%%/*}

    # <-- establish the minimum routing policy database;
    #     this is required so that the outbound packet gets
    #     rewritten to be from the IP which sent us the packet
    #
    ip rule add to $NET nat ${NAT%%/*} from $REAL prio $NPRIO 
    [ "$?" -eq "0" ] || \
      gripe cmd failed: ip rule add nat ${NAT%%/*} from $REAL prio $NPRIO 

    # <-- determine if the user has supplied networks or address to be
    #     excluded from the $NETwork address above
    #
    [ ! -z "$RPRIO" ] && [ ! -z "$EXCLUDE" ] && {
      for NETWORK in $EXCLUDE ; do
        ip rule add from $REAL to $NETWORK prio $RPRIO
        [ "$?" -eq "0" ] || \
          gripe cmd failed: ip rule add from $REAL to $NETWORK prio $RPRIO
      done;
    }
  done;
  # <-- We don't want to forget to flush the cache, or the user will
  #     sit around wondering for the next few minutes why the NAT rules
  #     aren't working.  After flushing the cache, the NAT rules will
  #     work right away.
  #
  ip route flush cache;
}

# see how we were called
case "$1" in 
      start) validate && nat
             ;;
       stop) flush
             ;;
    restart) $0 stop; $0 start
             ;;
     status) ip route show table local | grep ^nat
             ip rule show | grep map-to
             ;;
          *) echo "usage: nat {start|stop|restart|status}"
             ;;
esac

#   
# - end of nat

      

#  
# NAT configuration file
#
# -- This file is used to configure NAT routes and rules
#    via the iproute2 package.  A sysV init script (nat)
#    uses this file to set up the routes/rules.
#
#
# Copyright (c)2002 SecurePipe, Inc. - http://www.securepipe.com/
# 
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation, 
# Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
#
# -- file created by Matt Callaway <matt@securepipe.com>
#    2002-03-01; Martin A. Brown <mabrown@securepipe.com>
#      - first major revision; added comments
#    2002-08-14; Martin A. Brown <mabrown@securepipe.com>
#      - cleaned up the file; added copious commenting and examples
#      - provided support for NAT only from specified networks (backwards
#        incompatibility added here; benefit is huge flexibility gain)
#    2003-02-10; Martin A. Brown <mabrown@securepipe.com>
#      - example #6 added.  Thanks for identification and description of
#        this scenario, and the example in the format of the other
#        examples go to Shawn Balestracci <shawnb@securepipe.com>
#
# -- field descriptions:
#    field 1   this field contains a network address.  Any packets from
#              this network will be translated according to fields two and
#              three, with the exception of any networks specified in fields
#              6 and higher
#    field 2   contains the NAT IP, the IP that only exists as a publicly
#              reachable IP for an internal host
#    field 3   contains the real IP of the machine, usually an internal IP
#    field 4   contains the priority for the NAT rule itself in the RPDB
#    field 5   contains the priority for the routing rule in the RPDB.  In
#              order for the internal networks to reach the real IP of the
#              server/host, this priority must be higher than the priority
#              for the NAT rule.  **lower numbers == higher priority**
#    field 6+  contains a whitespace separated list of networks which
#              should be able to reach the real IP (field 2) directly.
#              The entries into the rule policy database (RPDB) for these
#              networks will prevent packets from real-IP to dest-network
#              from being rewritten with the NAT IP as the source IP.
#              Networks specified here should be subnets of the network
#              specified in field 1.
#
# -- notes
#
#    - white space, lines beginning with a comment and blank lines are ignored
#    - field 5 should always be a lower number (higher priority) than field 4
#    - fields 5 and 6+ are optional
#    - fields 5 and 6+ must be used together, if used at all
#
# -- examples
#
#    - each example is commented with an English description of the network
#      address translation which will occur
#    - followed by a pseudo shellcode description of how to understand
#      exactly what the NAT will look like
#
# -- example #1; NAT a single IP from anywhere
#
# 0/0  10.10.0.14  172.31.254.1  1000
#
# for packets from any address (0/0);
#   if destination_address is 10.10.0.14 ; then
#      rewrite destination address from 10.10.0.14 to 172.31.254.1
#   fi
# done
#
# -- example #2; NAT an entire network (from anywhere)
#
# 0/0  10.13.0.0/16  172.17.0.0/16  1000
#
# for packets from any address (0/0); do
#   if destination_address is in 10.13.0.0/16 ; then
#      rewrite destination address from 10.13.x.x to 172.17.x.x
#   fi
# done
#
# -- example #3; NAT an entire network, but only from a specified nework
#
# 10.10.0.0/16  10.15.0.0/24  192.168.0.0/24  1000
#
# if packet is from 10.10.0.0/16 ; then
#    if destination_address is in 10.15.0.0/24 ; then
#       rewrite destination address from 10.15.0.x to 192.168.0.x
#    fi
# fi
#
# -- example #4; NAT an entire network, but only from a specified nework;
#                make an exception for certain IP ranges
#
# 10.10.0.0/16  10.15.2.0/24  192.168.2.0/24  1000  990  10.10.38.0/24
#
# if packet is from 10.10.0.0/16 and not from 10.10.38.0/24 ; then
#    if destination_address is in 10.15.2.0/24 ; then
#       rewrite destination address from 10.15.2.x to 192.168.2.x
#    fi
# fi
#
# -- example #5; NAT a single IP from anywhere; don't NAT if from specified
#                IP ranges
#
# 0/0  10.74.1.8  192.168.73.15  1000  990  192.168.71.0/24  192.168.70.0/24
#
# for packets from any address except 192.168.71.0/24 and 192.168.70.0/24; do
#    if destination_address is 10.74.1.8 ; then
#       rewrite destination address from 10.74.1.8 to 192.168.73.15
#    fi
# done
#
# -- example #6; NAT to the same IP differently based on the source
#                network IP ranges
#
# 0/0              10.74.1.8      192.168.73.15   1000
# 192.168.71.0/24  192.168.71.15  192.168.73.15    400
# 192.168.70.0/24  192.168.71.15  192.168.73.15    400
#
# N.B., the RPDB must traverse lines two and three first, hence the higher
#       priority.  If the source network is not 192.168.{71,70}.0/24 then
#       the we'll meet the next entry, 1000.
# N.B., the third entry in this example will cause an RTNETLINK: file
#       exists error, because there is already an entry in the local
#       routing table for 192.168.71.15 --NAT--> 192.168.73.15.  Known bug.
#
# for packets from 192.168.71.0/24 or 192.168.70.0/24; do
#       if destination_address is 192.168.71.15 ; then
#         rewrite destination address from 192.168.71.15 to 192.168.73.15
#       fi
# done
#
# for packets from any address except 192.168.71.0/24 and 192.168.70.0/24; do
#       if destination_address is 10.74.1.8 ; then
#         rewrite destination address from 10.74.1.8 to 192.168.73.15
#       fi
# done
#
# -- add your own configuration here

# -- end /etc/sysconfig/static-nat
#   

      

The first thing to consider whenever somebody reports a strange networking problem is any recent change. What has changed recently in the network? Have any new machines been added? Is the user using a service which was recently decommissioned? Did a machine (firewall, mail server, DNS resolver) recently reboot? Did all of the services restart?