Router Configuration Management … Too Good to be True?
by Ivan Pepelnjak
If you’ve ever had to manage and configure more than a few routers in a production environment, you’ve probably stumbled across
questions like these:
Who changed the configuration on a router that stopped working? What was changed?
What’s the difference between the current configuration and the startup configuration?
Do we have a backup of a working configuration?
What was the router configuration before the last mistake was committed to the startup configuration?
Do I have a copy of the configuration that was used a month ago?
How do I rollback from the current mess to the startup configuration without reloading the router?
For the last 20 years, Cisco did almost nothing to help us, so router configuration management was a lucrative niche market for network management
vendors offering more and more complex tools with fancy graphic user interfaces. The landscape seemed to change radically with Cisco IOS release 12.4, which includes a number of router configuration management features, including Contextual Configuration Diff utility, Configuration Change Notification and Logging, Configuration archive and Configuration Replace and Rollback. They sounded almost too good to be true, so I decided to give them a few hard tasks. This article describes two of these features: the abilities of the Configuration Change Notification feature and the usage
guidelines for the Contextual Configuration Diff feature, their benefits as well as their shortcomings and current limitations.
What’s Going On?
Network managers who implemented centralized Authentication, Authorization and Accounting (AAA) with Cisco’s proprietary TACACS+ protocol long
had the ability to log any command executed on the routers in their network; the rest of us could only guess what someone
configuring our routers did to them. The Configuration Change Notification feature first introduced in IOS release 12.3(14)T
and integrated in mainstream release 12.4 solves this problem – after you configure it, all the configuration commands entered
on the router are stored in a circular buffer (you can even specify its length) and optionally sent to syslog server. A typical configuration is shown in Listing 1; you can get the description of individual commands in Cisco IOS documentation.
Listing 1
Configuration commands for Configuration Change Notification and Logging feature
archive
log config
logging enable
logging size 200
notify syslog
hidekeys
Note
The hidekeys command hides the passwords and other sensitive information in log buffer and syslog messages.
After you’ve configured the configuration change logging, all configuration commands are stored in a circular buffer in router’s
memory. You can inspect the commands with the show archive log config command, which displays all configuration commands recently entered on the router, or commands entered by a particular user or even within a particular
configuration session (from the moment you enter configure terminal to the time you exit the configuration mode). A sample printout of this command is shown in Listing 2.
Listing 2
Display of logged configuration commands
fw#show archive log config all
idx sess user@line Logged command
1 1 console@console | logging enable
2 1 console@console | logging size 200
3 1 console@console | notify syslog
4 2 console@console |archive
5 2 console@console | log config
6 2 console@console | hidekeys
If you’ve also configured the notify syslog option of the log config configuration command, all configuration commands entered on a router are also sent to the logging subsystem, which delivers them to various
logging destinations, including console and syslog hosts. The syslog messages usually contain the username and the configuration command, but they could also report changes
in significant data structures. For example, if you add a local user with the username command, the router will generate the two syslog messages in Listing 3.
Listing 3
Syslog messages generated by security-relevant configuration command
fw#conf t
Enter configuration commands, one per line. End with CNTL/Z.
fw(config)#user x password y
01:43:06: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:username x password *****
01:43:06: %PARSER-5-CFGLOG_LOGGEDCMD: User:console logged command:!config: USER TABLE MODIFIED
What’s the Difference?
If you don’t log the router configuration commands on an external syslog server, one of the first questions you’d ask when being presented with a malfunctioning router would be “What were the recent configuration changes on this box?” or “What made it stop working?”. The Contextual Configuration Diff feature introduced in IOS release 12.3(4)T and integrated into mainstream release 12.4 is supposed to answer these questions as long as you have a copy of a complete working configuration file. Startup configuration is usually a good starting point or you could use the configuration archive feature discussed in the next article to generate snapshots of the router configuration.
On the surface, this feature looks like a great idea – you give it two configuration files (they have to be complete and in
the format generated by the show running-config command) and the show archive config differences start-configuration target-configuration displays the IOS configuration commands (both removals and additions) needed to change the first file into the second one.
You can compare any two configuration files; most commonly you would compare the running configuration with the startup one with the show archive config differences nvram:startup-config system:running-config command, but you could also compare configurations stored in flash memory or on a remote server accessible via a variety of protocols (FTP, HTTP, SSL, RCP, SCP and TFTP), as seen in Listing 4.
Note
If you specify just one configuration file parameter, the IOS assumes it’s the target configuration and displays the differences
from the running-config to the target configuration (opposite from what I would expect).
Warning
The show archive config differences command syntax in Cisco IOS documentation for release 12.4 is wrong. The command will not accept startup-config or running-config as filenames but requires full file path (nvram:startup-config and system:running-config).
Listing 4
Compare running config with remote configuration file
fw#copy system:running-config tftp://10.0.0.2/fw-test
!!
2009 bytes copied in 0.592 secs (3394 bytes/sec)
fw#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
fw(config)#no access-list 100
fw(config)#access-list 120 permit ip any any
fw(config)#^Z
fw#show archive config differences tftp://10.0.0.2/fw-test system:running-config
Loading fw-test from 10.0.0.2 (via FastEthernet0/0): !
[OK - 2087 bytes]
Contextual Config Diffs:
+access-list 120 permit ip any any
-access-list 100 permit tcp any any eq www
-access-list 100 permit tcp any any eq telnet
-access-list 100 permit tcp any any eq smtp
Note
The file prompt quiet configuration command drastically reduces the clutter generated by the copy commands (see the first two lines of Listing 4 for a good example).
Warm-up Tests
I started testing this feature with a few simple test cases. As expected, IOS generated correct differences in all of them (see Table 1 and Listing 5 for a simple example).
Table 1
Simple configuration change
|
Startup configuration
|
Running configuration
|
|
interface Loopback0
ip address 10.1.0.1 255.255.255.255
|
interface Loopback0
description New loopback interface
ip address 10.1.0.3 255.255.255.255
|
Listing 5
Configuration difference after the changes made to Loopback0 interface
fw#show archive config differences nvram:startup-config system:running-config
Contextual Config Diffs:
interface Loopback0
+description New loopback interface
+ip address 10.1.0.3 255.255.255.255
interface Loopback0
-ip address 10.1.0.1 255.255.255.255
Even more interesting, when you remove a complete configuration section (for example, I’ve removed OSPF routing process),
the router lists all previously configured commands in the differences, making it very easy to identify exactly which settings
disappeared together with a major configuration object (see Table 2 and Listing 6)
Table 2
OSPF process is removed
|
Startup configuration
|
Running configuration
|
|
router ospf 101
log-adjacency-changes
network 10.1.0.0 0.0.1.255 area 2
network 0.0.0.0 255.255.255.255 area 0
|
no router ospf 101
|
Listing 6
Reported changes after OSPF process removal
fw#show archive config differences nvram:startup-config system:running-config
Contextual Config Diffs:
-router ospf 101
-log-adjacency-changes
-network 10.1.0.0 0.0.1.255 area 2
-network 0.0.0.0 255.255.255.255 area 0
By this time, I was tired of typing the same long command over and over again, so I decided to make an alias for it with the alias exec whatsnew show archive config differences nvram:startup-config system:running-config configuration command.
Getting Complex
Based on the first tests, it was obvious that the Contextual Configuration Diff feature correctly identifies most of the changes
made to IOS configuration. It was time to stress-test this feature with configuration structures that are order-sensitive.
I’ve started with traditional IP access lists (Table 3).
Table 3
Small change in extended numbered IP access list
|
Startup configuration
|
Running configuration
|
|
access-list 101 permit tcp any any eq www
access-list 101 permit icmp any any
|
Access-list 101 deny tcp host 10.0.0.2 host 192.168.0.2 eq www
access-list 101 permit tcp any any eq www
access-list 101 permit icmp any any echo
|
Even such a basic example exposed an inherent weakness in the contextual differences – while the removals and additions in
configuration files are correctly identified (but then any diff program would be able to do the same) you are not warned that the meaning of the access-list has changed drastically, as access list rules were inserted and/or reordered (Listing 7)
Listing 7
Reported changes in a traditional access list
fw#whatsnew
Contextual Config Diffs:
+access-list 101 deny tcp host 10.0.0.2 host 192.168.0.2 eq www
+access-list 101 permit icmp any any echo
-access-list 101 permit icmp any any
One would think that the named ip access-lists with their ability to have sequenced rules would offer greater possibilities to the contextual differences process. After
all, it would not be too hard to generate numbered rules that would be nicely inserted into their proper place in the access list. However, this is not how the process works; the results are the same as for traditional access lists (Table 4 and Listing 8)
Table 4
Changes in a named access list
|
Startup configuration
|
Running configuration
|
|
ip access-list extended Test
deny tcp host 10.0.0.2 host 192.168.0.2 eq www
permit tcp any any eq www
permit udp any host 192.168.0.2 eq domain
permit icmp any any
|
ip access-list extended Test
deny tcp host 10.0.0.3 host 192.168.0.2 eq www
permit tcp any any eq www
permit tcp any any eq ftp
permit udp any host 192.168.0.2 eq domain
permit icmp any any echo
deny icmp any any
|
Listing 8
Change report after the reconfiguration of a named access list
fw#whatsnew
Contextual Config Diffs:
ip access-list extended Test
+deny tcp host 10.0.0.3 host 192.168.0.2 eq www
+permit tcp any any eq ftp
+permit icmp any any echo
+deny icmp any any
ip access-list extended Test
-deny tcp host 10.0.0.2 host 192.168.0.2 eq www
-permit icmp any any
My last access-lists test involved a simple reordering of rules in a named access list (Table 5).
Table 5
Lines in a named access list are reordered
|
Startup configuration
|
Running configuration
|
|
ip access-list extended Test
deny tcp host 10.0.0.3 host 192.168.0.2 eq www
permit tcp any any eq www
permit tcp any any eq ftp
permit udp any host 192.168.0.2 eq domain
permit icmp any any echo
deny icmp any any
|
ip access-list extended Test
permit tcp any any eq ftp
deny tcp host 10.0.0.3 host 192.168.0.2 eq www
permit tcp any any eq www
permit udp any host 192.168.0.2 eq domain
permit icmp any any echo
deny icmp any any
|
The results produced by the router were a bit surprising (Listing 9) – it did correctly identify that the rules were reordered, but also indicated that two new rules were added to the access
list and none were removed (which is obviously incorrect).
Listing 9
Incomplete change report after a named access list line reordering
fw#whatsnew
Contextual Config Diffs:
ip access-list extended Test
+deny tcp host 10.0.0.3 host 192.168.0.2 eq www
+permit tcp any any eq www
!
!The following order-dependent line(s) were re-ordered
!ip access-list extended Test
! permit tcp any any eq ftp
Modular Failure
The final group of tests focused on Modular CLI commands (class-maps and policy-maps). As these structures are used to configure more and more Cisco IOS features (starting with Quality of Service, they are
now used for access control, firewall configuration and a variety of other IOS features), it’s very important that their changes
are correctly identified. I’ve started with an easy task – a few classes were added to a policy-map (Table 6).
Table 6
Changes to QoS classes and policy-map
|
Startup configuration
|
Running configuration
|
|
class-map match-all mail
match protocol smtp
class-map match-all web
match protocol http
!
policy-map WAN
class mail
police rate percent 20
conform-action transmit
exceed-action set-prec-transmit 0
exceed-action set-frde-transmit
class web
bandwidth percent 30
|
class-map match-all mail
match protocol smtp
class-map match-all ServerMail
match protocol smtp
match access-group 101
class-map match-all web
match protocol http
class-map match-all ServerWeb
match protocol http
match access-group 101
!
policy-map WAN
class ServerMail
priority 64
class ServerWeb
bandwidth percent 30
set precedence 3
class mail
police rate percent 20
conform-action transmit
exceed-action set-prec-transmit 0
exceed-action set-frde-transmit
class web
bandwidth percent 30
!
access-list 101 permit ip host 10.0.0.2 host 192.168.0.2
access-list 101 permit ip host 192.168.0.2 host 10.0.0.2
|
Not surprisingly, Cisco IOS correctly identified the additions to the running configuration, but did not warn me that the
new classes were inserted in front of the already-configured ones (Listing 10).
Listing 10
Correctly identified changes in a QoS policy-map
fw#whatsnew
Contextual Config Diffs:
+class-map match-all ServerMail
+match protocol smtp
+match access-group 101
+class-map match-all ServerWeb
+match protocol http
+match access-group 101
policy-map WAN
+class ServerMail
+priority 64
+class ServerWeb
+bandwidth percent 30
+set precedence 3
+access-list 101 permit ip host 10.0.0.2 host 192.168.0.2
+access-list 101 permit ip host 192.168.0.2 host 10.0.0.2
Similar to access-lists and route-maps, the policy-maps are order-dependent; the first class in the policy-map that matches the packet under inspection specifies the actions performed on the packet. My next test thus involved a misconfigured policy-map in startup-config that was fixed in the running configuration (Table 7).
Note
The ServerMail class is a subset of Mail class. If it’s placed after the Mail class in the policy-map it will match no traffic at all.
Table 7
Reordering of classes in a QoS policy-map
|
Startup configuration
|
Running configuration
|
|
policy-map WAN
class mail
police rate percent 20
conform-action transmit
exceed-action set-prec-transmit 0
exceed-action set-frde-transmit
class web
bandwidth percent 30
class ServerMail
priority 64
class ServerWeb
bandwidth percent 30
set precedence 3
|
policy-map WAN
class ServerMail
priority 64
class ServerWeb
bandwidth percent 30
set precedence 3
class mail
police rate percent 20
conform-action transmit
exceed-action set-prec-transmit 0
exceed-action set-frde-transmit
class web
bandwidth percent 30
|
Even though my expectations were a bit low after all the previous tests, the results nonetheless surprised me – the router reported no
changes in the configuration (Listing 11)
Listing 11
The router fails to identify reordered classes
fw#whatsnew
Contextual Config Diffs:
!No changes were found
Summary
The Cisco IOS release 12.4 gives you a variety of router configuration management tools. In this article, you’ve seen how you can use
the Configuration Change Notification and Logging feature to log all changes made to your router configuration. You can also use this feature to send all configuration commands entered on a router to an
external syslog server to ensure all changes to a router configuration are immediately stored in an off-site archive.
Note
If you want to use the configuration change logging feature as a security audit mechanism, you should configure syslog to use TCP as the transport protocol.
The Contextual Configuration Diff utility is supposed to give you a meaningful list of differences between two router configuration files. It’s definitely
a very useful utility, but don’t expect too much from it. While it correctly identifies the changed configuration objects
and lists additions and removals made to them, it ignores the order dependency of some IOS configuration commands (I’ve tested access-lists and policy-maps). The order dependency of these commands is sometimes completely ignored (although additions and removals are correctly identified),
sometimes you’re notified that the configuration commands were reordered but the additions and removals are incorrect, in worst cases (reordering of classes in a policy-map) the change is not detected at all.
However, even with these shortcomings (which I am positive Cisco will fix in later IOS releases), the Contextual Configuration Diff utility is a long-needed tool that will help us immensely in our day-to-day router management and troubleshooting.