Skip to content

Cisco Smart Install - IOS-XE Upgrade Caveat

This was created for and used with FreeZTP, but can likely be used with any Jinja2 templating system.

IOS-XE 3.7.4 cannot upgrade to 16.3.6 via smart-install because new force isn't appended. This workaround utilizes EEM applets in a Jinja2 switch template to download install the updated image.

Expand for Switch Upgrade Failure Log 
Would you like to enter the initial configuration dialog? [yes/no]:
Loading ztp_ios_upgrade from 172.17.251.251 (via Vlan1): !
[OK - 38 bytes]
Preparing install operation ...
[1]: Downloading file tftp://172.17.251.251/cat3k_caauniversalk9.16.03.06.
SPA.bin to active switch 1
[1]: Finished downloading file tftp://172.17.251.251/cat3k_caauniversalk9.16.03.06.
SPA.bin to active switch 1
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle cat3k_caa-universalk9.16.03.06.SPA.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle cat3k_caa-universalk9.16.03.06.SPA.bin
[1 2]: Verifying and copying expanded package files to flash:
[1 2]: Verified and copied expanded package files to flash:
[1 2]: Starting compatibility checks
[1]: % Candidate package compatibility checks failed because the following
package dependencies were not satisfied. Operation aborted.


[2]: % Candidate package compatibility checks failed because the following
package dependencies were not satisfied. Operation aborted.


[1]: % An internal error was encountered. Operation aborted.
[2]: % An internal error was encountered. Operation aborted.

ERROR: Software Installation Failed: 35 2

Loading network-confg from 172.17.251.251 (via Vlan1): !
[OK - 69 bytes]
Loading ZTP-23CFBA478F-confg from 172.17.251.251 (via Vlan1): !
[OK - 77012 bytes]

TAC confirmation

Case Notes

The behavior is expected due to the command syntax difference as you suspected. We documented the behavior in a bug below: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd49193. The running code 3.7.4E is affected and this issue is fixed on 3.6.8E.

Is it possible for you to try the following?

  1. Upgrade from 3.7.4E to 3.6.8E via smart install
  2. Upgrade from 3.6.8E to 16.3.6 via smart install

BugID_Screenshot_from_TAC

Considerations/Notes

  • Validated on Cisco 3850-12X48U-S switches with IOS-XE 3.7.4E installed out of the box; upgrading to IOS-XE 16.3.6.
  • Default TFTP blocksize is 512 on IOS-XE 3.7.4E, and 8192 on IOS-XE 16.3.6; adding this to the template significantly reduces the image transfer time.
  • The J2 template should not contain any configuration or syntax that is incompatible in IOS-XE version (3.7.4E). Any commands that are compatible with later IOS-XE versions only should be added to the post_ztp_2 applet.

Applet: post_ztp_1

  • Triggered by the switch receiving a DHCP address on Vlan1; notes regarding the maxrun and wait;

    • action 01.00 ... maxrun 900 - 15 minutes to accommodate the 2 minute wait, the 4-5 minute TFTP download, and the 5-6 minute install.
    • action 01.01 wait 120 - Vlan1 obtains a DHCP address approximately 1.5 minutes before the switch will allow configurations if stacked (see IOS-XE 3.7.4E Log below). This wait can be omitted for stand-alone switches.
    IOS-XE 3.7.4E Log
    *21:32:15.992: %DHCP-6-ADDRESS_ASSIGN: Interface Vlan1 assigned DHCP address 172.17.250.6, mask 255.255.254.0
...
*21:33:42.831: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*21:33:43.824: %RF-5-RF_TERMINAL_STATE: 1 ha_mgr:  Terminal state reached for (SSO)

Applet: post_ztp_2

  • Triggered by IOS-XE 16.x specific redundancy syslog message; notes regarding the maxrun and wait;

    • action 01.00 ... maxrun 600 - 10 minutes to accommodate the 2 minute wait, any optional configuration changes and the software package clean process.
    • action 01.01 wait 120 - Redundancy syslog message is logged approximately 1.75 minutes before the switch will allow configurations if stacked (see IOS-XE 16.3.6 Log below). This wait can be omitted for stand_alone switches.
    IOS-XE 16.3.6 Log
    21:54:16.002 PDT: %IOSXE_REDUNDANCY-6-PEER: Active detected switch 2 as standby.
...
21:56:00.711 PDT: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
21:56:01.735 PDT: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Process/Explanation

  1. Switch is powered up connected to the provisioning network and initiates smart-install. Switch requests an upgrade first but no image is downloaded since image download is disabled in FreeZTP.

  2. Switch requests config, FreeZTP gives a (merged) config containing the required config.

  3. Switch applies configuration with Vlan1 configured for DHCP addressing.

    • post_ztp_1 is triggered by Vlan1 obtaining a DHCP address.

  4. [EEM Applet post_ztp_1 loaded to memory.]

    1. Applet deletes itself from running config, downloads the bin file, cleans up temp configs and writes the startup-config.
    2. Switch then runs the software install command, answers y to reload prompt from install command.

  5. Switch reloads.

    • post_ztp_2 is triggered by a syslog command specific to IOS-XE 16.x (%IOSXE_REDUNDANCY-6-PEER).

      This trigger may need to be changed! I'm unsure if this syslog message is present when provisioning a stand-alone switch as I never tested this scenario. I will update if I get an opportunity to test this on a stand-alone switch.

  6. [EEM applet post_ztp_2 loaded into memory.]

    1. Applet deletes itself from running configuration and writes the startup-config.
    2. (Optional) Add action sequences for any IOS-XE 16.x specific commands.
    3. Applet runs the package clean process to delete the old .pkg and packages.conf file(s).

Required Config

  • Disable FreeZTP image downloads.

    Replace {{SCOPE}} with the name of your configured DHCP scope.

    $ ztp set dhcpd {{SCOPE}} imagediscoveryfile-option disable && \
~ztp request dhcpd-commit && \
~ztp service restart

  • Allocate a provisioning interface as prov_int; i.e. the interface connected to the provisioning network.

  • Modify the variables in the template config snippet below to suit network/needs, then add the whole snippet to the J2 switch template.

    These four variables can be defined in the keystore or left in the template.

    Variable Description
    tftp_addr Address of TFTP server, typically FreeZTP.
    access_vlan Vlan to configure on the provisioning interface (Gi1/0/48) after upgrade/reload is complete.
    prov_int Interface to be used for provisioning; e.g. Te1/0/48 (3850-12X48U-S interfaces 37-48 are TenGigabitEthernet.)
    image.bin|ver Name of the image file to download, and the image version short-hand.

Template Config Snippet

!-- EEM applet to upgrade switches accordingly (ALL SUBSEQUENT LINES ARE REQUIRED).
!---- {%set sw_count=idarray|count%}
!---- {%set tftp_addr="172.17.251.251"%}
!---- {%set access_vlan="501"%}
!---- {%set prov_int="Te1/0/48"%}
!---- {%set image={"bin":"cat3k_caa-universalk9.16.03.06.SPA.bin",
                   "ver":"16.3.6"}%}
!
!-- Required for EEM applet to function as intended.
logging buffered 20480 debugging
file prompt quiet
ip tftp blocksize 8192

!-- Required for TFTP transfers from FreeZTP (or other reachable TFTP server; i.e. `tftp_addr`).
interface Vlan1
  ip address dhcp
  no shutdown

!-- Interface that is connected to the provisioning network, must remain on Vlan1 for TFTP download.
interface GigabitEthernet1/0/48
  description TMP//PROVISION:Omit config; updated with post_ztp_2 EEM applet.
  switchport
  switchport mode access
  switchport nonegotiate
  switchport access vlan 1
  spanning-tree portfast
  no shutdown
!
event manager environment q "
!
event manager applet sw_upgrade
!-- Check all switches in stack to see if an upgrade is needed.
 event syslog occurs 1 pattern "Configured from tftp://{{tftp_addr}}" maxrun 960
 action 00.00 syslog msg "\n     ## Configuration received via TFTP, run 'sw_upgrade' EEM applet in 120s."
 action 00.01 wait 120
 action 00.02 syslog msg "\n     ## Checking all switches' version."
 action 00.03 cli command "enable"
 action 00.04 cli command "show mod | i ^.[1-9]"
 action 00.05 set stack "$_cli_result"
 action 00.06 syslog msg "\n     ## Current list of switches in stack;\n$stack"
 action 00.07 set error_list ""
 action 00.08 set upgrade_list ""
 !{% for sw in idarray %}
 !{% set i = loop.index %}
 action 0{{i}}.00 set sw_num "{{i}}"
 action 0{{i}}.01 set pri "16"
 action 0{{i}}.02 decrement pri {{i}}
 action 0{{i}}.03 regexp "{{sw}}" "$stack"
 action 0{{i}}.04 if $_regexp_result ne "1"
 action 0{{i}}.05  append error_list "\n     ##  {{sw}} is allocated (idarray_{{i}}) but was not found in the stack."
 action 0{{i}}.06 else
 action 0{{i}}.07  set i "0"
 action 0{{i}}.08  foreach line "$stack" "\n"
 action 0{{i}}.09   increment i
 action 0{{i}}.10   if $i le "{{sw_count}}"
 action 0{{i}}.11    string trim "$line"
 action 0{{i}}.12    set line "$_string_result"
 action 0{{i}}.13    regexp "{{sw}}" "$line"
 action 0{{i}}.14    if $_regexp_result eq "1"
 action 0{{i}}.15     regexp "([0-9\.A-Z]+$)" "$line" curr_ver
 action 0{{i}}.16     if $curr_ver ne "{{image.ver}}"
 action 0{{i}}.17      append upgrade_list "{{i}}"
 action 0{{i}}.18     end
 action 0{{i}}.19     break
 action 0{{i}}.20    end
 action 0{{i}}.21   end
 action 0{{i}}.22  end
 action 0{{i}}.23 end
 !{% endfor %}
 action 10.00 wait 5
 action 10.01 if $error_list ne ""
 action 10.02  syslog msg "\n     ## The following errors occurred; $error_list"
 action 10.03 end
 action 11.00 cli command "conf t"
 action 11.01 cli command "no event man app sw_upgrade"
 action 12.00 if $upgrade_list eq ""
 action 12.01  syslog msg "\n     ## All switches are running version {{target_ver}}, skipping download->upgrade/reload.\n     ## Finalizing config in 20s."
 action 12.02  cli command "no event man env q"
 action 12.03  cli command "event man env ztp_upgraded no"
 action 12.04  cli command " event man app post_upgrade"
 action 12.05  cli command " event timer countdown time 20 maxrun 480"
 action 12.06  cli command " no action 00.00"
 action 12.07  cli command " no action 00.01"
 action 12.08  cli command " end"
 action 12.09  cli command "write mem" pattern "confirm|#"
 action 12.10  cli command ""
 action 12.11 else
 action 12.12  syslog msg "\n     ## One or more switches require an upgrade to version {{target_ver}} ($upgrade_list).\n     ## Proceeding with download->upgrade/reload."
 action 12.13  cli command "event man env ztp_upgraded yes"
 action 12.14  cli command "event man app post_upgrade"
 action 12.15  cli command " event syslog occurs 1 pattern $q%IOSXE_REDUNDANCY-6-PEER$q maxrun 630"
 action 12.16  cli command " no event man env q"
 action 12.17  cli command "end"
 action 12.18  cli command "write mem" pattern "confirm|#"
 action 12.19  cli command ""
 action 12.20  syslog msg "\n     ## (Standby) Downloading image..."
 action 12.21  cli command "copy tftp://{{tftp_addr}}/{{image.bin}} flash:"
 action 12.22  syslog msg "\n     ## (Standby) Image downloaded, upgrading..."
 action 12.23  cli command "software install file flash:{{image.bin}} new force" pattern "proceed|#"
 action 12.24  syslog msg "\n     ## Upgrade complete, rebooting."
 action 12.25  cli command "y"
 action 12.26 end
 !
 event manager applet post_upgrade
!-- Clean up VL-1 and `prov_int` configs, write mem, and then perform package clean.
!-- (Optional) Add any desired configs between actions 03.00 and 04.00.
 event none
 action 00.00 syslog msg "\n     ## Switch reloaded on new image, running 'post_upgrade' EEM applet in 150s."
 action 00.01 wait 150
 action 00.02 syslog msg "\n     ## Applying global configs ignored by smart-install and generating crypto key."
 action 00.03 cli command "enable"
 action 00.04 set upgr "$ztp_upgraded"
 action 01.00 cli command "conf t"
 action 01.01 cli command "no event man env ztp_upgraded"
 action 01.02 cli command "no event man app post_upgrade"
 !-- action 02.00 cli command "" {# Use actions 02.00 - 02.99 for desired configs or configs specific to later IOS-XE version(s). #}
 action 03.00 syslog msg "\n     ## Disabling VL-1 SVI, updating Te#/0/48 configs, and writing startup config."
 action 03.01 cli command "int vl 1"
 action 03.02 cli command " no desc"
 action 03.03 cli command " no ip addr"
 action 03.04 cli command " shut"
 action 03.05 cli command "default range {{prov_int}}"
 action 03.06 cli command "int {{prov_int}}"
 !-- action 03.07 cli command "" {# Use actions 3.07 - 3.99 to configure `prov_int` as desired. #}
 action 04.00 cli command "end"
 action 04.01 cli command "write mem" pattern "confirm|#"
 action 04.02 cli command ""
 action 05.00 if $upgr eq "yes"
 action 05.01  syslog msg "\n     ## (Standby) ZTP upgrade detected, performing software package clean..."
 action 05.02  cli command "req plat soft pack clean sw all" pattern "proceed|#"
 action 05.03  cli command "y"
 action 05.04  syslog msg "\n     ## Unused .bin or .pkg files from previous version(s) have been deleted."
 action 05.05 else
 action 05.06  syslog msg "\n     ## ZTP upgrade not detected, skipping software package clean."
 action 05.07 end
 action 05.08 syslog msg "\n     ## Start-up config written, ({{hostname}}) is ready for deployment, OK to power off."

Comments