Cisco Smart Install - IOS-XE Stack Auto-Allocation

This was created for and used with FreeZTP, but can likely be used with any Jinja2 templating system.

When this snippet (located at the bottom of this post) is added to a Jinja2 template it will automatically build out an EEM applet that will set switch priorities and renumber all switches in the stack according to how they were allocated in the FreeZTP keystore.

All switches in the stack can be powered on simultaneously; i.e. there is no need to do the 2-minute-tango between powering up switches. After the election processes are complete, the stack will continue booting and then perform the smart-install procedure where it will receive this applet as part of the templated configuration.

In this use-case, FreeZTP's idarray variables are being used to define stack member serial numbers. When the template is merged with the keystore, an action sequence is generated for each serial number (idarray_#) associated with the hostname (keystore_id). Switch serial numbers should be assigned to idarray_#'s as they're meant to be numbered in the stack (see the CSV example below).

Example

In this example, there are four switches allocated to the stack ASW-TR01-01;

FOC11111111 (idarray_1) should be switch 1 in the stack, with a priority of 15.
FOC22222222 (idarray_2) should be switch 2 in the stack, with a priority of 14.
FOC33333333 (idarray_3) should be switch 3 in the stack, with a priority of 13.
FOC44444444 (idarray_4) should be switch 4 in the stack, with a priority of 12.

CSV

Variables defined in the keystore (FreeZTP);

keystore_id = Hostname of the switch.
association = J2 template configured in FreeZTP.
idarray_# = Switch serial number(s).

Populate the idarray_# fields with serial numbers as they are to be ordered in the stack.

keystore_id,association,idarray_1,idarray_2,idarray_3,idarray_4,idarray_5,idarray_6,idarray_7,idarray_8,idarray_9
ASW-TR01-01,TEST,FOC11111111,FOC22222222,FOC33333333,FOC44444444,,,,,

Switch Priorities

The default priority for all switches (out of the box) is 1; these priorities do not change during the election process.

Notice that all switches have a priority of 1 in the output below. The priority range is 1-15; the (online) switch with highest priority will be chosen as the active 'supervisor' switch during any election processes/failure events.

show switch
Switch/Stack Mac Address : abcd.ef22.2222 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State 
----------------------------------------------------------−
*1       Active   abcd.ef22.2222     1      V02     Ready               
 2       Standby  abcd.ef44.4444     1      V02     Ready               
 3       Member   abcd.ef11.1111     1      V02     Ready               
 4       Member   abcd.ef33.3333     1      V02     Ready

Switch Numbers

The default number for all switches (out of the box) is also 1; as the switches boot they detect stack neighbors and perform an election process which renumbers the switches automatically.

The election process for this example resulted in the switches being numbered as follows;

show module
Switch  Ports    Model                Serial No.   MAC address     Hw Ver.       Sw Ver. 
------  -----   ---------             -----------  --------------  -------       --------
 1       62     WS-C3850-12X48U-S     FOC22222222  abcd.ef22.2222  V02           03.07.04E   
 2       62     WS-C3850-12X48U-S     FOC44444444  abcd.ef44.4444  V02           03.07.04E   
 3       62     WS-C3850-12X48U-S     FOC11111111  abcd.ef11.1111  V02           03.07.04E   
 4       62     WS-C3850-12X48U-S     FOC33333333  abcd.ef33.3333  V02           03.07.04E

Process/Explanation

Switches are connected via stack cables and powered up simultaneously.

An interface on only one of the switches needs to be connected to the provisioning network.
Switches complete the election process and stack initiates smart-install.
Stack requests config, FreeZTP gives a (merged) config containing the code.
- J2 variable sw_count is set during the merge process by counting serial numbers found in idarray.
Stack applies configuration and a syslog message is generated; sw_stack applet is triggered.
[EEM Applet sw_stack loaded to memory.]
1. Waits 120 seconds for the stack redundancy operations to complete.
2. Executes command show module | inc ^.[1-9] (output as read by EEM for current example);
  
  This output is what EEM stores as $stack for parsing. All line numbers correlate with the stack's current switch allocation numbers; i.e. first line contains information for switch 1, second line contains information for switch 2, etc...
```
 1       62     WS-C3850-12X48U-S     FOC22222222  abcd.ef22.2222  V02           03.07.04E   
 2       62     WS-C3850-12X48U-S     FOC44444444  abcd.ef44.4444  V02           03.07.04E   
 3       62     WS-C3850-12X48U-S     FOC11111111  abcd.ef11.1111  V02           03.07.04E   
 4       62     WS-C3850-12X48U-S     FOC33333333  abcd.ef33.3333  V02           03.07.04E   
ASW-TR01-01#
```
3. [J2 Loop] For each switch, searches the entire output for its serial number;
  - If not found, a syslog message will be generated and the applet will move onto the next switch.
  - [EEM Loop] If found, the applet searches the output line-by-line until it finds the serial number;
    
    The last line of the output is the switch hostname (elevated prompt), which is ignored when searching for switch serial number(s); i.e. the number of lines that the applet will search is limited by the Jinja2 sw_count variable set in the template.
    - If the line number where the serial number was found matches the allocated switch number (idarray_#), only the priority will be set.
    - If the line number where the serial number was found does not match the allocated switch number, the priority will be set and the switch will be renumbered.
4. Syslog messages are generated outlining any errors and all changes made to priorities and numbers.
5. Applet deletes itself from running configuration, writes the startup-config, and generates a syslog message stating that the process is complete.
The stack can now be reloaded to finish the renumbering process.

Snippet

!-- Variables (keys) parsed from CSV keystore.
!---- IDARRAY_1 (switch 1 serial number): {{idarray_1}}
!---- IDARRAY_2 (switch 2 serial number): {{idarray_2}}
!---- IDARRAY_3 (switch 3 serial number): {{idarray_3}}
!---- IDARRAY_4 (switch 4 serial number): {{idarray_4}}
!---- IDARRAY_5 (switch 5 serial number): {{idarray_5}}
!---- IDARRAY_6 (switch 6 serial number): {{idarray_6}}
!---- IDARRAY_7 (switch 7 serial number): {{idarray_7}}
!---- IDARRAY_8 (switch 8 serial number): {{idarray_8}}
!---- IDARRAY_9 (switch 9 serial number): {{idarray_9}}
!---- IDARRAY (all serials): {{idarray}}
!
!-- EEM applet to renumber switches accordingly (ALL SUBSEQUENT LINES ARE REQUIRED).
!---- SW_COUNT (count of serials found in IDARRAY): {%set sw_count=idarray|count%}{{sw_count}}
event manager applet sw_stack
  event syslog occurs 1 pattern "%SYS-5-CONFIG_I: Configured from tftp" maxrun 75
  action 00.00 syslog msg "\n     ## FreeZTP configuration received via TFTP, run 'sw_stack' EEM applet in 15s."
  action 00.01 wait 15
  action 00.02 cli command "enable"
  action 00.03 cli command "show mod | i ^.[1-9]"
  action 00.04 set stack "$_cli_result"
  action 00.05 syslog msg "\n     ## Checking all switches' version and stack membership, adjusting where necessary.\n     ## Current order;\n$stack"
  action 00.06 set error_list ""
  action 00.07 set change_list ""
  action 00.08 set upgrade_list ""
  {%for sw in idarray%}
  {%-  set i=loop.index%}
  action 0{{i}}.00 set sw_num "{{i}}"
  action 0{{i}}.01 set pri "16"
  action 0{{i}}.02 decrement pri {{i}}
  action 0{{i}}.03 regexp "{{sw}}" "$stack"
  action 0{{i}}.04 if $_regexp_result ne "1"
  action 0{{i}}.05  syslog msg "\n     ## {{sw}} (Sw-{{i}} serial) not found in the stack, check 'show mod' output."
  action 0{{i}}.06  append error_list "\n     ##  {{sw}} is allocated (idarray_{{i}}) but was not found in the stack."
  action 0{{i}}.07 else
  action 0{{i}}.08  set i "0"
  action 0{{i}}.09  foreach line "$stack" "\n"
  action 0{{i}}.10   increment i
  action 0{{i}}.11   if $i le "{{sw_count}}"
  action 0{{i}}.12    string trim "$line"
  action 0{{i}}.13    set line "$_string_result"
  action 0{{i}}.14    regexp "{{sw}}" "$line"
  action 0{{i}}.15    if $_regexp_result eq "1"
  action 0{{i}}.16     regexp "([0-9\.A-Z]+$)" "$line" curr_ver
  action 0{{i}}.17     cli command "switch $i priority $pri" pattern "continue|#"
  action 0{{i}}.18     cli command "y"
  action 0{{i}}.19     if $i eq $sw_num
  action 0{{i}}.20      append change_list "\n     ##  {{sw}} (Priority: $pri // Numbered:       $sw_num  // Version: $curr_ver)"
  action 0{{i}}.21     else
  action 0{{i}}.22      cli command "switch $i renumber $sw_num" pattern "continue|#"
  action 0{{i}}.23      cli command "y"
  action 0{{i}}.24      append change_list "\n     ##  {{sw}} (Priority: $pri // Renumbered: $i > $sw_num* // Version: $curr_ver)"
  action 0{{i}}.25     end
  action 0{{i}}.26     break
  action 0{{i}}.27    end
  action 0{{i}}.28   end
  action 0{{i}}.29  end
  action 0{{i}}.30 end
  {%endfor%}
  action 10.00 wait 5
  action 10.01 if $error_list ne ""
  action 10.02  syslog msg "\n     ## The following errors occurred; $error_list"
  action 10.03 end
  action 10.04 syslog msg "\n     ## Switches below have been assigned a priority and renumbered* as needed; $change_list"
  action 10.05 cli command "conf t"
  action 10.06 cli command "no event man app sw_stack"
  action 10.07 cli command "end"
  action 10.08 cli command "write mem" pattern "confirm|#"
  action 10.09 cli command ""
  action 10.10 syslog msg "\n     ## EEM applet (sw_stack) deleted and config written, reload for changes to take effect."
  !

Merged Config Example

Merged configuration that is pushed to the switch from FreeZTP for this example;
- J2 templating functions do not print; i.e. anything enclosed with {% %} will not appear in the merged config.
- Any line starting with a ! will be ignored by the switch.

Expand for Merged Config Example

!-- Variables (keys) parsed from CSV keystore.
!---- IDARRAY_1 (switch 1 serial number): FOC11111111
!---- IDARRAY_2 (switch 2 serial number): FOC22222222
!---- IDARRAY_3 (switch 3 serial number): FOC33333333
!---- IDARRAY_4 (switch 4 serial number): FOC44444444
!---- IDARRAY_5 (switch 5 serial number): 
!---- IDARRAY_6 (switch 6 serial number): 
!---- IDARRAY_7 (switch 7 serial number): 
!---- IDARRAY_8 (switch 8 serial number): 
!---- IDARRAY_9 (switch 9 serial number): 
!---- IDARRAY (all serials): ['FOC11111111', 'FOC22222222', 'FOC33333333', 'FOC44444444']
!
!-- EEM applet to renumber switches accordingly (ALL SUBSEQUENT LINES ARE REQUIRED).
!---- SW_COUNT (count of serials found in IDARRAY): 4
event manager applet sw_stack
  event syslog occurs 1 pattern "%SYS-5-CONFIG_I: Configured from tftp" maxrun 75
  action 00.00 syslog msg "\n     ## FreeZTP configuration received via TFTP, run 'sw_stack' EEM applet in 15s."
  action 00.01 wait 15
  action 00.02 cli command "enable"
  action 00.03 cli command "show mod | i ^.[1-9]"
  action 00.04 set stack "$_cli_result"
  action 00.05 syslog msg "\n     ## Checking all switches' version and stack membership, adjusting where necessary.\n     ## Current order;\n$stack"
  action 00.06 set error_list ""
  action 00.07 set change_list ""
  action 00.08 set upgrade_list ""
  
  action 01.00 set sw_num "1"
  action 01.01 set pri "16"
  action 01.02 decrement pri 1
  action 01.03 regexp "FOC11111111" "$stack"
  action 01.04 if $_regexp_result ne "1"
  action 01.05  syslog msg "\n     ## FOC11111111 (Sw-1 serial) not found in the stack, check 'show mod' output."
  action 01.06  append error_list "\n     ##  FOC11111111 is allocated (idarray_1) but was not found in the stack."
  action 01.07 else
  action 01.08  set i "0"
  action 01.09  foreach line "$stack" "\n"
  action 01.10   increment i
  action 01.11   if $i le "4"
  action 01.12    string trim "$line"
  action 01.13    set line "$_string_result"
  action 01.14    regexp "FOC11111111" "$line"
  action 01.15    if $_regexp_result eq "1"
  action 01.16     regexp "([0-9\.A-Z]+$)" "$line" curr_ver
  action 01.17     cli command "switch $i priority $pri" pattern "continue|#"
  action 01.18     cli command "y"
  action 01.19     if $i eq $sw_num
  action 01.20      append change_list "\n     ##  FOC11111111 (Priority: $pri // Numbered:       $sw_num  // Version: $curr_ver)"
  action 01.21     else
  action 01.22      cli command "switch $i renumber $sw_num" pattern "continue|#"
  action 01.23      cli command "y"
  action 01.24      append change_list "\n     ##  FOC11111111 (Priority: $pri // Renumbered: $i > $sw_num* // Version: $curr_ver)"
  action 01.25     end
  action 01.26     break
  action 01.27    end
  action 01.28   end
  action 01.29  end
  action 01.30 end
  
  action 02.00 set sw_num "2"
  action 02.01 set pri "16"
  action 02.02 decrement pri 2
  action 02.03 regexp "FOC22222222" "$stack"
  action 02.04 if $_regexp_result ne "1"
  action 02.05  syslog msg "\n     ## FOC22222222 (Sw-2 serial) not found in the stack, check 'show mod' output."
  action 02.06  append error_list "\n     ##  FOC22222222 is allocated (idarray_2) but was not found in the stack."
  action 02.07 else
  action 02.08  set i "0"
  action 02.09  foreach line "$stack" "\n"
  action 02.10   increment i
  action 02.11   if $i le "4"
  action 02.12    string trim "$line"
  action 02.13    set line "$_string_result"
  action 02.14    regexp "FOC22222222" "$line"
  action 02.15    if $_regexp_result eq "1"
  action 02.16     regexp "([0-9\.A-Z]+$)" "$line" curr_ver
  action 02.17     cli command "switch $i priority $pri" pattern "continue|#"
  action 02.18     cli command "y"
  action 02.19     if $i eq $sw_num
  action 02.20      append change_list "\n     ##  FOC22222222 (Priority: $pri // Numbered:       $sw_num  // Version: $curr_ver)"
  action 02.21     else
  action 02.22      cli command "switch $i renumber $sw_num" pattern "continue|#"
  action 02.23      cli command "y"
  action 02.24      append change_list "\n     ##  FOC22222222 (Priority: $pri // Renumbered: $i > $sw_num* // Version: $curr_ver)"
  action 02.25     end
  action 02.26     break
  action 02.27    end
  action 02.28   end
  action 02.29  end
  action 02.30 end
  
  action 03.00 set sw_num "3"
  action 03.01 set pri "16"
  action 03.02 decrement pri 3
  action 03.03 regexp "FOC33333333" "$stack"
  action 03.04 if $_regexp_result ne "1"
  action 03.05  syslog msg "\n     ## FOC33333333 (Sw-3 serial) not found in the stack, check 'show mod' output."
  action 03.06  append error_list "\n     ##  FOC33333333 is allocated (idarray_3) but was not found in the stack."
  action 03.07 else
  action 03.08  set i "0"
  action 03.09  foreach line "$stack" "\n"
  action 03.10   increment i
  action 03.11   if $i le "4"
  action 03.12    string trim "$line"
  action 03.13    set line "$_string_result"
  action 03.14    regexp "FOC33333333" "$line"
  action 03.15    if $_regexp_result eq "1"
  action 03.16     regexp "([0-9\.A-Z]+$)" "$line" curr_ver
  action 03.17     cli command "switch $i priority $pri" pattern "continue|#"
  action 03.18     cli command "y"
  action 03.19     if $i eq $sw_num
  action 03.20      append change_list "\n     ##  FOC33333333 (Priority: $pri // Numbered:       $sw_num  // Version: $curr_ver)"
  action 03.21     else
  action 03.22      cli command "switch $i renumber $sw_num" pattern "continue|#"
  action 03.23      cli command "y"
  action 03.24      append change_list "\n     ##  FOC33333333 (Priority: $pri // Renumbered: $i > $sw_num* // Version: $curr_ver)"
  action 03.25     end
  action 03.26     break
  action 03.27    end
  action 03.28   end
  action 03.29  end
  action 03.30 end
  
  action 04.00 set sw_num "4"
  action 04.01 set pri "16"
  action 04.02 decrement pri 4
  action 04.03 regexp "FOC44444444" "$stack"
  action 04.04 if $_regexp_result ne "1"
  action 04.05  syslog msg "\n     ## FOC44444444 (Sw-4 serial) not found in the stack, check 'show mod' output."
  action 04.06  append error_list "\n     ##  FOC44444444 is allocated (idarray_4) but was not found in the stack."
  action 04.07 else
  action 04.08  set i "0"
  action 04.09  foreach line "$stack" "\n"
  action 04.10   increment i
  action 04.11   if $i le "4"
  action 04.12    string trim "$line"
  action 04.13    set line "$_string_result"
  action 04.14    regexp "FOC44444444" "$line"
  action 04.15    if $_regexp_result eq "1"
  action 04.16     regexp "([0-9\.A-Z]+$)" "$line" curr_ver
  action 04.17     cli command "switch $i priority $pri" pattern "continue|#"
  action 04.18     cli command "y"
  action 04.19     if $i eq $sw_num
  action 04.20      append change_list "\n     ##  FOC44444444 (Priority: $pri // Numbered:       $sw_num  // Version: $curr_ver)"
  action 04.21     else
  action 04.22      cli command "switch $i renumber $sw_num" pattern "continue|#"
  action 04.23      cli command "y"
  action 04.24      append change_list "\n     ##  FOC44444444 (Priority: $pri // Renumbered: $i > $sw_num* // Version: $curr_ver)"
  action 04.25     end
  action 04.26     break
  action 04.27    end
  action 04.28   end
  action 04.29  end
  action 04.30 end
  
  action 10.00 wait 5
  action 10.01 if $error_list ne ""
  action 10.02  syslog msg "\n     ## The following errors occurred; $error_list"
  action 10.03 end
  action 10.04 syslog msg "\n     ## Switches below have been assigned a priority and renumbered* as needed; $change_list"
  action 10.05 cli command "conf t"
  action 10.06 cli command "no event man app sw_stack"
  action 10.07 cli command "end"
  action 10.08 cli command "write mem" pattern "confirm|#"
  action 10.09 cli command ""
  action 10.10 syslog msg "\n     ## EEM applet (sw_stack) deleted and config written, reload for changes to take effect."
  !

Validation

This has been confirmed functional on stacked C3850-12X48U-S switches running the following IOS-XE versions;
- IOS-XE 3.7.4E
- IOS-XE 16.3.6, 16.3.7

Logs

Below is an abbreviated and sanitized log output from 4 stacked switches, real serial numbers have been replaced by those in this example.

Would you like to enter the initial configuration dialog? [yes/no]: 
Loading network-confg from 172.17.251.251 (via Vlan1): !
[OK - 94 bytes]

Loading ZTP-23D9F46EC4-confg from 172.17.251.251 (via Vlan1): !
[OK - 154241 bytes]

...
*Oct 17 2018 12:45:34.347 PDT: %SYS-5-CONFIG_I: Configured from tftp://172.17.251.251/ZTP-23D9F46EC4-confg by console
*Oct 17 2018 12:45:34.365 PDT: %HA_EM-6-LOG: sw_stack: 
     ## FreeZTP configuration received via TFTP, run 'sw_stack' EEM applet in 15s.
...
*Oct 17 2018 12:47:34.376 PDT: %HA_EM-6-LOG: sw_stack: 
     ## Checking all switches' version and stack membership, adjusting where necessary.
*Oct 17 2018 12:47:44.755 PDT: %HA_EM-6-LOG: sw_stack: 
     ## Switches below have been assigned a priority and renumbered* as needed; 
     ##  FOC11111111 (Priority: 15 // Renumbered: 3 > 1* // Version: 03.07.04E )
     ##  FOC22222222 (Priority: 14 // Renumbered: 1 > 2* // Version: 03.07.04E )
     ##  FOC33333333 (Priority: 13 // Renumbered: 4 > 3* // Version: 03.07.04E )
     ##  FOC44444444 (Priority: 12 // Renumbered: 2 > 4* // Version: 03.07.04E )
*Oct 17 2018 12:47:52.732 PDT: %SYS-5-CONFIG_I: Configured from console by eem_svc on vty0 (EEM:post_ztp_1)
*Oct 17 2018 12:47:52.756 PDT: %HA_EM-6-LOG: sw_stack: 
     ## EEM applet `sw_stack` deleted and config written, reload for changes to take effect.
...

Cisco Smart Install - IOS-XE Stack Auto-Allocation

Example

CSV

Switch Priorities

Switch Numbers

Process/Explanation

Snippet

Merged Config Example

Validation

Logs

Cisco EEM: Auto Interface Descriptions

Cisco Smart Install - IOS-XE Upgrade Caveat