The BGP community attribute is an excellent tool for controlling and scaling BGP peering. Most big ISPs use the BGP communities for their internal policies and customer traffic engineering.

In Part 1 of the series, we discussed simple concepts on how the whole thing works and allowed to the customer to influence how our ISP will send him the traffic. This is ok for the simplest scenarios. In this Part 2, I will extend the basics and the internal working of such network design with some real-life customer services.

Prefix classifications - the implementation

Now that we know the scenario, we need to classify and tag all the prefixes received with communities for all the peers and assign a policy. We will also adjust the statically routed customers and our own prefixes. For the sake of the example, let's assume we have one upstream, IXP, and one peer. Our AS is 65432.

The community list and function description

The community configuration will be as follows:

ip community-list 10 permit 65432:1000
ip community-list 11 permit 65432:1100
ip community-list 15 permit 65432:1500
ip community-list 20 permit 65432:2000
ip community-list 21 permit 65432:2100
ip community-list 22 permit 65432:2200
ip community-list 30 permit 65432:3000
ip community-list 40 permit 65432:4000
ip community-list 50 permit 65432:5000

The idea is to tag all the inbound prefixes with a community describing the service and filter them on the outbound so that the customer gets the prefixes from the partner network and vice versa.

Internally originated prefixes

Our AS65432 has some prefixes of its own; it also originates some prefixes, that are not our own - for example, prefixes of a customer, that is running his own Provider Independent prefix. Such configuration can look similar to this:

router bgp 65432
 network 192.0.2.0 mask 255.255.255.0 route-map as65432-prefixes
 redistribute static route-map static-to-as65432
!
ip route 192.0.2.0 255.255.255.0 Null0
!
ip prefix-list as65432-block permit 192.0.2.0/24 le 32
!
route-map as65432-prefixes permit 10
 set community 65432:1000
!
route-map static-to-as65432 permit 10
 match ip address prefix-list as65432-block
 set community 65432:1100
!
route-map static-to-as65432 permit 20
 set community 65432:1500

If we break down the configuration:

• we are originating the prefix 192.0.2.0/24 with community 65432:1000 (aggregate)
• in case we introduce any static route, it will be processed by a route-map static-to-as65432 and the decision will be made if it belongs to us via prefix-list as65432-block and receives community 65432:1100 (AS 65432 sub-prefixes) or not and receives 65432:1500 (statically routed PI).

Please bear in mind, that there are already some opinionated configuration details - for example - I don't have a problem introducing redistribution of static routes, because it is my design decision, I am aware of all the consequences and will act accordingly. Also, pointing an aggregate to Null to get the prefix has some alternatives. My approach is to originate my own prefixes everywhere, I will have the more specific networks from the IGP. I am sending traffic that doesn't have a route from my IGP to Null (I don't want to forward traffic to a destination that doesn't exist).

Customer services

Example services

Our ISP offers its customers the following services:

• Full Transit (all routes)
• Full Transit (default plus AS65432)
• Upstream only (routes from the upstream providers)
• IXP only (routes from the Internet Exchange Point(s))
• BGP customers only (only routes of our directly connected customers)

For all the services, we will be adding our aggregates, our statically routed PI customers, and directly connected customers, as we don't want to utilize our upstream transit provider for this traffic when we can push it through the customer link.

From a scalability point of view, the easiest way to implement such a solution is to use peer-groups or peer-templates. I will use peer-groups.

Services templates implementation

• Full Transit (all routes) service definition.
  The customer is getting all the routes, from all sources (plus the default route)

router bgp 65432
 neighbor transit-full peer-group
 neighbor transit-full route-map transit-full-out out
 neighbor transit-full route-map transit-full-in in
!
route-map transit-full-out permit 10
 match ip community 10 15 20 21 22 30 40 50
!
route-map transit-full-in permit 10
 set community 65432:2000 65432:2100 65432:2200

• Full Transit (default route plus AS65432 aggregates) service definition.
  The customer is getting a default route and our aggregates. The alternative here would be to do the default-originate for the peer-group with some route-map, validating the functionality of the upstream connectivity.

router bgp 65432
 neighbor transit-default peer-group
 neighbor transit-default route-map transit-default-out out
 neighbor transit-default route-map transit-default-in in
!
route-map transit-default-out permit 10
 match ip community 10 50
!
route-map transit-default-in permit 10
 set community 65432:2000 65432:2100 65432:2200

• Transit-only (Just the upstream provider prefixes) service definition.
  The customer is getting just our upstream and our aggregates.

router bgp 65432
 neighbor transit-only peer-group
 neighbor transit-only route-map transit-only-out out
 neighbor transit-only route-map transit-only-in in
!
route-map transit-only-out permit 10
 match ip community 10 15 20 22 50
!
route-map transit-only-in permit 10
 set community 65432:2000

• IXP-only (Just the IXP prefixes) service definition.
  The customer is getting just IXP prefixes and our aggregates.

router bgp 65432
 neighbor ixp-only peer-group
 neighbor ixp-only route-map ixp-only-out out
 neighbor ixp-only route-map ixp-only-in in
!
route-map ixp-only-out permit 10
 match ip community 10 15 20 30
!
route-map ixp-only-in permit 10
 set community 65432:2100

• BGP-only (Just the customer prefixes) service definition.
  The customer is getting just other BGP customer prefixes, our aggregates, and PI prefixes.

router bgp 65432
 neighbor bgp-only peer-group
 neighbor bgp-only route-map bgp-only-out out
 neighbor bgp-only route-map bgp-only-in in
!
route-map bgp-only-out permit 10
 match ip community 10 15 20 22
!
route-map bgp-only-in permit 10
 set community 65432:2200

It is crucial to achieve symmetry, i.e., ensure that if you advertise customers' prefixes to a peer, they should get prefixes from that peer. If there is a mistake, you will face asymmetric routing issues.

Customer service activation

So let's assume we have one customer for each service. How will the service activation look after we are done with binging up the connectivity?

router bgp 65432
 neighbor a.a.a.a remote-as 200
 neighbor a.a.a.a peer-group transit-full
 neighbor a.a.a.a prefix-list pfx-as200-in
 neighbor b.b.b.b remote-as 300
 neighbor b.b.b.b peer-group transit-default
 neighbor b.b.b.b prefix-list pfx-as300-in
 neighbor c.c.c.c remote-as 400
 neighbor c.c.c.c peer-group transit-only
 neighbor c.c.c.c prefix-list pfx-as400-in
 neighbor d.d.d.d remote-as 500
 neighbor d.d.d.d peer-group ixp-only
 neighbor d.d.d.d prefix-list pfx-as500-in
 neighbor e.e.e.e remote-as 600
 neighbor e.e.e.e peer-group bgp-only
 neighbor e.e.e.e prefix-list pfx-as600-in

Customers are placed into the correct peer-group according to the service they are buying. As an additional security measure, we apply the prefix-list filtering on the inbound to the customers.

Note: Originating the default route

There is an everlasting debate about whether and how to originate the default route and how to check it is functional. Dummy generating of the default route to the customers can cause serious issues in case your upstream connectivity fails, as the route gets generated no matter what, and the traffic will get blackholed. It is wise to implement some sort of test that the upstream is alive. This topic is outside the scope of this article, you can read more about this topic in a great article by Ivan Pepelnjak here.

Summary

In this Part 2 of the series, we established a connectivity service offering and how to implement it with a Cisco router. In the next part, we will look at the economy of the ISP and how it translates to our community design.

The BGP community attribute is an excellent tool for controlling and scaling BGP peering. Most big ISPs use the BGP communities for their internal policies and customer traffic engineering.

I would like to go through some basic design options and principles in this article.

The use of BGP community attribute for multi-homed scenarios is described in RFC1998. This document is quite aged but still contains valuable information we can build on. The RFC1998 has been extended for use with BGP Large Communities with RFC8092 and RFC8195. However, BGP Large communities are outside of the scope of this post. Implementation should be quite similar, though.

Simple scenario - dual-homing

RFC1998 uses an example of MCI, how to do load-sharing and backup with multiple upstream AS links. The BGP communities, in this case, determine the local preference in the upstream network. This, in turn, gives control to the customer, who can then use a special community himself, without having to call his ISP technical support. Which is great for both parties :)

As a basis, the RFC1998 uses an example of MCI network, for example:

If this is implemented on the ISP PE device, the customer can control how the ISP will handle his traffic. As an example. The customer is dual-homed to AS100 and wants to use that particular path as a backup path, he would announce his prefix to that peer with a community of 100:90. The ISP would then, based on this received community, set the local preference for this prefix to be 90.

Sample configuration of the customer router (Cisco IOS, IOS-XE). Please note this is just a minimal example. The customer originates only his own prefixes (via the as-path ACL) and sets the community as described above.

router bgp 200
 address-family ipv4 unicast
  neighbor 192.0.2.1 remote-as 100
  neighbor 192.0.2.1 description ISP backup path
  neighbor 192.0.2.1 route-map as100-out out
  neighbor 192.0.2.1 send-community
  neighbor 192.0.2.1 activate
! 
ip as-path access-list 10 permit ^$
!
route-map as100-out permit 10
 match as-path 10
 set community 100:90

So now the customer is sending his prefix with a community of 100:90. How is the configuration in the ISP PE router?

router bgp 100
 address-family ipv4 unicast
  neighbor 192.0.2.2 remote-as 200
  neighbor 192.0.2.2 route-map customer-in in
  neighbor 192.0.2.2 activate
!
! Homed to another ISP
ip community-list 7 permit 100:70
! Homed to another ISP with same ASPATH length
ip community-list 8 permit 100:80
! Customer backup routes
ip community-list 9 permit 100:90
!
route-map customer-in permit 10
 match community 7
 set local-preference 70
!
route-map customer-in permit 20
 match community 8
 set local-preference 80
!
route-map customer-in permit 30
 match community 9
 set local-preference 90
!
route-map customer-in permit 40
 set local-preference 100

From the ISP perspective, this looks more complicated than the usual per-customer peering and configuration, but it scales better in the long run.

This was a simple example based on RFC1998. It is probably ok for very simple dual-homed situations. Typical ISP has to handle more and more complex situations. Therefore they need to have a more complex policy that will ease the operations and provide their customers with more policy control.

Typical ISP usage

A more complex scenario would allow more control for the customer. In this case, we would like to allow customers to change local preference and our as-prepending towards our upstream providers. There can also be a situation when the customer is under attack. In that case, he can force us to blackhole all the traffic coming to his network with Remote-Triggered Black Hole (RTBH) functionality.

The actual example implementation of this is going to look like this (in the example, we have one customer and one upstream provider.

router bgp 100
 address-family ipv4 unicast
  neighbor 192.168.0.2 remote-as 200
  neighbor 192.168.0.2 route-map customer-in in
  neighbor 192.168.0.2 activate
  neighbor 10.0.0.2 remote-as 300
  neighbor 10.0.0.2 route-map upstream-out out
  neighbor 10.0.0.2 activate
!
ip community-list 1 permit 100:1
ip community-list 2 permit 100:2
ip community-list 3 permit 100:3
ip community-list 80 permit 100:80
ip community-list 120 permit 100:120
ip community-list 666 permit 100:666
!
!
ip route 192.0.2.1 255.255.255.255 Null0
!
!
route-map customer-in permit 10
 match community 80
 set local-preference 80
!
route-map customer-in permit 20
 match community 120
 set local-preference 120
!
route-map customer-in permit 30
 match community 666
 set ip next-hop 192.0.2.1
!
route-map customer-in permit 40
 set local-preference 100
!
!
route-map upstream-out permit 10
 match community 1
 set as-path prepend 100
!
route-map upstream-out permit 20
 match community 2
 set as-path prepend 100 100
!
route-map upstream-out permit 30
 match community 3
 set as-path prepend 100 100 100
!
route-map upstream-out permit 40
! etc., etc.
  

In summary, what can the customer do with such a configuration on his ISP side?

• Change local preference on the ISP side, by using communities 100:80 or 100:120, which can change which path the ISP uses to send the traffic to the customer. This is useful for maintenance - the customer changes his primary line himself without any outage caused by waiting for BGP timers to time out.
• Change ISP prepend behavior towards his upstream provider with community 100:1, 100:2, and 100:3 - can be useful in multi-homed scenarios to influence the prefix behavior in the internet (i.e., we want the traffic to go via the other ISP we have).
• Black-hole a prefix in case he is under DoS attack so that it is no longer reachable with RTBH triggered by the use of the 100:666 community.

Most Internet Service Providers publish their peering policy and list the communities you can use with descriptions of what it is doing. For example, here is the Verizon Business BGP policy as published in RIPE DB. We will cover most of these in this series.

VzBi uses the following communities with its customers: 
702:80 Set Local Pref 80 within AS702 
702:120 Set Local Pref 120 within AS702 
702:20 Announce only to VzBi AS'es and VzBi customers 
702:30 Keep within Europe, don't announce to other VzBi AS's 
702:1 Prepend AS702 once at edges of VzBi to Peers 
702:2 Prepend AS702 twice at edges of VzBi to Peers 
702:3 Prepend AS702 thrice at edges of VzBi to Peers 

Advanced communities for customers:
702:7020 Do not announce to AS702 peers with a scope of National but advertise to Global Peers, European Peers and VzBi customers. 
702:7001 Prepend AS702 once at edges of VzBi to AS702 peers with a scope of National. 
702:7002 Prepend AS702 twice at edges of VzBi to AS702 peers with a scope of National. 
702:7003 Prepend AS702 thrice at edges of VzBi to AS702 peers with a scope of National. 
702:8020 Do not announce to AS702 peers with a scope of European but advertise to Global Peers, National Peers and VzBi customers. 
702:8001 Prepend AS702 once at edges of VzBi to AS702 peers with a scope of European. 
702:8002 Prepend AS702 twice at edges of VzBi to AS702 peers with a scope of European. 
702:8003 Prepend AS702 thrice at edges of VzBi to AS702 peers with a scope of European.

Summary

This was a brief introduction to a basic BGP peering policy with BGP communities. In the next posts, we will look at further possibilities of using communities for internal traffic engineering and external service design and scaling.

Cisco ISR1000 is a popular router series aimed at branch deployment, and it contains many models depending on the required deployment. You can pick a router with a DSL WAN interface, GigabitEthernet, dual-SIM 4G interfaces, Wi-Fi, and PoE. A ton of options there.

I won't go into the details of all the devices in this post, and I will cover one aspect of licensing that I found complicated: licensing. To not dive deep into the number of models available, I will keep this blog post limited to Cisco C1100 models - C1100-8P and C1100-4P as these are the basis for the other alternatives, and the approach to licensing should be similar.

Step 1 - Choosing the right platform.

With ISR1100, there are two parameters we should check.

1. The number of LAN-side interfaces - this parameter is easy - the routers have 4 or 8 LAN switch interfaces.
2. Throughput - the 8-port variant has a more powerful CPU. Details of the performance characteristic are below, and I will elaborate on the differences.
3. Crypto throughput - this is an area where I got lost because of the misleading Cisco documentation.

And some real-life test figures:

Some observations from this summary:

• The CPU frequency directly translates to throughput figures.
• The LAN-side switch connects to the backplane with a 1G uplink in C1100-4P and a 2.5G uplink in C1100-8P. It can be a limitation in some use cases because of the oversubscription.

Step 2 - Choosing the correct feature set.

Cisco routers have a long history of using feature licenses, and the point is to get (and pay for) the licenses you will use. In the case of C1100, there are two main branches of licenses.

There are three feature licenses available:

• IP Base - this technology package is a default option and contains basic features like routing protocols, ACL, NAT, QoS, BFD, VRF Lite, and IP SLA Responder.
• APP - this technology package contains everything from the IP Base package and adds advanced networking protocols: L2TPv3 and MPLS. It also has Application Experience functionalities like PfRv3, NBAR2, AVC, and IPSLA Initiator. For the hybrid cloud connectivity, it adds LISP, Virtual Private LAN Services (VPLS), and Ethernet over MPLS.
• SEC - this technology package adds to IP Base functionality security and cryptographical features and protocols: Zone-based firewall, IPsec VPN, Dynamic Multipoint VPN (DMVPN), FlexVPN, and GETVPN. The router default limits the encrypted traffic throughput to 50 Mbps, where the IPSec performance and HSEC licenses come in.

Step 3 - Choosing the correct IPSec performance.

If you have the SEC feature set, the router, by default, limits the encrypted traffic throughput to 50 Mbps. If you need more, you need to add a Security Performance license. This license comes in two variants, VPERF, which raises the throughput level, and HSEC, which removes the performance restriction completely.

For C1100-4P, the VPERF license adds 100 Mbps for 150 Mbps of total crypto throughput.

For C1100-8P, the VPERF license adds 200 Mbps for 250 Mbps of total crypto throughput.

If even this limit isn't enough, then there is an HSEC feature license, which removes the limit, and the performance is limited just by the performance of the device itself. From the performance figures higher in this article, it is reasonable to expect 230 Mbps of crypto throughput for C1100-4P and 335 Mbps for C1100-8P.

Step 4: How to apply the license to a router

After ordering and receiving the router with desired licenses, you will need to apply the license to the device. We are using the Smart Licensing portal for the portability of the licenses.

In this part of the post, I will show the options we have with the router's configuration.

Prerequisites:

First of all, we need to establish some basic connectivity to the internet.

licensetest(config)#interface GigabitEthernet0/0/0
licensetest(config-if)#ip address 192.0.2.2 255.255.255.0
licensetest(config-if)#no ip redirects
licensetest(config-if)#no ip unreachables
licensetest(config-if)#no ip proxy-arp
licensetest(config-if)#load-interval 30
licensetest(config-if)#negotiation auto
licensetest(config-if)#exit
licensetest(config)#ip route 0.0.0.0 0.0.0.0 192.0.2.1

We will also need a DNS service to be able to resolve the Smart Portal name.

licensetest(config)#ip name-server 8.8.8.8 8.8.4.4 
licensetest(config)#ip domain-lookup

And it will be reasonable to have some other name for the device than Router.

licensetest(config)#hostname licensetest ip domain-name example.com

Now we can safely proceed to connect the device to the Smart Account. For this, we need to know the idtoken we can get in the software.cisco.com Smart Software Manager. As we may be adding some export-controlled functionalities, we need to select the according field for the token.

After we have the token, we need to tell the device to use it for registration. But first, we need to change the device's default behavior, which is to use Cisco Smart License Utility (CSLU), to use a Cisco Smart Software Manager (CSSM).

licensetest(config)#license smart url https://smartreceiver.cisco.com/licservice/license
licensetest(config)#license smart url smart https://smartreceiver.cisco.com/licservic/license
licensetest(config)#license smart transport smart

Then we can register the device (this is done in the exec mode, not config).

licensetest(config)#license smart trust idtoken {{token}}

After we enter this command, the router tries to register itself. If all goes well, we should get this output of show license all:

licensetest#show license all
Smart Licensing Status
======================

Smart Licensing is ENABLED

!!! OUTPUT REMOVED FOR BREWITY

Trust Code Installed: Oct 14 06:11:40 2022 UTC

License Usage
=============

Product Information
===================
UDI: PID:C1111-8P,SN:XXXXXXXXXXXX

Agent Version
=============
Smart Agent for Licensing: 5.0.14_rel/89

License Authorizations
======================
Overall status:
  Active: PID:C1111-8P,SN:XXXXXXXXXXXX
      Status: SMART AUTHORIZATION INSTALLED on Oct 14 06:13:48 2022 UTC
      Last Confirmation code: b2edbdac

Purchased Licenses:
  No Purchase Information Available

The device is connected to the Smart Account portal. Now we need to apply some licenses to it.

Option 1: SEC license only

In this case, if we get a device without any preinstalled license, we need to add one command and reboot the router.

licensetest(config)#license boot level securityk9
licensetest(config)#exit
licensetest#write memory
licensetest#reload

This will result in a change in the show license all output:

License Usage
=============

securityk9 (ISR_1100_8P_Security):
  Description: securityk9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: securityk9
  Feature Description: securityk9
  Enforcement type: NOT ENFORCED
  License type: Perpetual

You will also see the license is "In Use" in the Smart Account "Licenses" view:

Option 2: SEC license and HSEC license

In this case, if we get a device without any preinstalled license, we need to add two commands and reboot the device.

licensetest(config)#license boot level securityk9 
licensetest(config)#platform hardware throughput crypto unthrottled 
licensetest(config)#exit 
licensetest#write memory
licensetest#reload

The platform hardware throughput crypto unthrottled is, per my observation, equivalent to a commandlicense feature hseck9installed automatically. You have, therefore, two options to configure the HSEC functionality. However, I recommend you do the configuration with platform hardware throughput crypto. See the detail in part "Caveats."

Again, this will result in a change in the show license all output:

License Usage
=============

hseck9 (ISR_1100_8P_Hsec):
  Description: hseck9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: RESTRICTED - ALLOWED
  Feature Name: hseck9
  Feature Description: hseck9
  Enforcement type: EXPORT RESTRICTED
  License type: Perpetual

securityk9 (ISR_1100_8P_Security):
  Description: securityk9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: securityk9
  Feature Description: securityk9
  Enforcement type: NOT ENFORCED
  License type: Perpetual
 
License Authorizations
======================
Overall status:
  Active: PID:C1111-8P,SN:XXXXXXXXXXX
      Status: SMART AUTHORIZATION INSTALLED on Oct 14 06:13:48 2022 UTC
      Last Confirmation code: b2edbdac

Authorizations:
  ISR_1100_8P_Hsec (ISR_1100_8P_Hsec):
    Description: Cisco 1100 Series with 8 LAN Ports, U.S. Export Restriction Compliance license
    Total available count: 1
    Enforcement type: EXPORT RESTRICTED
    Term information:
      Active: PID:C1111-8P,SN:XXXXXXXXXXX
        Authorization type: SMART AUTHORIZATION INSTALLED 
        License type: PERPETUAL
          Term Count: 1

You will also see the license being consumed in the Smart Account "Licenses" view:

Option 3 - SEC license and Performance license

In this case, if we get a device without any preinstalled license, we need to add two commands and reboot the device.

We will apply the desired crypto throughput level depending on the device platform. For C1100-4P, it is going to be:

licensetest#conf t
licensetest(config)#license boot level securityk9 
licensetest(config)#platform hardware throughput crypto 150000
licensetest(config)#exit 
licensetest#write memory
licensetest#reload

For C1100-8P, it is going to be:

licensetest#conf t
licensetest(config)#license boot level securityk9 
licensetest(config)#platform hardware throughput crypto 250000
licensetest(config)#exit 
licensetest#write memory
licensetest#reload

We can see this license is IN USE with show license all command. I don't have this license, so that the output will show failed authorization.

License Usage
=============

throughput (ISR_1100_8P_IPSEC_Throughput_200Mbps):
  Description: throughput
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: throughput
  Feature Description: throughput
  Enforcement type: NOT ENFORCED
  License type: Perpetual

securityk9 (ISR_1100_8P_Security):
  Description: securityk9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: securityk9
  Feature Description: securityk9
  Enforcement type: NOT ENFORCED
  License type: Perpetual

The license portal now shows one missing throughput license:

Caveats & lessons learned:

There are a couple of hiccups in the configuration that may strike you, though.

• Unable to remove HSEC license with throughput other than 50000 (default).
  It is not possible with my software version to go from HSEC level to 250 Mbps level. You need to remove the throughput level entirely, so the device resets itself to default before applying for a 250 Mbps license. Even with the platform hardware throughput level 250000 configuration, it throws this error when removing the HSEC license with no license feature hseck9. You need to enter the command no platform hardware throughput level unthrottled, which is not even applied in the configuration, and then you get the platform level to 50000, allowing you to remove the hseck9 feature.

licensetest#show run | i platform
platform hardware throughput crypto 250000
licensetest#conf t
licensetest(config)#no license feature hseck9
% HSECK9 cannot be disabled with unthrottled crypto level configured, please change throughput level 
licensetest(config)#no platform hardware throughput crypto unthrottled
% Please write mem and reload
% The config will take effect on next reboot

After the reload, it is possible to remove it:

licensetest#show run | i platform
platform hardware throughput crypto 50000
licensetest#show run | i license
license feature hseck9
licensetest#conf t
licensetest(config)#no license feature hseck9
% use 'write' command to disable 'hseck9' license on next boot
licensetest(config)#end
licensetest#write
licensetest#reload

After another reload, you can install the IPSec performance license and reload again :)

licensetest#conf t
licensetest(config)#license boot level securityk9 
licensetest(config)#platform hardware throughput crypto 250000
licensetest(config)#exit 
licensetest#write memory
licensetest#reload

And after another reload, the license is finally applied correctly:

License Usage
=============

throughput (ISR_1100_8P_IPSEC_Throughput_200Mbps):
  Description: throughput
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: throughput
  Feature Description: throughput
  Enforcement type: NOT ENFORCED
  License type: Perpetual

securityk9 (ISR_1100_8P_Security):
  Description: securityk9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: securityk9
  Feature Description: securityk9
  Enforcement type: NOT ENFORCED
  License type: Perpetual

• Licenses can be applied in nonsense combinations.
  I noticed this accidentally - you can have both an HSEC license and a throughput level of 250 Mbps. This a logical nonsense - you can have 50 Mbps or 250 Mbps, or unlimited (HSEC). You can't have limited (250Mbps) and unlimited (HSEC) simultaneously. Yet, you can get that combination if you do as follows:

licensetest(config)#platform hardware throughput crypto 250000
licensetest(config)#license feature hseck9

The resulting license usage will be like this:

License Usage
=============

throughput (ISR_1100_8P_IPSEC_Throughput_200Mbps):
  Description: throughput
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: throughput
  Feature Description: throughput
  Enforcement type: NOT ENFORCED
  License type: Perpetual

hseck9 (ISR_1100_8P_Hsec):
  Description: hseck9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: RESTRICTED - ALLOWED
  Feature Name: hseck9
  Feature Description: hseck9
  Enforcement type: EXPORT RESTRICTED
  License type: Perpetual

securityk9 (ISR_1100_8P_Security):
  Description: securityk9
  Count: 1
  Version: 1.0
  Status: IN USE
  Export status: NOT RESTRICTED
  Feature Name: securityk9
  Feature Description: securityk9
  Enforcement type: NOT ENFORCED
  License type: Perpetual

Summary

This guide was about choosing an ISR1100 series router and its licenses. I believe the licensing is logical and straightforward. As with everything, the information in the documentation is sometimes contradictory and points in multiple directions. The approach to licensing the device outlined in this article should lead to fewer reloads and less headache with license troubleshooting.

As I am using LibreNMS to monitor my customer's networks, I like to keep everything updated as much as possible, using the LTS releases of Ubuntu server. One of the last servers I didn't yet upgrade to Ubuntu 22.04, is the LibreNMS central server and its two pollers.

Intro

My installation of LibreNMS consists of a central server, which runs all the databases and web front end. This server is responsible for polling and monitoring publicly accessible devices (i.e., those internet-facing routers), saving the data in the database, alerting, and presenting it with a web interface.

Another type of server in my installation is a poller, which polls devices that are not publicly reachable, like devices running inside my customers' networks - switches, servers, etc. This server polls such devices and sends the collected data to the central monitoring database, and it doesn't have any web interface or a database.

Prerequisites

As always - Have a BACKUP of your server/data. You can't reverse these commands. If you are doing this in a cloud hosting like DigitalOcean, take a snapshot of your machine before proceeding further!

After you have a backup of your server, you should see something similar to this after login:

Welcome to Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-128-generic x86_64)

0 updates can be applied immediately.

New release '22.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

We will be using do-release-upgrade to upgrade the OS version.

First, it is reasonable to update everything to the latest versions.

sudo apt update
sudo apt dist-upgrade

Press y and ENTER when prompted.

We need to do the same thing for LibreNMS and validate that everything is functional before starting the upgrade. My user is librenms and my installation folder is /opt/librenms/.

sudo su librenms
cd /opt/librenms
./daily.sh

daily.sh is a script that is supposed to run daily to install updates and maintenance operations like database optimization, etc.

./daily.sh 
Updating to latest codebase                        OK
Updating Composer packages                         OK
Updating SQL-Schema                                OK
Updating submodules                                OK
Cleaning up DB                                     OK
Fetching notifications                              OK
Caching PeeringDB data                             OK
Caching Mac OUI data                               OK

If everything is OK, we can validate the installation if there are any issues.

./validate.php 
===========================================
Component | Version
--------- | -------
LibreNMS  | 22.9.0-59-g43cb72549 (2022-10-17T09:04:22+02:00)
DB Schema | 2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP       | 8.1.11
Python    | 3.8.10
Database  | MariaDB 10.3.34-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | 5.8
===========================================

[OK]    Composer Version: 2.4.3
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connection to memcached is ok
[OK]    Connected to rrdcached
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

Everything seems to be running, so we can proceed to the serious stuff.

Step 1: Run the do-release-upgrade

Ubuntu has a one-shot command to upgrade the OS to the newest version. For LTS OS like 20.04, this command searches for more recent LTS releases. In this case, 22.04. This command must run as a root user or user with sudo privileges.

sudo do-release-upgrade

The server verifies how you are performing the upgrade. It is not recommended to install over SSH in case the connection breaks. For this case, the installer creates an additional SSH daemon running on port 1022 for added security.

Continue running under SSH? 

This session appears to be running under ssh. It is not recommended 
to perform a upgrade over ssh currently because in case of failure it 
is harder to recover. 

If you continue, an additional ssh daemon will be started at port 
'1022'. 
Do you want to continue? 

Continue [yN] 

Press y and ENTERto continue the installation.

Starting additional sshd 

To make recovery in case of failure easier, an additional sshd will 
be started on port '1022'. If anything goes wrong with the running 
ssh you can still connect to the additional one. 


To continue please press [ENTER]

Press ENTERto continue the installation.

Now the installer needs to update all the repositories. If you are using any third-party repositories (like ondrej/ppa for newer PHP), these will be disabled.

Updating repository information

Third party sources disabled 

Some third party entries in your sources.list were disabled. You can 
re-enable them after the upgrade with the 'software-properties' tool 
or your package manager. 

To continue please press [ENTER]

Press ENTERto continue the installation.

After updating itself with the lastest repository information the installer asks whether it should download the latest packages and install them.

Do you want to start the upgrade? 


14 installed packages are no longer supported by Canonical. You can 
still get support from the community. 

21 packages are going to be removed. 146 new packages are going to be 
installed. 837 packages are going to be upgraded. 

You have to download a total of 761 M. This download will take about 
2 minutes with a 40Mbit connection and about 20 minutes with a 5Mbit 
connection. 

Fetching and installing the upgrade can take several hours. Once the 
download has finished, the process cannot be canceled. 

 Continue [yN]  Details [d]

You can continue with y or list the details with d before. After you continue you may get some prompts about your customized configuration files. If you want to keep your version or rewrite it with newer package maintainers version.

Configuration file '/etc/mysql/mariadb.conf.d/50-server.cnf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** 50-server.cnf (Y/I/N/O/D/Z) [default=N] ? 

Usually you want to keep your version (press N). Then the installer asks to remove obsolete software.

Searching for obsolete software
Reading state information... Done

Remove obsolete packages? 


123 packages are going to be removed. 

Removing the packages can take several hours. 

 Continue [yN]  Details [d]

Again, press y and ENTER. After the removal of the packages, the server needs to restart.

System upgrade is complete.

Restart required 

To finish the upgrade, a restart is required. 
If you select 'y' the system will be restarted. 

Continue [yN]

Press y and ENTER to reboot the server.

Step 2: Verify LibreNMS is functional after the OS upgrade

After the restart and logging into the server again, you should see this:

Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-50-generic x86_64)

0 updates can be applied immediately.

Again, check if there are any updates available and if so, install them.

sudo apt update

sudo apt dist-upgrade

Then log on as librenms user and validate the LibreNMS installation once again.

sudo su librenms
cd /opt/librenms
./validate.sh

In my case, there are some issues:

./validate.php 
===========================================
Component | Version
--------- | -------
LibreNMS  | 22.9.0-59-g43cb72549 (2022-10-17T09:04:22+02:00)
DB Schema | 2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP       | 8.1.11
Python    | 3.10.6
Database  | MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool   | 1.7.2
SNMP      | 5.9.1
===========================================

[OK]    Composer Version: 2.4.3
[OK]    Dependencies up-to-date.
[FAIL]  APP_KEY does not match key used to encrypt data. APP_KEY must be the same on all nodes.
        [FIX]: 
        If you rotated APP_KEY, run lnms key:rotate to resolve.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connection to memcached is ok
[OK]    Connected to rrdcached
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[FAIL]  Some poller nodes have not checked in recently
        Inactive Nodes:
         monitoring-nms0
[OK]    Redis is unavailable
[FAIL]  Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=3.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
        [FIX]: 
        pip3 install -r /opt/librenms/requirements.txt
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

There is an issue with the python installation - the installer reinstalled python and installed no dependencies. Luckily the validate.sh script gives information on how to fix the problem. It should be a simple pip installation of LibreNMS-required libraries listed in the file requirements.txt.

pip3 install -r /opt/librenms/requirements.txt

After the installation of the libraries, we can validate the installation again and check the monitoring.

./validate.php 
===========================================
Component | Version
--------- | -------
LibreNMS  | 22.9.0-59-g43cb72549 (2022-10-17T09:04:22+02:00)
DB Schema | 2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP       | 8.1.11
Python    | 3.10.6
Database  | MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool   | 1.7.2
SNMP      | 5.9.1
===========================================

[OK]    Composer Version: 2.4.3
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Distributed Polling setting is enabled globally
[OK]    Connection to memcached is ok
[OK]    Connected to rrdcached
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[OK]    rrdtool version ok
[OK]    Connected to rrdcached

Now everything is running correctly on a Ubuntu 22.04 LTS server.

As I am using LibreNMS to monitor my customer's networks, I like to keep everything updated as much as possible. LibreNMS stopped updating itself, requiring PHP 8, ideally PHP 8.1. This upgrade is a simple task. However, to record how to do it, in case I need it in the future, here is the how-to.

Intro

I used this how-to on Ubuntu 20.04. With Ubuntu 22.04, there is no need to add Ondrej's repository.

Prerequisites

Have a BACKUP of your server/data. You can't reverse these commands. If you are doing this in a cloud hosting like DigitalOcean, take a snapshot of your machine before proceeding further!

Check the version of PHP running on your server:

php -v

You can proceed with this how-to if you are running PHP 7, as I do.

Step 1: Check your installed PHP packages

During the installation, you do not only upgrade PHP core packages but all of the used extensions as well. For example, if you use a PHP zip extension, you will install this at the end of this how-to.

At the end of this how-to, there is a standard list of commonly used extensions. However, it is wise to check what you are currently using and ensure you reinstall all the 7.4 extensions after the upgrade.

dpkg -l | grep php | tee packages.txt

The result will list all the PHP packages in the system:

ii  php-common                            2:75                              all          Common files for PHP packages
ii  php-composer-ca-bundle                1.2.6-1                           all          utility library to find a path to the system CA bundle
ii  php-composer-semver                   1.5.1-1                           all          utilities, version constraint parsing and validation
ii  php-composer-spdx-licenses            1.5.3-1                           all          SPDX licenses list and validation library
ii  php-composer-xdebug-handler           1.4.0-1                           all          Restarts a process without Xdebug
ii  php-igbinary                          3.1.2+2.0.8-1build1               amd64        igbinary PHP serializer
ii  php-json-schema                       5.2.9-1                           all          implementation of JSON schema
ii  php-memcached                         3.1.4+2.2.0-1                     amd64        memcached extension module for PHP, uses libmemcached
ii  php-msgpack                           2.1.0beta1-0ubuntu2               amd64        PHP extension for interfacing with MessagePack
ii  php-psr-container                     1.0.0-2                           all          Common Container Interface (PHP FIG PSR-11)
ii  php-psr-log                           1.1.2-1                           all          common interface for logging libraries
ii  php-symfony-console                   4.3.8+dfsg-1ubuntu1               all          run tasks from the command line
ii  php-symfony-filesystem                 4.3.8+dfsg-1ubuntu1               all          basic filesystem utilities
ii  php-symfony-finder                     4.3.8+dfsg-1ubuntu1               all          find files and directories
ii  php-symfony-process                   4.3.8+dfsg-1ubuntu1               all          execute commands in sub-processes
ii  php-symfony-service-contracts         1.1.8-1                           all          Generic abstractions related to writing services
ii  php7.4-cli                            7.4.3-4ubuntu2.13                 amd64        command-line interpreter for the PHP scripting language
ii  php7.4-common                         7.4.3-4ubuntu2.13                 amd64        documentation, examples and common module for PHP
ii  php7.4-curl                           7.4.3-4ubuntu2.13                 amd64        CURL module for PHP
ii  php7.4-fpm                            7.4.3-4ubuntu2.13                 amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php7.4-gd                             7.4.3-4ubuntu2.13                 amd64        GD module for PHP
ii  php7.4-gmp                            7.4.3-4ubuntu2.13                 amd64        GMP module for PHP
ii  php7.4-json                           7.4.3-4ubuntu2.13                 amd64        JSON module for PHP
ii  php7.4-mbstring                       7.4.3-4ubuntu2.13                 amd64        MBSTRING module for PHP
ii  php7.4-mysql                          7.4.3-4ubuntu2.13                 amd64        MySQL module for PHP
ii  php7.4-opcache                        7.4.3-4ubuntu2.13                 amd64        Zend OpCache module for PHP
ii  php7.4-readline                       7.4.3-4ubuntu2.13                 amd64        readline module for PHP
ii  php7.4-snmp                           7.4.3-4ubuntu2.13                 amd64        SNMP module for PHP
ii  php7.4-xml                            7.4.3-4ubuntu2.13                 amd64        DOM, SimpleXML, XML, and XSL module for PHP
ii  php7.4-zip                            7.4.3-4ubuntu2.13                 amd64        Zip module for PHP

The output above lists all my extensions. You can see that I am running PHP 7.4.3 before upgrading to 8.1. Keep your output safe for reference.

Step 2: Uninstall PHP 7.x and all its extensions

Run the command below to uninstall PHP 7 and all its extensions.

sudo apt purge php7.*

Press y and ENTER when prompted.

Step 3: Autoclean and Autoremove

After uninstalling packages from Linux, it's advised to run these two commands.

sudo apt autoclean

sudo apt autoremove

Again, press y and ENTER when prompted.

Step 4: Add PHP repository

If you are running Ubuntu 18.04 or Ubuntu 20.04, PHP 8 is not available in Ubuntu repositories, and you need to add Ondrej Sury's repository to be able to install it.

sudo add-apt-repository ppa:ondrej/php

You will see a welcome message as follows‌:

Co-installable PHP versions: PHP 5.6, PHP 7.x and most requested extensions are included. Only Supported Versions of PHP (http://php.net/supported-versions.php) for Supported Ubuntu Releases (https://wiki.ubuntu.com/Releases) are provided. Don't ask for end-of-life PHP versions or Ubuntu release, they won't be provided.

Debian oldstable and stable packages are provided as well: https://deb.sury.org/#debian-dpa

You can get more information about the packages at https://deb.sury.org

IMPORTANT: The -backports is now required on older Ubuntu releases.

BUGS&FEATURES: This PPA now has a issue tracker:
https://deb.sury.org/#bug-reporting

CAVEATS:
1. If you are using php-gearman, you need to add ppa:ondrej/pkg-gearman
2. If you are using apache2, you are advised to add ppa:ondrej/apache2
3. If you are using nginx, you are advised to add ppa:ondrej/nginx-mainline
   or ppa:ondrej/nginx

PLEASE READ: If you like my work and want to give me a little motivation, please consider donating regularly: https://donate.sury.org/

WARNING: add-apt-repository is broken with non-UTF-8 locales, see
https://github.com/oerdnj/deb.sury.org/issues/56 for workaround:

# LC_ALL=C.UTF-8 add-apt-repository ppa:ondrej/php
 More info: https://launchpad.net/~ondrej/+archive/ubuntu/php
Press [ENTER] to continue or Ctrl-c to cancel adding it.

Press ENTER to add the repository.

Step 5: Install PHP 8.1

As of the time of writing (October 2022), LibreNMS requires 8.1. Therefore I am going to install PHP version 8.1.

sudo apt udpate

sudo apt install php8.1

You will see this output:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libapache2-mod-php8.1 libargon2-1 libpcre2-8-0 libsodium23 php-common php8.1-cli php8.1-common
  php8.1-opcache php8.1-readline
Suggested packages:
  php-pear
The following NEW packages will be installed:
  libapache2-mod-php8.1 libargon2-1 libpcre2-8-0 libsodium23 php-common php8.1 php8.1-cli php8.1-common
  php8.1-opcache php8.1-readline
0 upgraded, 10 newly installed, 0 to remove and 159 not upgraded.
Need to get 5035 kB of archives.
After this operation, 22.0 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Press y and ENTERto continue the installation.

In my case, the installer also installed the apache2 package. I don't need it. Therefore I remove it.

sudo apt purge apache2

Press y and ENTERto continue the removal.

Step 6: Install the extensions

The command below installs the typical extensions required by the standard WordPress site.

sudo apt install php8.1-common php8.1-mysql php8.1-xml php8.1-xmlrpc php8.1-curl php8.1-gd php8.1-imagick php8.1-cli php8.1-dev php8.1-imap php8.1-mbstring php8.1-opcache php8.1-soap php8.1-zip php8.1-intl -y

However, I don't host a WordPress site but a LibreNMS monitoring server with slightly different requirements. Therefore my apt install is going to contain different extensions.

sudo apt install php8.1-cli php8.1-common php8.1-curl php8.1-fpm php8.1-gd php8.1-gmp php8.1-mbstring php8.1-mysql php8.1-opcache php8.1-readline php8.1-snmp php8.1-xml php8.1-zip php8.1-memcached -y

You should also check which extensions were installed before. There can be some differences, though - for example, php7.4-json doesn't have a php8.1-json counterpart as its functionality has been moved into the core code.

Step 7: Check the PHP version

Now it is time to verify the current version of PHP.

php -v

The output should be similar to this:

PHP 8.1.11 (cli) (built: Sep 29 2022 22:28:49) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.11, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.11, Copyright (c), by Zend Technologies

PHP 8 is now installed and running. For use with NGINX, you will need to adjust couple more things.

Step 8: Adjusting php.ini and NGINX (optional, depending on the case)

In my installation, I need to have the timezones aligned between PHP and MariaDB database. For this, I need to update also php.ini file, which is now located in /etc/php/8.1/cli and /etc/php/8.1/fpm folders. I am using nano. Feel free to use whatever editor suits you best.

sudo nano /etc/php/8.1/cli/php.ini

Here search for this line:

;date.timezone = UTC

Uncomment the line and adjust the timezone to your need.

date.timezone = Europe/Prague

Then do the same for /etc/php/8.1/fpm/php.ini.

Then you should head to the NGINX configuration file for the server. In my case, it is located here: /etc/nginx/conf.d/librenms.conf, but typically this would be in /etc/nginx/sites-available/default or some other file in that folder.

sudo nano /etc/nginx/conf.d/librenms.conf

In this file, look for the server location which handles .php files. It should look like this:

server {
# ... OUTPUT OMITTED FOR BREWITY ...
 location ~ \.php {
  include fastcgi.conf;
  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;

The server still uses the old version of PHP and probably isn't able to run the PHP code at all. We need to change it to the existing php8.1. Like this:

server {
# ... OUTPUT OMITTED FOR BREWITY ...
 location ~ \.php {
  include fastcgi.conf;
  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;

After this change, we need the server to use the new configuration of PHP and NGINX:

sudo systemctl restart php8.1-fpm
sudo systemctl restart nginx

And that's it. The server is now running NGINX with PHP 8.1.