How to upgrade Ubuntu 20.04 to Ubuntu 22.04 with LibreNMS monitoring
As I am using LibreNMS to monitor my customer's networks, I like to keep everything updated as much as possible, using the LTS releases of Ubuntu server. One of the last servers I didn't yet upgrade to Ubuntu 22.04, is the LibreNMS central server and its two pollers.
Intro
My installation of LibreNMS consists of a central server, which runs all the databases and web front end. This server is responsible for polling and monitoring publicly accessible devices (i.e., those internet-facing routers), saving the data in the database, alerting, and presenting it with a web interface.
Another type of server in my installation is a poller, which polls devices that are not publicly reachable, like devices running inside my customers' networks - switches, servers, etc. This server polls such devices and sends the collected data to the central monitoring database, and it doesn't have any web interface or a database.
Prerequisites
As always - Have a BACKUP of your server/data. You can't reverse these commands. If you are doing this in a cloud hosting like DigitalOcean, take a snapshot of your machine before proceeding further!
After you have a backup of your server, you should see something similar to this after login:
Welcome to Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-128-generic x86_64)
0 updates can be applied immediately.
New release '22.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.We will be using do-release-upgrade to upgrade the OS version.
First, it is reasonable to update everything to the latest versions.
sudo apt update
sudo apt dist-upgradePress y and ENTER when prompted.
We need to do the same thing for LibreNMS and validate that everything is functional before starting the upgrade. My user is librenms and my installation folder is /opt/librenms/.
sudo su librenms
cd /opt/librenms
./daily.shdaily.sh is a script that is supposed to run daily to install updates and maintenance operations like database optimization, etc.
./daily.sh
Updating to latest codebase OK
Updating Composer packages OK
Updating SQL-Schema OK
Updating submodules OK
Cleaning up DB OK
Fetching notifications OK
Caching PeeringDB data OK
Caching Mac OUI data OKIf everything is OK, we can validate the installation if there are any issues.
./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 22.9.0-59-g43cb72549 (2022-10-17T09:04:22+02:00)
DB Schema | 2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP | 8.1.11
Python | 3.8.10
Database | MariaDB 10.3.34-MariaDB-0ubuntu0.20.04.1
RRDTool | 1.7.2
SNMP | 5.8
===========================================
[OK] Composer Version: 2.4.3
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Distributed Polling setting is enabled globally
[OK] Connection to memcached is ok
[OK] Connected to rrdcached
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcachedEverything seems to be running, so we can proceed to the serious stuff.
Step 1: Run the do-release-upgrade
Ubuntu has a one-shot command to upgrade the OS to the newest version. For LTS OS like 20.04, this command searches for more recent LTS releases. In this case, 22.04. This command must run as a root user or user with sudo privileges.
sudo do-release-upgradeThe server verifies how you are performing the upgrade. It is not recommended to install over SSH in case the connection breaks. For this case, the installer creates an additional SSH daemon running on port 1022 for added security.
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?
Continue [yN] Press y and ENTERto continue the installation.
Starting additional sshd
To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
To continue please press [ENTER]Press ENTERto continue the installation.
Now the installer needs to update all the repositories. If you are using any third-party repositories (like ondrej/ppa for newer PHP), these will be disabled.
Updating repository information
Third party sources disabled
Some third party entries in your sources.list were disabled. You can
re-enable them after the upgrade with the 'software-properties' tool
or your package manager.
To continue please press [ENTER]Press ENTERto continue the installation.
After updating itself with the lastest repository information the installer asks whether it should download the latest packages and install them.
Do you want to start the upgrade?
14 installed packages are no longer supported by Canonical. You can
still get support from the community.
21 packages are going to be removed. 146 new packages are going to be
installed. 837 packages are going to be upgraded.
You have to download a total of 761 M. This download will take about
2 minutes with a 40Mbit connection and about 20 minutes with a 5Mbit
connection.
Fetching and installing the upgrade can take several hours. Once the
download has finished, the process cannot be canceled.
Continue [yN] Details [d]You can continue with y or list the details with d before. After you continue you may get some prompts about your customized configuration files. If you want to keep your version or rewrite it with newer package maintainers version.
Configuration file '/etc/mysql/mariadb.conf.d/50-server.cnf'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** 50-server.cnf (Y/I/N/O/D/Z) [default=N] ? Usually you want to keep your version (press N). Then the installer asks to remove obsolete software.
Searching for obsolete software
Reading state information... Done
Remove obsolete packages?
123 packages are going to be removed.
Removing the packages can take several hours.
Continue [yN] Details [d]Again, press y and ENTER. After the removal of the packages, the server needs to restart.
System upgrade is complete.
Restart required
To finish the upgrade, a restart is required.
If you select 'y' the system will be restarted.
Continue [yN]Press y and ENTER to reboot the server.
Step 2: Verify LibreNMS is functional after the OS upgrade
After the restart and logging into the server again, you should see this:
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-50-generic x86_64)
0 updates can be applied immediately.Again, check if there are any updates available and if so, install them.
sudo apt updatesudo apt dist-upgradeThen log on as librenms user and validate the LibreNMS installation once again.
sudo su librenms
cd /opt/librenms
./validate.shIn my case, there are some issues:
./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 22.9.0-59-g43cb72549 (2022-10-17T09:04:22+02:00)
DB Schema | 2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP | 8.1.11
Python | 3.10.6
Database | MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool | 1.7.2
SNMP | 5.9.1
===========================================
[OK] Composer Version: 2.4.3
[OK] Dependencies up-to-date.
[FAIL] APP_KEY does not match key used to encrypt data. APP_KEY must be the same on all nodes.
[FIX]:
If you rotated APP_KEY, run lnms key:rotate to resolve.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Distributed Polling setting is enabled globally
[OK] Connection to memcached is ok
[OK] Connected to rrdcached
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[FAIL] Some poller nodes have not checked in recently
Inactive Nodes:
monitoring-nms0
[OK] Redis is unavailable
[FAIL] Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=3.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
[FIX]:
pip3 install -r /opt/librenms/requirements.txt
[OK] rrdtool version ok
[OK] Connected to rrdcachedThere is an issue with the python installation - the installer reinstalled python and installed no dependencies. Luckily the validate.sh script gives information on how to fix the problem. It should be a simple pip installation of LibreNMS-required libraries listed in the file requirements.txt.
pip3 install -r /opt/librenms/requirements.txtAfter the installation of the libraries, we can validate the installation again and check the monitoring.
./validate.php
===========================================
Component | Version
--------- | -------
LibreNMS | 22.9.0-59-g43cb72549 (2022-10-17T09:04:22+02:00)
DB Schema | 2022_09_03_091314_update_ports_adsl_table_with_defaults (246)
PHP | 8.1.11
Python | 3.10.6
Database | MariaDB 10.6.7-MariaDB-2ubuntu1.1
RRDTool | 1.7.2
SNMP | 5.9.1
===========================================
[OK] Composer Version: 2.4.3
[OK] Dependencies up-to-date.
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Distributed Polling setting is enabled globally
[OK] Connection to memcached is ok
[OK] Connected to rrdcached
[OK] Active pollers found
[OK] Dispatcher Service not detected
[OK] Locks are functional
[OK] Python poller wrapper is polling
[OK] Redis is unavailable
[OK] rrdtool version ok
[OK] Connected to rrdcachedNow everything is running correctly on a Ubuntu 22.04 LTS server.