Upgrading iocage jails from FreeBSD 10.3 to 11.0 (major upgrade)

Iocage makes upgrading jails quite simple. It can be a little more involved than the host upgrade but not by much.

First on the host server we need to fetch the right version of FreeBSD for iocage to use. This will create the iocage zfs file system structures for upgrading to and building 11.0 jails.

$ sudo iocage fetch 11.0-RELEASE

$ zfs list
NAME                                                              USED  AVAIL  REFER  MOUNTPOINT
pgdata                                                           7.63G   262G    19K  legacy
pgdata/pg94                                                      7.63G   262G  7.63G  /pg94
sys                                                               257G   580G    19K  legacy
sys/ROOT                                                         2.27G   580G    19K  legacy
sys/ROOT/default                                                 2.27G   580G  1.38G  /
sys/home                                                         4.62G   580G  4.60G  /home
sys/iocage                                                       94.2G   580G  24.9G  /iocage
sys/iocage/.defaults                                               28K   580G    19K  /iocage/.defaults
sys/iocage/base                                                  1.35G   580G    19K  /iocage/base
sys/iocage/base/10.3-RELEASE                                      655M   580G    19K  /iocage/base/10.3-RELEASE
sys/iocage/base/10.3-RELEASE/root                                 655M   580G  69.5M  /iocage/base/10.3-RELEASE/root
sys/iocage/base/10.3-RELEASE/root/bin                             804K   580G   790K  /iocage/base/10.3-RELEASE/root/bin
sys/iocage/base/10.3-RELEASE/root/boot                           1.24M   580G  1.23M  /iocage/base/10.3-RELEASE/root/boot
sys/iocage/base/10.3-RELEASE/root/lib                            5.35M   580G  5.34M  /iocage/base/10.3-RELEASE/root/lib
sys/iocage/base/10.3-RELEASE/root/libexec                         196K   580G   185K  /iocage/base/10.3-RELEASE/root/libexec
sys/iocage/base/10.3-RELEASE/root/rescue                         4.94M   580G  4.93M  /iocage/base/10.3-RELEASE/root/rescue
sys/iocage/base/10.3-RELEASE/root/sbin                           3.81M   580G  3.79M  /iocage/base/10.3-RELEASE/root/sbin
sys/iocage/base/10.3-RELEASE/root/usr                             569M   580G    21K  /iocage/base/10.3-RELEASE/root/usr
sys/iocage/base/10.3-RELEASE/root/usr/bin                        48.0M   580G  47.9M  /iocage/base/10.3-RELEASE/root/usr/bin
sys/iocage/base/10.3-RELEASE/root/usr/include                    8.11M   580G  7.94M  /iocage/base/10.3-RELEASE/root/usr/include
sys/iocage/base/10.3-RELEASE/root/usr/lib                        39.7M   580G  39.7M  /iocage/base/10.3-RELEASE/root/usr/lib
sys/iocage/base/10.3-RELEASE/root/usr/lib32                      39.0M   580G  38.9M  /iocage/base/10.3-RELEASE/root/usr/lib32
sys/iocage/base/10.3-RELEASE/root/usr/libdata                     139K   580G   127K  /iocage/base/10.3-RELEASE/root/usr/libdata
sys/iocage/base/10.3-RELEASE/root/usr/libexec                    1.80M   580G  1.75M  /iocage/base/10.3-RELEASE/root/usr/libexec
sys/iocage/base/10.3-RELEASE/root/usr/sbin                       8.08M   580G  8.07M  /iocage/base/10.3-RELEASE/root/usr/sbin
sys/iocage/base/10.3-RELEASE/root/usr/share                      29.9M   580G  29.5M  /iocage/base/10.3-RELEASE/root/usr/share
sys/iocage/base/10.3-RELEASE/root/usr/src                         394M   580G   389M  /iocage/base/10.3-RELEASE/root/usr/src
sys/iocage/base/11.0-RELEASE                                      724M   580G    19K  /iocage/base/11.0-RELEASE
sys/iocage/base/11.0-RELEASE/root                                 724M   580G  47.2M  /iocage/base/11.0-RELEASE/root
sys/iocage/base/11.0-RELEASE/root/bin                             824K   580G   824K  /iocage/base/11.0-RELEASE/root/bin
sys/iocage/base/11.0-RELEASE/root/boot                           1.48M   580G  1.48M  /iocage/base/11.0-RELEASE/root/boot
sys/iocage/base/11.0-RELEASE/root/lib                            5.96M   580G  5.96M  /iocage/base/11.0-RELEASE/root/lib
sys/iocage/base/11.0-RELEASE/root/libexec                         192K   580G   192K  /iocage/base/11.0-RELEASE/root/libexec
sys/iocage/base/11.0-RELEASE/root/rescue                         5.75M   580G  5.75M  /iocage/base/11.0-RELEASE/root/rescue
sys/iocage/base/11.0-RELEASE/root/sbin                           3.99M   580G  3.99M  /iocage/base/11.0-RELEASE/root/sbin
sys/iocage/base/11.0-RELEASE/root/usr                             658M   580G   120K  /iocage/base/11.0-RELEASE/root/usr
sys/iocage/base/11.0-RELEASE/root/usr/bin                        77.8M   580G  77.8M  /iocage/base/11.0-RELEASE/root/usr/bin
sys/iocage/base/11.0-RELEASE/root/usr/include                    8.08M   580G  8.08M  /iocage/base/11.0-RELEASE/root/usr/include
sys/iocage/base/11.0-RELEASE/root/usr/lib                        48.7M   580G  48.7M  /iocage/base/11.0-RELEASE/root/usr/lib
sys/iocage/base/11.0-RELEASE/root/usr/lib32                      40.8M   580G  40.8M  /iocage/base/11.0-RELEASE/root/usr/lib32
sys/iocage/base/11.0-RELEASE/root/usr/libdata                     129K   580G   129K  /iocage/base/11.0-RELEASE/root/usr/libdata
sys/iocage/base/11.0-RELEASE/root/usr/libexec                    1.91M   580G  1.91M  /iocage/base/11.0-RELEASE/root/usr/libexec
sys/iocage/base/11.0-RELEASE/root/usr/sbin                       8.57M   580G  8.57M  /iocage/base/11.0-RELEASE/root/usr/sbin
sys/iocage/base/11.0-RELEASE/root/usr/share                      38.4M   580G  38.4M  /iocage/base/11.0-RELEASE/root/usr/share
sys/iocage/base/11.0-RELEASE/root/usr/src                         434M   580G   434M  /iocage/base/11.0-RELEASE/root/usr/src

 

$ sudo iocage upgrade jail01 -r 11.0-RELEASE
* Creating back-out snapshot.
* Upgrading jail.
Looking up update.FreeBSD.org mirrors... 4 mirrors found.
Fetching metadata signature for 10.3-RELEASE from update4.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic src/src world/base world/doc world/lib32

The following components of FreeBSD do not seem to be installed:
world/games

Does this look reasonable (y/n)? y

Fetching metadata signature for 11.0-RELEASE from update4.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Fetching files from 10.3-RELEASE for merging... done.
Preparing to download files... done.
Fetching 45107 patches.....10....20....30.  done.
Applying patches... done.
Fetching 12420 files...

Now we need to upgrade all the packages in the jail.

$ sudo iocage console jail01
$ pkg-static install -f pkg
$ pkg upgrade
Shared object "libssl.so.7" not found, required by "pkg"

Here we have an issue with the old pkg binary which is linking to a version of libssl that no longer exists. To fix this we need to reinstall the pkg binary.

$ pkg-static install -f pkg
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.8.8 -> 1.9.4_1

Number of packages to be upgraded: 1

3 MiB to be downloaded.

Proceed with this action? [y/N]: y
[blackbox@mars.com] Fetching pkg-1.9.4_1.txz: 100%    3 MiB   2.9MB/s    00:01
Checking integrity... done (0 conflicting)
[blackbox@mars.com] [1/1] Upgrading pkg from 1.8.8 to 1.9.4_1...
[blackbox@mars.com] [1/1] Extracting pkg-1.9.4_1: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
        pkg-1.9.4_1

Number of packages to be reinstalled: 1

Proceed with this action? [y/N]: y
[1/1] Reinstalling pkg-1.9.4_1...
[1/1] Extracting pkg-1.9.4_1: 100%

Now let’s see if it works.

$ pkg info
alsa-lib-1.1.2                 ALSA compatibility library
ap24-mod_perl2-2.0.9,3         Embeds a Perl interpreter in the Apache server
apache24-2.4.23_1              Version 2.4.x of Apache web server
apr-1.5.2.1.5.4_1              Apache Portability Library
augeas-1.4.0                   Configuration editing tool
...

It works! Let’s continue upgrading the packages..

$ pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (324 candidates): 100%
Processing candidates (324 candidates): 100%
The following 353 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
 readline: 6.3.8
 p5-Sub-Quote: 2.003001
 p5-List-UtilsBy: 0.10
 p5-Math-Random-Secure: 0.06_1
 p5-Any-Moose: 0.26
...

Installed packages to be UPGRADED:
 zsh: 5.2_3 -> 5.3.1
 xproto: 7.0.28 -> 7.0.31
 wget: 1.18 -> 1.18_2
 vim-lite: 8.0.0019_1 -> 8.0.0134_1
...

Installed packages to be REINSTALLED:
 zfs-stats-1.2.2_1 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 xextproto-7.3.0 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 unzip-6.0_7 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 unixODBC-2.3.4 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
...

Number of packages to be installed: 29
Number of packages to be upgraded: 86
Number of packages to be reinstalled: 238

The process will require 46 MiB more space.
203 MiB to be downloaded.

Proceed with this action? [y/N]:
[blackbox@mars.com] Fetching zsh-5.3.1.txz: 100% 4 MiB 4.1MB/s 00:01
[blackbox@mars.com] Fetching zfs-stats-1.2.2_1.txz: 100% 9 KiB 9.5kB/s 00:01
[blackbox@mars.com] Fetching xproto-7.0.31.txz: 100% 59 KiB 60.2kB/s 00:01
[blackbox@mars.com] Fetching xextproto-7.3.0.txz: 100% 21 KiB 21.9kB/s 00:01
[blackbox@mars.com] Fetching wget-1.18_2.txz: 100% 578 KiB 592.2kB/s 00:01
[blackbox@mars.com] Fetching vim-lite-8.0.0134_1.txz: 100% 5 MiB 5.5MB/s 00:01
...

Number of packages to be installed: 29
Number of packages to be upgraded: 85
Number of packages to be reinstalled: 238

The process will require 46 MiB more space.

Proceed with this action? [y/N]:


[blackbox@mars.com] [1/354] Upgrading perl5 from 5.20.3_15 to 5.24.1.r4_1...
[blackbox@mars.com] [1/354] Extracting perl5-5.24.1.r4_1: 100%
[blackbox@mars.com] [2/354] Reinstalling p5-Sub-Install-0.928_1...
[blackbox@mars.com] [2/354] Extracting p5-Sub-Install-0.928_1: 100%
...
[blackbox@mars.com] [54/354] Installing libassuan-2.4.3...
[blackbox@mars.com] [54/354] Extracting libassuan-2.4.3: 100%
[blackbox@mars.com] [55/354] Installing libtasn1-4.9...
[blackbox@mars.com] [55/354] Extracting libtasn1-4.9: 100%
[blackbox@mars.com] [56/354] Installing tpm-emulator-0.7.4_1...
===> Creating groups.
Creating group '_tss' with gid '601'.
===> Creating users
Creating user '_tss' with uid '601'.
pw: user '_tss' disappeared during update
pkg: PRE-INSTALL script failed

Uh oh. Houston we have a problem. No biggie. Just run this to regenerate the passwd file which will fix it.

$ vipw
:w! (to save it)

Now lets finish up the pkg  upgrade.

$ pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (273 candidates): 100%
Processing candidates (273 candidates): 100%
Checking integrity... done (1 conflicting)
 - p5-Sub-Quote-2.003001 conflicts with p5-Moo-2.002004 on /usr/local/lib/perl5/site_perl/Sub/Defer.pm
Checking integrity... done (0 conflicting)
The following 298 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
 readline: 6.3.8
 p5-Sub-Quote: 2.003001
 p5-Math-Random-Secure: 0.06_1
 p5-Any-Moose: 0.26
...

Installed packages to be UPGRADED:
 zsh: 5.2_3 -> 5.3.1
 wget: 1.18 -> 1.18_2
 vim-lite: 8.0.0019_1 -> 8.0.0134_1
...

Installed packages to be REINSTALLED:
 zfs-stats-1.2.2_1 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 xextproto-7.3.0 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 unzip-6.0_7 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 unixODBC-2.3.4 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 ttcp-1.12_1 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
 tmux-2.3 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
...

Number of packages to be installed: 25
Number of packages to be upgraded: 68
Number of packages to be reinstalled: 205

The process will require 40 MiB more space.

Proceed with this action? [y/N]: y
[eastern01.cello.com] [1/298] Installing tpm-emulator-0.7.4_1...
===> Creating groups.
Using existing group '_tss'.
===> Creating users
Using existing user '_tss'.
[blackbox@mars.com] [1/298] Extracting tpm-emulator-0.7.4_1: 100%
[blackbox@mars.com] [2/298] Deinstalling p5-Moo-2.002004...
[blackbox@mars.com] [2/298] Deleting files for p5-Moo-2.002004: 100%
[blackbox@mars.com] [3/298] Reinstalling xextproto-7.3.0...
...

Finally let’s upgrade any cpan modules we have installed in the jail.

$ cpan-outdated | cpanm
--> Working on I/IB/IBB/Acme-Damn-0.08.tar.gz
Fetching http://www.cpan.org/authors/id/I/IB/IBB/Acme-Damn-0.08.tar.gz ... OK
Configuring Acme-Damn-0.08 ... OK
Building and testing Acme-Damn-0.08 ... OK
Successfully installed Acme-Damn-0.08
--> Working on K/KA/KAZEBURO/Apache-LogFormat-Compiler-0.33.tar.gz
Fetching http://www.cpan.org/authors/id/K/KA/KAZEBURO/Apache-LogFormat-Compiler-0.33.tar.gz ... OK
Configuring Apache-LogFormat-Compiler-0.33 ... OK
Building and testing Apache-LogFormat-Compiler-0.33 ... OK
Successfully installed Apache-LogFormat-Compiler-0.33
--> Working on B/BI/BINGOS/Archive-Tar-2.24.tar.gz
Fetching http://www.cpan.org/authors/id/B/BI/BINGOS/Archive-Tar-2.24.tar.gz ... OK
Configuring Archive-Tar-2.24 ... OK
Building and testing Archive-Tar-2.24 ... OK
Successfully installed Archive-Tar-2.24
...

All done!

Advertisements

Upgrading from FreeBSD 10.3 to 11.0 (major upgrade)

Performing a major upgrade isn’t much different from doing a minor upgrade as far as ease goes. However more can go wrong. Here we’ll discuss the basic commands to get through it.

First we want to get to the latest and greatest minor version patch level of FreeBSD.

$ sudo freebsd-update fetch
Looking up update.FreeBSD.org mirrors... 4 mirrors found.
Fetching metadata signature for 10.3-RELEASE from update6.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata patches.. done.
Applying metadata patches... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 324 patches.....10....20....30. done.
Applying patches... done.
Fetching 40 files... done.

The following files are affected by updates, but no changes have
been downloaded because the files have been modified locally:
/etc/mail/freebsd.cf
/etc/mail/freebsd.submit.cf
/etc/mail/sendmail.cf
...

The following files will be removed as part of updating to 10.3-RELEASE-p15:
/usr/share/zoneinfo/America/Santa_Isabel
/usr/share/zoneinfo/Asia/Rangoon

The following files will be added as part of updating to 10.3-RELEASE-p15:
/usr/share/zoneinfo/Asia/Barnaul
/usr/share/zoneinfo/Asia/Famagusta
/usr/share/zoneinfo/Asia/Tomsk
/usr/share/zoneinfo/Asia/Yangon
...

The following files will be updated as part of updating to 10.3-RELEASE-p15:
/bin/freebsd-version
/lib/libc.so.7
/rescue/[
/rescue/atmconfig
/rescue/badsect
...

$ sudo freebsd-update install
Installing updates... done.

Now let’s do the major upgrade:

$ sudo freebsd-update upgrade -r 11.0-RELEASE
Looking up update.FreeBSD.org mirrors... 4 mirrors found.
Fetching metadata signature for 10.3-RELEASE from update3.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic src/src world/base world/games world/lib32

The following components of FreeBSD do not seem to be installed:
world/doc

Does this look reasonable (y/n)? y

Fetching metadata signature for 11.0-RELEASE from update3.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Fetching files from 10.3-RELEASE for merging... done.
Preparing to download files... done.
Fetching 45032 patches.....10....20....30....40....50. done.
Applying patches... done.
Fetching 12099 files... done.
Attempting to automatically merge changes in files... done.

The following file could not be merged automatically: /etc/defaults/rc.conf
Press Enter to edit this file in /usr/bin/vi and resolve the conflicts
manually...

At this point you’ll be placed into various files that have merge conflicts. They will need to be resolved. In each file you will see the markers that designate the old files content vs the new files content such as:

<<<<<<< current version
# $FreeBSD: release/10.3.0/etc/syslog.conf 194005 2014-06-11 15:07:02Z avg $
=======
# $FreeBSD: release/11.0.0/etc/syslog.conf 238473 2016-07-15 10:55:43Z brueffer $
>>>>>>> 11.0-RELEASE

Basically you will choose which content wins (typically the new content) by removing the markers and the old content. So in this example you would end up with:

# $FreeBSD: release/11.0.0/etc/syslog.conf 238473 2016-07-15 10:55:43Z brueffer $

Now we will continue on. Next you will be asked if various merges that did not have conflicts look reasonable.

The following changes, which occurred between FreeBSD 10.3-RELEASE and
FreeBSD 11.0-RELEASE have been merged into /etc/defaults/periodic.conf:
Does this look reasonable (y/n)?

The following files are affected by updates, but no changes have
been downloaded because the files have been modified locally:
/.cshrc
/.profile
/root/.cshrc
/root/.k5login
/root/.login
/root/.profile
...
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".

$ sudo freebsd-update install
$ sudo shutdown -r now

$ hostver
FreeBSD 10.3 RELEASE-p15

$ sudo freebsd-update install
Installing updates...

Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/usr/sbin/freebsd-update install"
again to finish installing updates.

$ hostver
FreeBSD 11.0 RELEASE-p6

$ uname -r
11.0-RELEASE-p2

$ sudo shutdown -r now

Now lets upgrade the packages on the system.

$ sudo pkg upgrade
pkg: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
Updating FreeBSD repository catalogue...
Repository FreeBSD has a wrong packagesite, need to re-create database
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   5.9MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 25901 packages processed.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.9.3 -> 1.9.4_1

Number of packages to be upgraded: 1

3 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching pkg-1.9.4_1.txz: 100%    3 MiB   2.9MB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.9.3 to 1.9.4_1...
[1/1] Extracting pkg-1.9.4_1: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (102 candidates): 100%
Processing candidates (102 candidates): 100%
The following 104 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        readline: 6.3.8
        libnghttp2: 1.18.1

Installed packages to be UPGRADED:
        zsh: 5.2_4 -> 5.3.1
        vim-lite: 8.0.0063 -> 8.0.0134_1
        trousers: 0.3.13_1 -> 0.3.14_1
        sudo: 1.8.18p1 -> 1.8.19p1
        subversion: 1.9.4 -> 1.9.5
        sqlite3: 3.14.1_1 -> 3.15.1_1
        ruby: 2.2.5_1,1 -> 2.2.6_1,1
        rsync: 3.1.2_5 -> 3.1.2_6
        python27: 2.7.12 -> 2.7.13_1
        ...

Installed packages to be REINSTALLED:
        zfs-stats-1.2.2_1 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
        urlview-0.9.20131021_1 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
        unzip-6.0_7 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
        ttcp-1.12_1 (ABI changed: 'freebsd:10:x86:64' -> 'freebsd:11:x86:64')
        ...

Number of packages to be installed: 2
Number of packages to be upgraded: 38
Number of packages to be reinstalled: 64

The process will require 5 MiB more space.
110 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching zsh-5.3.1.txz: 100%    4 MiB   4.1MB/s    00:01
Fetching zfs-stats-1.2.2_1.txz: 100%    9 KiB   9.5kB/s    00:01
Fetching vim-lite-8.0.0134_1.txz: 100%    5 MiB   5.5MB/s    00:01
...
Checking integrity... done (0 conflicting)
[1/104] Reinstalling indexinfo-0.2.6...
[1/104] Extracting indexinfo-0.2.6: 100%
[2/104] Reinstalling libffi-3.2.1...
[2/104] Extracting libffi-3.2.1: 100%
[3/104] Upgrading gettext-runtime from 0.19.8.1 to 0.19.8.1_1...
[3/104] Extracting gettext-runtime-0.19.8.1_1: 100%
...

Finally we’ll upgrade any perl modules on the system from cpan.

$ sudo cpan-outdated | sudo cpanm

So, to recap the commands are as follows and in this order:

$ sudo freebsd-update fetch
$ sudo freebsd-update install
$ sudo freebsd-update upgrade -r 11.0-RELEASE
$ sudo freebsd-update install
$ sudo shutdown -r now
$ sudo freebsd-update install
$ sudo pkg upgrade
$ sudo cpan-outdated | sudo cpanm

Display “real” FreeBSD version

Typically when you want to find the current version of FreeBSD you type the following:

$ uname -r
10.3-RELEASE-p4

That’s great but it doesn’t tell the full story. If updates were made that do not affect the kernel, than it will not be reflected here. For instance, if the system had been upgraded to patch 12 I would not know that from looking at this. It makes it look like its only been updated to patch 4. Enter the “/usr/src/sys/conf/newvers.sh” file. This file gives a more clear picture of the system’s current state of affairs. I wrote a quick script that grabs the relevant info from this file and displays it similarly to the uname output.

#!/usr/local/bin/perl

use strict;
use warnings;

my $file = "/usr/src/sys/conf/newvers.sh";
my $cnt = 0;
my ($type, $rev, $branch);

open (FILE, "$file") or die "File $file does not exist!";
while (<FILE>) {
 chomp ($_);
     if ($_ =~ /^TYPE="(.*)?"/) {
         $type = $1;
         $cnt++;
     }
    if ($_ =~ /^REVISION="(.*)?"/ && $cnt == 1) {
         $rev = $1;
         $cnt++;
     }
     if ($_ =~ /^BRANCH="(.*)?"/ && $cnt == 2) {
         $branch = $1;
     }
}
close (FILE);

print "$type $rev $branch\n";

I put this script in /usr/local/sbin/hostver on all the systems I administer.

$ hostver
FreeBSD 10.3 RELEASE-p12

Now I always know the current patch level of the system. There are other ways to get this info (freebsd-version) but this works fine for me.

Monit (fix /var/log/messages)

Recently I discovered monit for FreeBSD, a monitoring system that is highly configurable and can be used to monitor various system happenings including service checks, disk space usage and process health. I installed it on all of the systems we have that require such monitoring. It has been very helpful in letting us know when things go offline.

The one issue I have noticed with it has to do with monitoring PostgreSQL. The PostgreSQL part of our config is setup like so:

# POSTGRESQL
check host PostgreSQL with address 127.0.0.1
    if failed ping then alert
    if failed port 5432 protocol pgsql then alert

So when it can’t connect to PostgreSQL we will receive an email message about it. Great! However if you look in the /var/log/messages you will notice something like this every 30 seconds (since we have monit setup to check everything in 30 second intervals):

Nov  1 00:12:41 blackbox postgres[70271]: [2-1] FATAL:  role "root" does not exist

To fix this and start cleaning up our log its pretty straight forward. Create that role:

psql -U pgsql template1
create role root login nocreaterole nocreatedb nosuperuser noinherit;

After this we now start seeing this error in the log:

Nov  9 12:47:50 blackbox postgres[94875]: [2-1] FATAL:  database "root" does not exist

To fix this we simple create that database:

create database root;

I also added this to the pg_hba.conf:

# TYPE  DATABASE        USER            ADDRESS                 METHOD
...
host    root            root            127.0.0.1/32            trust
...

That fixed our issues and now we no longer see those /var/log/messages. Monit wants to always connect using the “root” user and there’s no way to configure it to use a different user. So I came up with this as the workaround.

FreeBSD minor version upgrading

Upgrading FreeBSD to a new minor version is relatively simple. Here are the commands that will get you there: [upgrading from 10.1-p16]


[steve@blackbox:/var/empty]%uname -a
FreeBSD blackbox.cello.com 10.1-RELEASE-p16 FreeBSD 10.1-RELEASE-p16 #0: Tue Jul 17 05:25:45 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

sudo freebsd-update upgrade -r 10.2-RELEASE
sudo freebsd-update install
sudo reboot now
sudo freebsd-update install
sudo reboot now
sudo pkg upgrade

I experienced a couple issues along the way, but they were easily correctable. During the upgrade (first command) the process hung while attempting to download the software it needs. I completely lost my connection so I logged back in and ran it again with no issues. Then during the second “freebsd-update install” the same thing happened. Again, log back in and run it again. No issues. Finally, while running the last command to upgrade the packages on the system I ran into one package it would not upgrade. Running “pkg upgrade” twice seemed to ignore the problem.

[steve@blackbox:/var/empty]%sudo pkg upgrade
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 5 MiB 1.1MB/s 00:05
Processing entries: 100%
FreeBSD repository update completed. 24442 packages processed.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
pkg: 1.5.5 -> 1.5.6

The process will require 319 B more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching pkg-1.5.6.txz: 100% 2 MiB 1.2MB/s 00:02
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.5.5 to 1.5.6...
[1/1] Extracting pkg-1.5.6: 100%
Message for pkg-1.5.6:
If you are upgrading from the old package format, first run:

# pkg2ng
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
The following 32 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
postgresql93-client: 9.3.9

Installed packages to be UPGRADED:
...
curl: 7.43.0_2 -> 7.44.0
ca_root_nss: 3.19.2 -> 3.19.3
apache24: 2.4.16 -> 2.4.16_1

Installed packages to be REINSTALLED:
...
p5-DBD-Pg-3.5.1 (direct dependency changed: postgresql93-client)

The process will require 9 MiB more space.
43 MiB to be downloaded.

Proceed with this action? [y/N]: y
...
Fetching curl-7.44.0.txz: 100% 1 MiB 1.4MB/s 00:01
Fetching ca_root_nss-3.19.3.txz: 100% 334 KiB 341.7kB/s 00:01
Fetching apache24-2.4.16_1.txz: 100% 4 MiB 1.3MB/s 00:03
Fetching postgresql93-client-9.3.9.txz: 100% 2 MiB 2.0MB/s 00:01
Checking integrity... done (1 conflicting)
pkg: Cannot solve problem using SAT solver:
upgrade rule: upgrade local p5-DBD-Pg-3.5.1 to remote p5-DBD-Pg-3.5.1
cannot install package p5-DBD-Pg, remove it from request? [Y/n]: y
pkg: cannot find p5-DBD-Pg in the request
pkg: cannot solve job using SAT solver
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.

[steve@nprod:/var/empty]%sudo pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
Checking integrity... done (1 conflicting)
pkg: Cannot solve problem using SAT solver:
upgrade rule: upgrade local p5-DBD-Pg-3.5.1 to remote p5-DBD-Pg-3.5.1
cannot install package p5-DBD-Pg, remove it from request? [Y/n]:
pkg: cannot find p5-DBD-Pg in the request
pkg: cannot solve job using SAT solver
Checking integrity... done (0 conflicting)
Your packages are up to date.

And there you have it.

[steve@blackbox:/var/empty]%uname -a
FreeBSD blackbox.cello.com 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 09 09:55:07 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

Cookie management and saving state in Android apps.

I just wrote my first Android app using the ADT plugin for Eclipse. This particular app I wrote interacts with existing web services we deployed on our main production servers. I was having the hardest time maintaining a session state during web service calls on the back-end. All I needed was the following one line of code in my onCreate function of the Activity which calls the web service that sets up the session:

CookieHandler.setDefault(new CookieManager( null, CookiePolicy.ACCEPT_ALL));

Presto! State saved across all subsequent web service calls.

Windows boot up .. 70,000+ updates?!?!?

I normally leave my Windows 7 PC up and running at the office for various reasons. The other day I came into work and we had lost power. I was greeted with the following update message while booting:

2014-07-07 10.15.25

20140707_101330

20140707_101354(0)

Now, to be fair, it ran through all these updates in a flash which was nice. I’m assuming this happened because I always turn off auto updates because I loath when Windows decides to reboot me to do an auto update and I loose work. Anyone else notice these fly-by recently or was it just catching up on updates?

Cricket monitoring – old library reference

We recently shutdown one of many systems we have monitored by the free and open source Cricket monitoring software. I noticed we got no notification from Cricket about this shutdown, which was puzzling. Looking through the logs I saw this error message:

Can't load '/usr/local/lib/perl5/site_perl/5.16/mach/auto/RRDs/RRDs.so' for module RRDs: Shared object "libpixman-1.so.30" not found, required by "libpangocairo-1.0.so.0" at /usr/local/lib/perl5/5.16/mach/DynaLoader.pm line 190.
 at /usr/local/cricket/cricket/./collector line 32.
Compilation failed in require at /usr/local/cricket/cricket/./collector line 32.
BEGIN failed--compilation aborted at /usr/local/cricket/cricket/./collector line 32.

So it was clear that the pango library file libpangocairo-1.0.so.0 was looking for a pixman library file it couldn’t find. The first thing I did was check out which version of pixman we currently had installed:

steve@someserver-335: pkg info pixman
pixman-0.32.4_2
Name           : pixman
Version        : 0.32.4_2
Installed on   : Mon May  5 15:30:07 EDT 2014
Origin         : x11/pixman
Architecture   : freebsd:10:x86:32
Prefix         : /usr/local
Categories     : x11
Maintainer     : x11@FreeBSD.org
WWW            : http://www.freedesktop.org/Software/xlibs
Comment        : Low-level pixel manipulation library
Shared Libs provided:
        libpixman-1.so.0.32.4
Flat size      : 1.43MiB
Description    :
This package contains the pixman library.

Ok, so now I was able to confirm that pango was trying to reference the old version (libpixman-1.so.30) of the pixman library, even though we have a newer version (libpixman-1.so.32.4) installed. Time to rebuild the pango port to update that reference.

$ sudo portmaster pango

And finally I ran the following to test Cricket out:

$ cd /usr/local/cricket/cricket
$ ./collect-subtrees normal

All was back to normal!

JSON-ing and request headers

We have this web widget that calls a particular webservice and does a JSON post. This was failing miserably on the back-end. For some reason, the data that was ending up in the session was not ending up in $session->params->{POSTDATA} but instead $session->params->{keywords}. In turn, our webservice was bombing when it was trying to access the data from POSTDATA since it wasn’t there in the session. Here’s an excerpt of the Perl data dump for the bad session:

_QUERY       => bless({
                      ".charset"     => "ISO-8859-1",
                      ".cookies"     => {
                                          sid => bless({
                                                   name  => "sid",
                                                   path  => "/",
                                                   value => ["XXXXXXXXXXXXXXXXXXXXX"],
                                                 }, "CGI::Cookie"),
                                        },
                      ".fieldnames"  => {},
                      ".parameters"  => ["keywords"],
                      "escape"       => 1,
                      "param"        => {
                                          keywords => [
                                            "{\"fname\":\"Steve\",\"lname\":\"Dickinson\",\"phone\":\"5557771000\",\"email\":\"steve\@someplace.com\",\"date\":\"2014-07-05\",\"time\":\"18:00:00\",\"size\":2,\"notes\":\"notes\"}",
                                          ],
                                        },
                      "use_tempfile" => 1,
                    }, "CGI"),
    _STATUS      => 5,
  }, "CGI::Session");

It should have instead looked like this:

_QUERY       => bless({
                      ".charset"     => "ISO-8859-1",
                      ".cookies"     => {
                                          sid => bless({
                                                   name  => "sid",
                                                   path  => "/",
                                                   value => ["XXXXXXXXXXXXXXXXXXXXX"],
                                                 }, "CGI::Cookie"),
                                        },
                      ".fieldnames"  => {},
                      ".parameters"  => ["keywords"],
                      "escape"       => 1,
                      "param"        => {
                                          POSTDATA => [
                                            "{\"fname\":\"Steve\",\"lname\":\"Dickinson\",\"phone\":\"5557771000\",\"email\":\"steve\@someplace.com\",\"date\":\"2014-07-05\",\"time\":\"18:00:00\",\"size\":2,\"notes\":\"notes\"}",
                                          ],
                                        },
                      "use_tempfile" => 1,
                    }, "CGI"),
    _STATUS      => 5,
  }, "CGI::Session");

During this time I was also building a similar Android app to mimic how this widget works. The funny thing was, that the Android app worked but the widget didn’t. In thinking back to my app code I soon realized the issue. In the app code I had correctly set the header like so:

httppost.setHeader("Accept", "application/json");
httppost.setHeader("Content-type", "application/json");

The widget had no such code for this webservice call. So I added the following handleAs & headers definitions to the developer’s JavaScript code:

var req = {
    data: JSON.stringify(requestData),
        handleAs: 'json',
        headers: {
            "Accept": "application/json",
            "Content-Type": "application/json"
            }
    };
request.post('http://somedomain.com/webservice_call', req).then(

That’s all it took!

PC reinstall – the importance of setting the date correctly.

Today I had to do a Windows 7 re-install. Nothing new there. The PC was running very sluggish and it was time to wipe everything clean. So when it came time to set the date I incorrectly set the Meridiem Indicator, or period, to AM (2:30 AM) instead of PM (2:30 PM). This made the PC’s time 12 hours in the past. This has a great affect on online services such as LastPass and its kin XMarks. I noticed I couldn’t login to LastPass or sync bookmarks using XMarks via their respective FireFox add-ons. I knew the passwords I was entering were correct. So I then tried going to LastPass’s website and got this:

lastpass_error_bad_date

Oddly enough I recalled this same issue coming up on a recent Tech Guy podcast. Leo Laport’s recommendation was to check the system time and low and behold he was right! I then corrected the period to PM (2:30 PM) and everything was back to normal.