Monit (fix /var/log/messages)

Recently I discovered monit for FreeBSD, a monitoring system that is highly configurable and can be used to monitor various system happenings including service checks, disk space usage and process health. I installed it on all of the systems we have that require such monitoring. It has been very helpful in letting us know when things go offline.

The one issue I have noticed with it has to do with monitoring PostgreSQL. The PostgreSQL part of our config is setup like so:

# POSTGRESQL
check host PostgreSQL with address 127.0.0.1
    if failed ping then alert
    if failed port 5432 protocol pgsql then alert

So when it can’t connect to PostgreSQL we will receive an email message about it. Great! However if you look in the /var/log/messages you will notice something like this every 30 seconds (since we have monit setup to check everything in 30 second intervals):

Nov  1 00:12:41 blackbox postgres[70271]: [2-1] FATAL:  role "root" does not exist

To fix this and start cleaning up our log its pretty straight forward. Create that role:

psql -U pgsql template1
create role root login nocreaterole nocreatedb nosuperuser noinherit;

After this we now start seeing this error in the log:

Nov  9 12:47:50 blackbox postgres[94875]: [2-1] FATAL:  database "root" does not exist

To fix this we simple create that database:

create database root;

I also added this to the pg_hba.conf:

# TYPE  DATABASE        USER            ADDRESS                 METHOD
...
host    root            root            127.0.0.1/32            trust
...

That fixed our issues and now we no longer see those /var/log/messages. Monit wants to always connect using the “root” user and there’s no way to configure it to use a different user. So I came up with this as the workaround.

FreeBSD minor version upgrading

Upgrading FreeBSD to a new minor version is relatively simple. Here are the commands that will get you there: [upgrading from 10.1-p16]


[steve@blackbox:/var/empty]%uname -a
FreeBSD blackbox.cello.com 10.1-RELEASE-p16 FreeBSD 10.1-RELEASE-p16 #0: Tue Jul 17 05:25:45 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

sudo freebsd-update upgrade -r 10.2-RELEASE
sudo freebsd-update install
sudo reboot now
sudo freebsd-update install
sudo reboot now
sudo pkg upgrade

I experienced a couple issues along the way, but they were easily correctable. During the upgrade (first command) the process hung while attempting to download the software it needs. I completely lost my connection so I logged back in and ran it again with no issues. Then during the second “freebsd-update install” the same thing happened. Again, log back in and run it again. No issues. Finally, while running the last command to upgrade the packages on the system I ran into one package it would not upgrade. Running “pkg upgrade” twice seemed to ignore the problem.

[steve@blackbox:/var/empty]%sudo pkg upgrade
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 5 MiB 1.1MB/s 00:05
Processing entries: 100%
FreeBSD repository update completed. 24442 packages processed.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
pkg: 1.5.5 -> 1.5.6

The process will require 319 B more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching pkg-1.5.6.txz: 100% 2 MiB 1.2MB/s 00:02
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.5.5 to 1.5.6...
[1/1] Extracting pkg-1.5.6: 100%
Message for pkg-1.5.6:
If you are upgrading from the old package format, first run:

# pkg2ng
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
The following 32 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
postgresql93-client: 9.3.9

Installed packages to be UPGRADED:
...
curl: 7.43.0_2 -> 7.44.0
ca_root_nss: 3.19.2 -> 3.19.3
apache24: 2.4.16 -> 2.4.16_1

Installed packages to be REINSTALLED:
...
p5-DBD-Pg-3.5.1 (direct dependency changed: postgresql93-client)

The process will require 9 MiB more space.
43 MiB to be downloaded.

Proceed with this action? [y/N]: y
...
Fetching curl-7.44.0.txz: 100% 1 MiB 1.4MB/s 00:01
Fetching ca_root_nss-3.19.3.txz: 100% 334 KiB 341.7kB/s 00:01
Fetching apache24-2.4.16_1.txz: 100% 4 MiB 1.3MB/s 00:03
Fetching postgresql93-client-9.3.9.txz: 100% 2 MiB 2.0MB/s 00:01
Checking integrity... done (1 conflicting)
pkg: Cannot solve problem using SAT solver:
upgrade rule: upgrade local p5-DBD-Pg-3.5.1 to remote p5-DBD-Pg-3.5.1
cannot install package p5-DBD-Pg, remove it from request? [Y/n]: y
pkg: cannot find p5-DBD-Pg in the request
pkg: cannot solve job using SAT solver
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.

[steve@nprod:/var/empty]%sudo pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
Checking integrity... done (1 conflicting)
pkg: Cannot solve problem using SAT solver:
upgrade rule: upgrade local p5-DBD-Pg-3.5.1 to remote p5-DBD-Pg-3.5.1
cannot install package p5-DBD-Pg, remove it from request? [Y/n]:
pkg: cannot find p5-DBD-Pg in the request
pkg: cannot solve job using SAT solver
Checking integrity... done (0 conflicting)
Your packages are up to date.

And there you have it.

[steve@blackbox:/var/empty]%uname -a
FreeBSD blackbox.cello.com 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 09 09:55:07 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

PC reinstall – the importance of setting the date correctly.

Today I had to do a Windows 7 re-install. Nothing new there. The PC was running very sluggish and it was time to wipe everything clean. So when it came time to set the date I incorrectly set the Meridiem Indicator, or period, to AM (2:30 AM) instead of PM (2:30 PM). This made the PC’s time 12 hours in the past. This has a great affect on online services such as LastPass and its kin XMarks. I noticed I couldn’t login to LastPass or sync bookmarks using XMarks via their respective FireFox add-ons. I knew the passwords I was entering were correct. So I then tried going to LastPass’s website and got this:

Oddly enough I recalled this same issue coming up on a recent Tech Guy podcast. Leo Laport’s recommendation was to check the system time and low and behold he was right! I then corrected the period to PM (2:30 PM) and everything was back to normal.