Monit (fix /var/log/messages)

Recently I discovered monit for FreeBSD, a monitoring system that is highly configurable and can be used to monitor various system happenings including service checks, disk space usage and process health. I installed it on all of the systems we have that require such monitoring. It has been very helpful in letting us know when things go offline.

The one issue I have noticed with it has to do with monitoring PostgreSQL. The PostgreSQL part of our config is setup like so:

# POSTGRESQL
check host PostgreSQL with address 127.0.0.1
    if failed ping then alert
    if failed port 5432 protocol pgsql then alert

So when it can’t connect to PostgreSQL we will receive an email message about it. Great! However if you look in the /var/log/messages you will notice something like this every 30 seconds (since we have monit setup to check everything in 30 second intervals):

Nov  1 00:12:41 blackbox postgres[70271]: [2-1] FATAL:  role "root" does not exist

To fix this and start cleaning up our log its pretty straight forward. Create that role:

psql -U pgsql template1
create role root login nocreaterole nocreatedb nosuperuser noinherit;

After this we now start seeing this error in the log:

Nov  9 12:47:50 blackbox postgres[94875]: [2-1] FATAL:  database "root" does not exist

To fix this we simple create that database:

create database root;

I also added this to the pg_hba.conf:

# TYPE  DATABASE        USER            ADDRESS                 METHOD
...
host    root            root            127.0.0.1/32            trust
...

That fixed our issues and now we no longer see those /var/log/messages. Monit wants to always connect using the “root” user and there’s no way to configure it to use a different user. So I came up with this as the workaround.

Cricket monitoring – old library reference

We recently shutdown one of many systems we have monitored by the free and open source Cricket monitoring software. I noticed we got no notification from Cricket about this shutdown, which was puzzling. Looking through the logs I saw this error message:

Can't load '/usr/local/lib/perl5/site_perl/5.16/mach/auto/RRDs/RRDs.so' for module RRDs: Shared object "libpixman-1.so.30" not found, required by "libpangocairo-1.0.so.0" at /usr/local/lib/perl5/5.16/mach/DynaLoader.pm line 190.
 at /usr/local/cricket/cricket/./collector line 32.
Compilation failed in require at /usr/local/cricket/cricket/./collector line 32.
BEGIN failed--compilation aborted at /usr/local/cricket/cricket/./collector line 32.

So it was clear that the pango library file libpangocairo-1.0.so.0 was looking for a pixman library file it couldn’t find. The first thing I did was check out which version of pixman we currently had installed:

steve@someserver-335: pkg info pixman
pixman-0.32.4_2
Name           : pixman
Version        : 0.32.4_2
Installed on   : Mon May  5 15:30:07 EDT 2014
Origin         : x11/pixman
Architecture   : freebsd:10:x86:32
Prefix         : /usr/local
Categories     : x11
Maintainer     : x11@FreeBSD.org
WWW            : http://www.freedesktop.org/Software/xlibs
Comment        : Low-level pixel manipulation library
Shared Libs provided:
        libpixman-1.so.0.32.4
Flat size      : 1.43MiB
Description    :
This package contains the pixman library.

Ok, so now I was able to confirm that pango was trying to reference the old version (libpixman-1.so.30) of the pixman library, even though we have a newer version (libpixman-1.so.32.4) installed. Time to rebuild the pango port to update that reference.

$ sudo portmaster pango

And finally I ran the following to test Cricket out:

$ cd /usr/local/cricket/cricket
$ ./collect-subtrees normal

All was back to normal!