Monit (fix /var/log/messages)

Recently I discovered monit for FreeBSD, a monitoring system that is highly configurable and can be used to monitor various system happenings including service checks, disk space usage and process health. I installed it on all of the systems we have that require such monitoring. It has been very helpful in letting us know when things go offline.

The one issue I have noticed with it has to do with monitoring PostgreSQL. The PostgreSQL part of our config is setup like so:

# POSTGRESQL
check host PostgreSQL with address 127.0.0.1
    if failed ping then alert
    if failed port 5432 protocol pgsql then alert

So when it can’t connect to PostgreSQL we will receive an email message about it. Great! However if you look in the /var/log/messages you will notice something like this every 30 seconds (since we have monit setup to check everything in 30 second intervals):

Nov  1 00:12:41 blackbox postgres[70271]: [2-1] FATAL:  role "root" does not exist

To fix this and start cleaning up our log its pretty straight forward. Create that role:

psql -U pgsql template1
create role root login nocreaterole nocreatedb nosuperuser noinherit;

After this we now start seeing this error in the log:

Nov  9 12:47:50 blackbox postgres[94875]: [2-1] FATAL:  database "root" does not exist

To fix this we simple create that database:

create database root;

I also added this to the pg_hba.conf:

# TYPE  DATABASE        USER            ADDRESS                 METHOD
...
host    root            root            127.0.0.1/32            trust
...

That fixed our issues and now we no longer see those /var/log/messages. Monit wants to always connect using the “root” user and there’s no way to configure it to use a different user. So I came up with this as the workaround.

FreeBSD minor version upgrading

Upgrading FreeBSD to a new minor version is relatively simple. Here are the commands that will get you there: [upgrading from 10.1-p16]


[steve@blackbox:/var/empty]%uname -a
FreeBSD blackbox.cello.com 10.1-RELEASE-p16 FreeBSD 10.1-RELEASE-p16 #0: Tue Jul 17 05:25:45 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64

sudo freebsd-update upgrade -r 10.2-RELEASE
sudo freebsd-update install
sudo reboot now
sudo freebsd-update install
sudo reboot now
sudo pkg upgrade

I experienced a couple issues along the way, but they were easily correctable. During the upgrade (first command) the process hung while attempting to download the software it needs. I completely lost my connection so I logged back in and ran it again with no issues. Then during the second “freebsd-update install” the same thing happened. Again, log back in and run it again. No issues. Finally, while running the last command to upgrade the packages on the system I ran into one package it would not upgrade. Running “pkg upgrade” twice seemed to ignore the problem.

[steve@blackbox:/var/empty]%sudo pkg upgrade
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 5 MiB 1.1MB/s 00:05
Processing entries: 100%
FreeBSD repository update completed. 24442 packages processed.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
pkg: 1.5.5 -> 1.5.6

The process will require 319 B more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching pkg-1.5.6.txz: 100% 2 MiB 1.2MB/s 00:02
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.5.5 to 1.5.6...
[1/1] Extracting pkg-1.5.6: 100%
Message for pkg-1.5.6:
If you are upgrading from the old package format, first run:

# pkg2ng
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
The following 32 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
postgresql93-client: 9.3.9

Installed packages to be UPGRADED:
...
curl: 7.43.0_2 -> 7.44.0
ca_root_nss: 3.19.2 -> 3.19.3
apache24: 2.4.16 -> 2.4.16_1

Installed packages to be REINSTALLED:
...
p5-DBD-Pg-3.5.1 (direct dependency changed: postgresql93-client)

The process will require 9 MiB more space.
43 MiB to be downloaded.

Proceed with this action? [y/N]: y
...
Fetching curl-7.44.0.txz: 100% 1 MiB 1.4MB/s 00:01
Fetching ca_root_nss-3.19.3.txz: 100% 334 KiB 341.7kB/s 00:01
Fetching apache24-2.4.16_1.txz: 100% 4 MiB 1.3MB/s 00:03
Fetching postgresql93-client-9.3.9.txz: 100% 2 MiB 2.0MB/s 00:01
Checking integrity... done (1 conflicting)
pkg: Cannot solve problem using SAT solver:
upgrade rule: upgrade local p5-DBD-Pg-3.5.1 to remote p5-DBD-Pg-3.5.1
cannot install package p5-DBD-Pg, remove it from request? [Y/n]: y
pkg: cannot find p5-DBD-Pg in the request
pkg: cannot solve job using SAT solver
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.

[steve@nprod:/var/empty]%sudo pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (32 candidates): 100%
Processing candidates (32 candidates): 100%
Checking integrity... done (1 conflicting)
pkg: Cannot solve problem using SAT solver:
upgrade rule: upgrade local p5-DBD-Pg-3.5.1 to remote p5-DBD-Pg-3.5.1
cannot install package p5-DBD-Pg, remove it from request? [Y/n]:
pkg: cannot find p5-DBD-Pg in the request
pkg: cannot solve job using SAT solver
Checking integrity... done (0 conflicting)
Your packages are up to date.

And there you have it.

[steve@blackbox:/var/empty]%uname -a
FreeBSD blackbox.cello.com 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 09 09:55:07 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

Cookie management and saving state in Android apps.

I just wrote my first Android app using the ADT plugin for Eclipse. This particular app I wrote interacts with existing web services we deployed on our main production servers. I was having the hardest time maintaining a session state during web service calls on the back-end. All I needed was the following one line of code in my onCreate function of the Activity which calls the web service that sets up the session:

CookieHandler.setDefault(new CookieManager( null, CookiePolicy.ACCEPT_ALL));

Presto! State saved across all subsequent web service calls.

Windows boot up .. 70,000+ updates?!?!?

I normally leave my Windows 7 PC up and running at the office for various reasons. The other day I came into work and we had lost power. I was greeted with the following update message while booting:

2014-07-07 10.15.25

20140707_101330

20140707_101354(0)

Now, to be fair, it ran through all these updates in a flash which was nice. I’m assuming this happened because I always turn off auto updates because I loath when Windows decides to reboot me to do an auto update and I loose work. Anyone else notice these fly-by recently or was it just catching up on updates?

Cricket monitoring – old library reference

We recently shutdown one of many systems we have monitored by the free and open source Cricket monitoring software. I noticed we got no notification from Cricket about this shutdown, which was puzzling. Looking through the logs I saw this error message:

Can't load '/usr/local/lib/perl5/site_perl/5.16/mach/auto/RRDs/RRDs.so' for module RRDs: Shared object "libpixman-1.so.30" not found, required by "libpangocairo-1.0.so.0" at /usr/local/lib/perl5/5.16/mach/DynaLoader.pm line 190.
 at /usr/local/cricket/cricket/./collector line 32.
Compilation failed in require at /usr/local/cricket/cricket/./collector line 32.
BEGIN failed--compilation aborted at /usr/local/cricket/cricket/./collector line 32.

So it was clear that the pango library file libpangocairo-1.0.so.0 was looking for a pixman library file it couldn’t find. The first thing I did was check out which version of pixman we currently had installed:

steve@someserver-335: pkg info pixman
pixman-0.32.4_2
Name           : pixman
Version        : 0.32.4_2
Installed on   : Mon May  5 15:30:07 EDT 2014
Origin         : x11/pixman
Architecture   : freebsd:10:x86:32
Prefix         : /usr/local
Categories     : x11
Maintainer     : x11@FreeBSD.org
WWW            : http://www.freedesktop.org/Software/xlibs
Comment        : Low-level pixel manipulation library
Shared Libs provided:
        libpixman-1.so.0.32.4
Flat size      : 1.43MiB
Description    :
This package contains the pixman library.

Ok, so now I was able to confirm that pango was trying to reference the old version (libpixman-1.so.30) of the pixman library, even though we have a newer version (libpixman-1.so.32.4) installed. Time to rebuild the pango port to update that reference.

$ sudo portmaster pango

And finally I ran the following to test Cricket out:

$ cd /usr/local/cricket/cricket
$ ./collect-subtrees normal

All was back to normal!

JSON-ing and request headers

We have this web widget that calls a particular webservice and does a JSON post. This was failing miserably on the back-end. For some reason, the data that was ending up in the session was not ending up in $session->params->{POSTDATA} but instead $session->params->{keywords}. In turn, our webservice was bombing when it was trying to access the data from POSTDATA since it wasn’t there in the session. Here’s an excerpt of the Perl data dump for the bad session:

_QUERY       => bless({
                      ".charset"     => "ISO-8859-1",
                      ".cookies"     => {
                                          sid => bless({
                                                   name  => "sid",
                                                   path  => "/",
                                                   value => ["XXXXXXXXXXXXXXXXXXXXX"],
                                                 }, "CGI::Cookie"),
                                        },
                      ".fieldnames"  => {},
                      ".parameters"  => ["keywords"],
                      "escape"       => 1,
                      "param"        => {
                                          keywords => [
                                            "{\"fname\":\"Steve\",\"lname\":\"Dickinson\",\"phone\":\"5557771000\",\"email\":\"steve\@someplace.com\",\"date\":\"2014-07-05\",\"time\":\"18:00:00\",\"size\":2,\"notes\":\"notes\"}",
                                          ],
                                        },
                      "use_tempfile" => 1,
                    }, "CGI"),
    _STATUS      => 5,
  }, "CGI::Session");

It should have instead looked like this:

_QUERY       => bless({
                      ".charset"     => "ISO-8859-1",
                      ".cookies"     => {
                                          sid => bless({
                                                   name  => "sid",
                                                   path  => "/",
                                                   value => ["XXXXXXXXXXXXXXXXXXXXX"],
                                                 }, "CGI::Cookie"),
                                        },
                      ".fieldnames"  => {},
                      ".parameters"  => ["keywords"],
                      "escape"       => 1,
                      "param"        => {
                                          POSTDATA => [
                                            "{\"fname\":\"Steve\",\"lname\":\"Dickinson\",\"phone\":\"5557771000\",\"email\":\"steve\@someplace.com\",\"date\":\"2014-07-05\",\"time\":\"18:00:00\",\"size\":2,\"notes\":\"notes\"}",
                                          ],
                                        },
                      "use_tempfile" => 1,
                    }, "CGI"),
    _STATUS      => 5,
  }, "CGI::Session");

During this time I was also building a similar Android app to mimic how this widget works. The funny thing was, that the Android app worked but the widget didn’t. In thinking back to my app code I soon realized the issue. In the app code I had correctly set the header like so:

httppost.setHeader("Accept", "application/json");
httppost.setHeader("Content-type", "application/json");

The widget had no such code for this webservice call. So I added the following handleAs & headers definitions to the developer’s JavaScript code:

var req = {
    data: JSON.stringify(requestData),
        handleAs: 'json',
        headers: {
            "Accept": "application/json",
            "Content-Type": "application/json"
            }
    };
request.post('http://somedomain.com/webservice_call', req).then(

That’s all it took!

PC reinstall – the importance of setting the date correctly.

Today I had to do a Windows 7 re-install. Nothing new there. The PC was running very sluggish and it was time to wipe everything clean. So when it came time to set the date I incorrectly set the Meridiem Indicator, or period, to AM (2:30 AM) instead of PM (2:30 PM). This made the PC’s time 12 hours in the past. This has a great affect on online services such as LastPass and its kin XMarks. I noticed I couldn’t login to LastPass or sync bookmarks using XMarks via their respective FireFox add-ons. I knew the passwords I was entering were correct. So I then tried going to LastPass’s website and got this:

lastpass_error_bad_date

Oddly enough I recalled this same issue coming up on a recent Tech Guy podcast. Leo Laport’s recommendation was to check the system time and low and behold he was right! I then corrected the period to PM (2:30 PM) and everything was back to normal.

NTP attack

Yep we got bit by this one. It brought our network to its knees. Fortunately the fix was easy. First, to stop the attack we turned off ntp temporarily:

service ntp stop

Then we added this to our /etc/ntp.conf file:

Disable monitor
Restrict -4 default kod notrap nomodify nopeer noquery

Then we started ntp:

service ntp start

The system returned to normal.

FreeBSD update broke Apache perl modules

After a recent freebsd update fetch && freebsd update install Apache would not restart properly. It was complaining about missing perl modules it relied on. So we rebuilt the perl port and all its dependencies.

sudo portmaster -m BATCH=yes --no-confirm -D -r perl

The -m BATCH=yes chooses defaults at setup screens and bypasses them, --no-confirm avoids prompting at the command line, -D keeps distfiles in tact and -r updates all its port dependencies. Once this was done Apache fired right up!

ZFS zpool crash

Recently we had an issue where one of our servers unexpectedly crashed and we had to hard reboot it. When it rebooted we began seeing kernel panics and it would not complete its boot cycle. In comes my mfsbsd stick to the rescue! The machine in question was running FreeBSD 10. We had just built it so we had one of these sticks handy. To create a mfsbsd stick yourself grab the image here:
http://mfsbsd.vx.sk

FreeBSD Build Image Instructions:
http://www.freebsd.org/doc/en/articles/remote-install/preparation.html

We ended up using Win32 Disk Imager:
https://sourceforge.net/projects/win32diskimager/

Once we got booted off the usb stick to a command prompt (beyond the scope of this post but simple, follow the prompts) we simply did the following to remount the zfs pools that were fubar’d:
zpool import -f -a

All zfs pools imported with no problems. Then we rebooted the machine as normal and voila, no errors!