Windows boot up .. 70,000+ updates?!?!?

I normally leave my Windows 7 PC up and running at the office for various reasons. The other day I came into work and we had lost power. I was greeted with the following update message while booting:

2014-07-07 10.15.25



Now, to be fair, it ran through all these updates in a flash which was nice. I’m assuming this happened because I always turn off auto updates because I loath when Windows decides to reboot me to do an auto update and I loose work. Anyone else notice these fly-by recently or was it just catching up on updates?

Cricket monitoring – old library reference

We recently shutdown one of many systems we have monitored by the free and open source Cricket monitoring software. I noticed we got no notification from Cricket about this shutdown, which was puzzling. Looking through the logs I saw this error message:

Can't load '/usr/local/lib/perl5/site_perl/5.16/mach/auto/RRDs/' for module RRDs: Shared object "" not found, required by "" at /usr/local/lib/perl5/5.16/mach/ line 190.
 at /usr/local/cricket/cricket/./collector line 32.
Compilation failed in require at /usr/local/cricket/cricket/./collector line 32.
BEGIN failed--compilation aborted at /usr/local/cricket/cricket/./collector line 32.

So it was clear that the pango library file was looking for a pixman library file it couldn’t find. The first thing I did was check out which version of pixman we currently had installed:

steve@someserver-335: pkg info pixman
Name           : pixman
Version        : 0.32.4_2
Installed on   : Mon May  5 15:30:07 EDT 2014
Origin         : x11/pixman
Architecture   : freebsd:10:x86:32
Prefix         : /usr/local
Categories     : x11
Maintainer     :
WWW            :
Comment        : Low-level pixel manipulation library
Shared Libs provided:
Flat size      : 1.43MiB
Description    :
This package contains the pixman library.

Ok, so now I was able to confirm that pango was trying to reference the old version ( of the pixman library, even though we have a newer version ( installed. Time to rebuild the pango port to update that reference.

$ sudo portmaster pango

And finally I ran the following to test Cricket out:

$ cd /usr/local/cricket/cricket
$ ./collect-subtrees normal

All was back to normal!

JSON-ing and request headers

We have this web widget that calls a particular webservice and does a JSON post. This was failing miserably on the back-end. For some reason, the data that was ending up in the session was not ending up in $session->params->{POSTDATA} but instead $session->params->{keywords}. In turn, our webservice was bombing when it was trying to access the data from POSTDATA since it wasn’t there in the session. Here’s an excerpt of the Perl data dump for the bad session:

_QUERY       => bless({
                      ".charset"     => "ISO-8859-1",
                      ".cookies"     => {
                                          sid => bless({
                                                   name  => "sid",
                                                   path  => "/",
                                                   value => ["XXXXXXXXXXXXXXXXXXXXX"],
                                                 }, "CGI::Cookie"),
                      ".fieldnames"  => {},
                      ".parameters"  => ["keywords"],
                      "escape"       => 1,
                      "param"        => {
                                          keywords => [
                      "use_tempfile" => 1,
                    }, "CGI"),
    _STATUS      => 5,
  }, "CGI::Session");

It should have instead looked like this:

_QUERY       => bless({
                      ".charset"     => "ISO-8859-1",
                      ".cookies"     => {
                                          sid => bless({
                                                   name  => "sid",
                                                   path  => "/",
                                                   value => ["XXXXXXXXXXXXXXXXXXXXX"],
                                                 }, "CGI::Cookie"),
                      ".fieldnames"  => {},
                      ".parameters"  => ["keywords"],
                      "escape"       => 1,
                      "param"        => {
                                          POSTDATA => [
                      "use_tempfile" => 1,
                    }, "CGI"),
    _STATUS      => 5,
  }, "CGI::Session");

During this time I was also building a similar Android app to mimic how this widget works. The funny thing was, that the Android app worked but the widget didn’t. In thinking back to my app code I soon realized the issue. In the app code I had correctly set the header like so:

httppost.setHeader("Accept", "application/json");
httppost.setHeader("Content-type", "application/json");

The widget had no such code for this webservice call. So I added the following handleAs & headers definitions to the developer’s JavaScript code:

var req = {
    data: JSON.stringify(requestData),
        handleAs: 'json',
        headers: {
            "Accept": "application/json",
            "Content-Type": "application/json"
    };'', req).then(

That’s all it took!

PC reinstall – the importance of setting the date correctly.

Today I had to do a Windows 7 re-install. Nothing new there. The PC was running very sluggish and it was time to wipe everything clean. So when it came time to set the date I incorrectly set the Meridiem Indicator, or period, to AM (2:30 AM) instead of PM (2:30 PM). This made the PC’s time 12 hours in the past. This has a great affect on online services such as LastPass and its kin XMarks. I noticed I couldn’t login to LastPass or sync bookmarks using XMarks via their respective FireFox add-ons. I knew the passwords I was entering were correct. So I then tried going to LastPass’s website and got this:


Oddly enough I recalled this same issue coming up on a recent Tech Guy podcast. Leo Laport’s recommendation was to check the system time and low and behold he was right! I then corrected the period to PM (2:30 PM) and everything was back to normal.

NTP attack

Yep we got bit by this one. It brought our network to its knees. Fortunately the fix was easy. First, to stop the attack we turned off ntp temporarily:

service ntp stop

Then we added this to our /etc/ntp.conf file:

Disable monitor
Restrict -4 default kod notrap nomodify nopeer noquery

Then we started ntp:

service ntp start

The system returned to normal.

FreeBSD update broke Apache perl modules

After a recent freebsd update fetch && freebsd update install Apache would not restart properly. It was complaining about missing perl modules it relied on. So we rebuilt the perl port and all its dependencies.

sudo portmaster -m BATCH=yes --no-confirm -D -r perl

The -m BATCH=yes chooses defaults at setup screens and bypasses them, --no-confirm avoids prompting at the command line, -D keeps distfiles in tact and -r updates all its port dependencies. Once this was done Apache fired right up!

ZFS zpool crash

Recently we had an issue where one of our servers unexpectedly crashed and we had to hard reboot it. When it rebooted we began seeing kernel panics and it would not complete its boot cycle. In comes my mfsbsd stick to the rescue! The machine in question was running FreeBSD 10. We had just built it so we had one of these sticks handy. To create a mfsbsd stick yourself grab the image here:

FreeBSD Build Image Instructions:

We ended up using Win32 Disk Imager:

Once we got booted off the usb stick to a command prompt (beyond the scope of this post but simple, follow the prompts) we simply did the following to remount the zfs pools that were fubar’d:
zpool import -f -a

All zfs pools imported with no problems. Then we rebooted the machine as normal and voila, no errors!

Windows 8 Power User Menu

Bring up the power user menu in Windows 8:

A life saver when you are a UNIX admin/user trying to setup a Windows 8 machine.

Some of the more useful admin shortcuts displayed are:
Event Viewer
Device Management
Disk Management
Command Prompt
Task Manager
Control Panel

Fun with a FreeBSD named port upgrade.

::Begin Prologue::
Recently we performed the following on our DNS server running FreeBSD 9.1:
freebsd update fetch && freebsd update install
Always smooth sailing. 🙂
::End Prologue::

So, today we got a new laptop which meant setting up the DNS records for it to run on our network. We added the necessary forward and reverse DNS records in named. After doing so, the normal practice is to run rndc reload so the system will reload and recognize the new DNS records. A very simple process. But today was a very different story. Today, running rndc reload produced this error message:

rndc: neither /usr/local/etc/rndc.conf nor /usr/local/etc/rndc.key was found

Huh? Out of the blue rndc does this? Well guess what, the named port got upgraded, so go figure.

Now we noticed the symbolic link /etc/named/rndc.key to /usr/local/etc/rndc.key was missing, so let’s create it:
ln -s /etc/named.rndc.key /usr/local/etc/rndc.key

Now running rndc reload produced this error:

rndc: 'reload' failed: not found

It would have been nice if the message had told me exactly what file was not found. But looking in the /var/log/daemon.log file pointed me in the right direction as to what was going wrong:

May 19 15:11:19 ns named[61129]: received control channel command 'reload'
May 19 15:11:19 ns named[61129]: loading configuration from '/usr/local/etc/named.conf'
May 19 15:11:19 ns named[61129]: open: /usr/local/etc/named.conf: file not found
May 19 15:11:19 ns named[61129]: reloading configuration failed: file not found

Normally it should look in /etc/named/named.conf for the configuration file but our new version now has a new path it is looking in.

So we then added the following to the /etc/rc.conf file:


And ran:

service named restart
Stopping named.
Starting named.
/etc/rc.d/named: WARNING: failed to start named

Blamo! Named stopped and could not start back up, no thanks to our new config line. It again produced the same error in the log file. And now, there was no named service running. Double ugh.

This command fixed that and pointed it to the correct config file:

/usr/local/sbin/named -t /var/named -u bind -c /etc/namedb/named.conf

Now the named service is up and running and rndc reload runs as normal. Great!
Now we just need to do a reboot of the system (when there is little traffic) to see if named starts up normally after adding that line to /etc/rc.conf.