SSL Certificate Expiry Warning Script

With the increasing trend of SSL on the web, where Google values SSL sites higher and you can have your site be added to the HSTS preload list (the browser will first try HTTPS before trying HTTP), it is a good idea to start using SSL yourself.

The downside: you need to get a certificate through a (pre-trusted by the browser) CA, or certificate authority. This usually costs money, though there are some services that give you a certificate for free. The free certificates only last for one year or less, this means you need to request and install a new certificate frequently, especially when you have multiple domains.

Now it can happen to anyone, even Microsoft (Windows Azure Service Disruption from Expired Certificate), that you forget to renew (and update) your certificate in time.

There is a nice service called certalert.me (interestingly enough not over HTTPS) that will send you an e-mail when a certificate is due to be updated. But as with any web service, unfortunately you can never be sure how long it's going to live.

So, I have created a script that I run through a cronjob every day that will send me a notification e-mail several times in advance (1 day and 2 7 14 30 60 days ahead), so that you are not dependent on a third party to get notified about expiries. As it is supposed to be with cronjobs, there is no output when there is nothing to report (thus no e-mail).

Here is the script (download warn_about_certificate_expiry.sh):


#!/bin/sh 

CertExpiries=$(mktemp)
for i in /etc/certificates/*.pem; do
	echo $(basename $i): $(openssl x509 -in $i -inform PEM -text -noout -enddate | grep "Not After" | tail -1 | awk '{print $4, $5, $7}') >> $CertExpiries
done

Date=$(date -ud "+1 day" | awk '{print $2, $3, $6}')
Expiries=$(grep "$Date" $CertExpiries)
if [ $? -eq 0 ]; then 
	echo These Certificates expire TOMORROW!
	echo $Expiries
	echo
fi
for i in 2 7 14 30 60; do
	Date=$(date -ud "+$i day" | awk '{print $2, $3, $6}')
	Expiries=$(grep "$Date" $CertExpiries)
	if [ $? -eq 0 ]; then 
		echo These Certificates expire in $i days:
		echo $Expiries
		echo
	fi
done
rm $CertExpiries;

Add a Rate Limit to Your Website

Suppose you have a ressource on the web (for example an API) that either generates a lot of load, or that is prone to be abused by excessive use, you want to rate-limit it. That is, only a certain number of requests is allowed per time-period.

A possible way to do this is to use Memcache to record the number of requests received per a certain time period.

Task: Only allow 1000 requests per 5 minutes

First attempt:
The naive approach would be to have a key rate-limit-1.2.3.4 (where 1.2.3.4 would be the client's IP address) with a expiration time of 5 minutes (aka 300 seconds) and increment it with every request. But consider this:

10:00: 250 reqs -> value 250
10:02: 500 reqs -> value 750
10:04: 250 reqs -> value 1000
10:06: 100 reqs -> value 1250 -> fails! (though there were only 850 requests in the last 5 minutes)

Whats the problem?

Memcache renews the expiration time with every set.

Second attempt:
Have a new key every 5 minutes: rate-limit-1.2.3.4-${minutes modulo 5}. This circumvents the problem that the key expiration but creates another one:

10:00: 250 reqs -> value 250
10:02: 500 reqs -> value 750
10:04: 250 reqs -> value 1000
10:06: 300 reqs -> value 300 -> doesn't fail! (though there were 1050 requests in the last 5 minutes)

Solution:
Store the value for each minute separately: rate-limit-1.2.3.4-$hour$minute. When checking, query all the keys in the last 5 minutes to calculate the requests in the last 5 minutes.

Sample code:


foreach ($this->getKeys($minutes) as $key) {
    $requests += $this->memcache->get($key);
}

$this->memcache->increment($key, 1);

if ($requests > $allowedRequests) throw new RateExceededException;

For your convenience I have open sourced my code at github: php-ratelimiter.

munin smart plugin: ignore error in the past

As a hard drive in my server failed, my hosting provider exchanged the drive with another one which obviously had some sort of error in its past, but now seems to be fully ok again. I would have wished to receive a drive without any problems but as my server is RAID 1, I can live with that.

I do my monitoring with Munin and for monitoring my hard drives I use the smart plugin. Now this plugin also monitors the exit code of smartctl, where smartctl sets bit no 6 if there was an error in the past, so now while everything is alright, the exit code is always numeric 64.

Now the smart plugin reports this as an error, if the exit code is > 0, i.e. now it always reports a problem.

I could set the threshold to 65, but then I wouldn't be notified of other errors which essentially makes the plugin useless.

I asked at Serverfault but no one seems to have a solution for that.

So I attacked the problem on my own and patched the plugin. In the source code the important line is here:


if exit_status!=None :
# smartctl exit code is a bitmask, check man page.
num_exit_status=int(exit_status/256)

which I have modified to look like this:

if exit_status!=None :
# smartctl exit code is a bitmask, check man page.
num_exit_status=int(exit_status/256)
# filter out bit 6
num_exit_status &= 191
if num_exit_status<=2 : exit_status=None if exit_status!=None :

Now it doesn't bug me anymore when bit 6 is set, but if any other bit goes on again, I will still be notified. The most interesting part is the line where there is a bitwise operation with 191: this is 0x11011111 in binary, so doing an AND operation with the current value it will just set bit no 6 to 0 while letting the other values untouched.

Therefore a value of 64 (as mine does) will be reported as 0 while a value of 8 would remain at 8. But also, very importantly, a value of 72 (bit 6 set as always and bit 3 set because the disk is failing) it would also report 8.

And there we have another reason, why it is good to be firm with knowledge about how bits and bytes behave in a computer. Saved me from a warning message every 5 minutes :-)

preg_match, UTF-8 and whitespace

Just a quick note, be careful when using the whitespace character \s in preg_match when operating with UTF-8 strings.

Suppose you have a string containing a dagger symbol. When you try to strip all whitespace from the string like this, you will end up with an invalid UTF-8 character:

$ php -r 'echo preg_replace("#\s#", "", "?");' | xxd
0000000: e280

(On a side note: xxd displays all bytes in hexadecimal representation. The resulting string here consists of two bytes e2 and 80)

\s stripped away the a0 byte. I was unaware that this character was included in the whitespace list, but actually it represents the non-breaking space.

So actually use the u (PCRE8) modifier as it will be aware of the a0 "belonging" to the dagger:

$ php -r 'echo preg_replace("#\s#u", "", "?");' | xxd
0000000: e280 a0

By the way, trim() doesn't strip non-breaking spaces and can therefore safely be used for UTF-8 strings. (If you still want to trim non-breaking spaces with trim, read this comment on PHP.net)

Finally here you can see the ASCII characters matched by \s when using the u modifier.

$ php -r '$i = 0; while (++$i < 256) echo preg_replace("#[^\s]#", "", chr($i));' | xxd 0000000: 090a 0c0d 2085 a0 $ php -r '$i = 0; while (++$i < 256) echo preg_replace("#[^\s]#u", "", chr($i));' | xxd 0000000: 090a 0c0d 20

Functions operating just on the ASCII characters (with a byte code below 128) are generally safe, as the multi-byte characters of UTF-8 have a leading bit of one (and are therefore above 128).

Restoring single objects in mongodb

Today I had the need to restore single objects from a mongodb installation. mongodb offers two tools for this mongodump and mongorestore, both of which seem to be designed to only dump and restore whole collections.

So I'll demonstrate the workflow just to restore a bunch of objects. Maybe it's a clumsy way, but we'll improve this over time.

So, we have an existing backup, done with mongodump (for example through a daily, over-night backup). This consists of several .bson files, one for each collection.

  1. Restore a whole collection to a new database: mongorestore -d newdb collection.bson
  2. Open this database: mongo newdb
  3. Find the items you want to restore through a query, for example: db.collection.find({"_id": {"$gte": ObjectId("4da4231c747359d16c370000")}});
  4. Back on the command line again, just dump these lines to a new bson file: mongodump -d newdb -c collection -q '{"_id": {"$gte": ObjectId("4da4231c747359d16c370000")}}'
  5. Now you can finally import just those objects into your existing collection: mongorestore -d realdb collection.bson

trac Report for Feature Voting

I use trac for quite a few projects of mine. Recently I tried to find a plugin for deciding which features to implement next. Usually trac hacks has something in store for that, but not this time.

I wanted to be able to create a ticket and then collect user feedback as comments for the feature, with each piece of feedback being a vote for that feature, like this:

After searching for a bit I came up with a solution by using just a report with a nicely constructed SQL query.

SELECT p.value AS __color__,
   t.type AS `type`, id AS ticket, count(tc.ticket) as votes, summary, component, version, milestone,
   t.time AS created,
   changetime AS _changetime, description AS _description,
   reporter AS _reporter
  FROM ticket t, ticket_change tc, enum p
  WHERE t.status <> 'closed'
AND tc.ticket = t.id and tc.field = 'comment' and tc.newvalue like '%#vote%'
AND p.name = t.priority AND p.type = 'priority'
GROUP BY id, summary, component, version, milestone, t.type, owner, t.time,
  changetime, description, reporter, p.value, status
HAVING count(tc.ticket) >= 1
 ORDER BY votes DESC, milestone, t.type, t.time

So just by including "#vote" in a comment, it would count towards the number of votes. You can change this text to anything you want, of course. For example like this:

I hope this can be useful for someone else, too.

iOS 2011 Alarm Clock Bug

Just to add to the speculation about the causes of the 2011 alarm clock bug of iOS where the one-time alarms would not activate on January 1 and January 2, 2011.

My guess is that the code that sets off the alarm takes day, month and year into account when checking whether the alarm should go off. But instead of using the "normal" year, an ISO-8601 year could have been used. This type of year is calculated by the number of the week (with Monday as the first day of the week), thus for the week 52 (from December 27, 2010 to January 2, 2011) the respective year remains 2010.

When setting the date to January 1, 2012, the alarm doesn't go off as well (week 52 of 2011). This adds to my theory and also means that this hasn't been a one-time issue and requires a bug fix by Apple.

Debugging PHP on Mac OS X

Operating system

  • the software component of a computer system that is responsible for the management and coordination of activities and the sharing of the resources of the computer

OS X

  • a line of computer operating systems developed, marketed, and sold by Apple Inc, the latest of which is pre-loaded on all currently shipping Macintosh computers

PHP

  • an open source programming language

I have been using Mac OS X as my primary operating system for a few years now, and only today I have found a very neat way to debug PHP code, like it is common for application code (i.e. stepping through code for debugging purposes).

The solution is a combination of Xdebug and MacGDBp.

macgdbp-debugger

I am using the PHP package by Marc Liyanage almost ever since I have been working on OS X, because it's far more flexible than the PHP shipped with OS X.

Unfortunately, installing Xdebug the usual pecl install xdebug doesn't work. But on the internetz you can find a solution to this problem.

Basically you need to download the source tarball and use the magic command CFLAGS='-arch x86_64' ./configure --enable-xdebug for configuring it. (The same works for installing APC by the way)


/usr/local/php5/php.d $ cat 50-extension-xdebug.ini
[xdebug]
zend_extension=/usr/local/php5/lib/php/extensions/no-debug-non-zts-20060613/xdebug.so

xdebug.remote_autostart=on
xdebug.remote_enable=on
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=localhost
xdebug.remote_port=9000

Now you can use MacGDBp. There is an article on Particletree that describes the interface in a little more detail.

I really enjoy using this method to only fire up this external program, when I want to debug some PHP code, and can continue to use my small editor, so that I don't have to switch to a huge IDE to accomplish the same.

Posted in php

Website Optimization, a book by Andrew B. King

Website Optimization

This time I'm reviewing a book by Andy King. Unlike High Performance website by Steve Souders, it doesn't solely focus on the speed side of optimization, but it adds the art of Search Engine Optimization to form a compelling mix in a single book.

If you have a website that underperforms your expectations, this single book can be your one-stop shop to get all the knowledge you need.

Andy uses interesting examples of how he succeeded in improving his clients' pages that illustrate well what he describes in theory before. He not only focuses on how to make your website show up at high ranks in search engines (what he calls "natural SEO"), but also discusses in detail how to use pay per click (PPC) ads to drive even more people to one's site. I especially liked how Andy describes how to find the best keywords to pick and also describes how to monitor success of PPC.

The part about the optimization for speed feels a little too separated in the book. It is a good read and provides similar content as Steve Souders book, though the level of detail feels a little awkward considering how different the audience for the SEO part of the book is. Still, programmers can easily get deep knowledge about how to get that page load fast.

Unfortunately Andy missed out a little on bringing this all into the grand picture. Why would I want to follow not only SEO but also optimize the speed of the page? There is a chapter meant to "bridge" the topics, but it turns out to be about how to properly do statistics and use the correct metrics. Important, but not enough to really connect the topics (and actually I would have expected this bridging beforehand).

Altogether I would have structured things a little different. For example: It's the content that makes search engines find the page and makes people return to a page, yet Andy explains how to pick the right keywords for the content first whereas he tells the reader how to create it only afterwards.
Everything is there, I had just hoped for a different organization of things.

All in all, the book really deserves the broad title "Website Optimization." Other books leave out SEO which usually is the thing that people mean when they want to optimize their websites (or have them optimized).

I really liked that the topics are combined a book and I highly recommend the book for everyone who wants to get his or her website in shape.

The book has been published by O'Reilly in July 2008, ISBN 9780596515089. Also take a look at the Website Optimization Secrets companion site.

Thanks to Andy for providing me a review copy of this book.

Upgrade WordPress Script

Whenever a new version of WordPress comes out (as just WordPress 2.6 did), it is somewhat of a pain to upgrade it.

But not for me anymore, because I have created a small (and simple) script some versions ago which I would like to share with you.


$ cat upgrade_wordpress.sh
wget http://www.wordpress.org/latest.tar.gz
mv www wordpress
tar --overwrite -xzf latest.tar.gz
rm latest.tar.gz
mv wordpress www

www is my document root and the script sits outside of it. It downloads the most recent version, extracts it while overwriting the already existing files. The script doesn't contain anything extra-ordinary, but makes upgrading real easy.

Of course this script is only useful if you have ssh access to your web server, but if you do that script might ease the (almost too frequent) pain of upgrading WordPress.

bash completion for the pear command

I am only scratching my own itch here, but maybe someone can use it or expand from it.

I just always found annoying that pear run-tests tab gives all files instead of just *.phpt. This is what this snippet actually does.

Paste this into the file /opt/local/etc/bash_completion on OSX (for me it is just before _filedir_xspec()) or into a new file /etc/bash_completion.d/pear on Debian.


# pear completion
#
have pear &&
{
_pear()
{
local cur prev commands options command

COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}

commands='build bundle channel-add channel-alias channel-delete channel-discover channel-info channel-update clear-cache config-create config-get config-help config-set config-show convert cvsdiff cvstag download download-all info install list list-all list-channels list-files list-upgrades login logout makerpm package package-dependencies package-validate pickle remote-info remote-list run-scripts run-tests search shell-test sign uninstall update-channels upgrade upgrade-all'

if [[ $COMP_CWORD -eq 1 ]] ; then
if [[ "$cur" == -* ]]; then
COMPREPLY=( $( compgen -W '-V' -- $cur ) )
else
COMPREPLY=( $( compgen -W "$commands" -- $cur ) )
fi
else

command=${COMP_WORDS[1]}

case $command in
run-tests)
_filedir 'phpt'
;;
esac
fi

return 0
}
complete -F _pear $default pear
}

Then re-source your bashrc or logout and re-login.

I am far from being an expert in bash_completion programming, so I hope someone can go on from here (or maybe has something more complete lying around?).

High Performance Web Sites, a book by Steve Souders

I'd like to introduce you to this great book by Steve Souders. There already have been several reports on the Internet about it, for example on the Yahoo Developers Blog. There is also a video of Steve Souders talking about the book.

The book is structured into 14 rules, which, when applied properly, can vastly improve the speed of a web site or web application.

Alongside with the book he also introduced YSlow, an extension for the Firefox extension FireBug. YSlow helps the developer to see how good his site complies with the rules Steve has set up.

I had the honour to do the technical review on this book, and I love it. Apart from some standard techniques (for example employing HTTP headers like Expires or Last-Modified/Etag), Steve certainly has some tricks up his sleave:

For instance he shows how it is possible to reduce the number of HTTP requests (by inlining the script sources) for first time visitors, while still filling up their cache for their next page load (see page 59ff).

The small down side of this book is that some rules need to be taken with care when applied to smaller environments; for example, it does not make sense (from a cost-benefit perspective) for everyone to employ a CDN. A book just can't be perfect for all readers.

If you are interested in web site performance and have a developer background, then buy this book (or read it online). It is certainly something for you.

The book has been published by O'Reilly in September 2007, ISBN 9780596529307.

Some more links on the topic:

,