munin smart plugin: ignore error in the past

As a hard drive in my server failed, my hosting provider exchanged the drive with another one which obviously had some sort of error in its past, but now seems to be fully ok again. I would have wished to receive a drive without any problems but as my server is RAID 1, I can live with that.

I do my monitoring with Munin and for monitoring my hard drives I use the smart plugin. Now this plugin also monitors the exit code of smartctl, where smartctl sets bit no 6 if there was an error in the past, so now while everything is alright, the exit code is always numeric 64.

Now the smart plugin reports this as an error, if the exit code is > 0, i.e. now it always reports a problem.

I could set the threshold to 65, but then I wouldn't be notified of other errors which essentially makes the plugin useless.

I asked at Serverfault but no one seems to have a solution for that.

So I attacked the problem on my own and patched the plugin. In the source code the important line is here:


if exit_status!=None :
# smartctl exit code is a bitmask, check man page.
num_exit_status=int(exit_status/256)

which I have modified to look like this:

if exit_status!=None :
# smartctl exit code is a bitmask, check man page.
num_exit_status=int(exit_status/256)
# filter out bit 6
num_exit_status &= 191
if num_exit_status<=2 : exit_status=None if exit_status!=None :

Now it doesn't bug me anymore when bit 6 is set, but if any other bit goes on again, I will still be notified. The most interesting part is the line where there is a bitwise operation with 191: this is 0x11011111 in binary, so doing an AND operation with the current value it will just set bit no 6 to 0 while letting the other values untouched.

Therefore a value of 64 (as mine does) will be reported as 0 while a value of 8 would remain at 8. But also, very importantly, a value of 72 (bit 6 set as always and bit 3 set because the disk is failing) it would also report 8.

And there we have another reason, why it is good to be firm with knowledge about how bits and bytes behave in a computer. Saved me from a warning message every 5 minutes :-)

Genial Daneben Analyse

Zur Abwechslung mal ein Post auf deutsch. Ich bin Fan der (inzwischen abgesetzten) Fernsehsendung Genial daneben. Es gibt da eine Genial Daneben Datenbank mit (nahezu) allen Fragen, die in den Sendungen vorkamen. Ohne konkreten Nutzen habe ich diese Daten aus der Text-Form in eine echte Datenbank konviertiert (Script hier) und bin zu folgender Tabelle gekommen. Für alle die es interessiert:

Fragen: 2732 (1087 beantwortet, das sind fast 40%)

Teilnehmer Teilnahmen gelöst gelöst pro Teilnahme
Bernhard Hoecker 399 223 0.56
Hella von Sinnen 390 202 0.52
Wigald Boning 120 56 0.47
Guido Cantz 96 51 0.53
Barbara Schöneberger 62 14 0.23
Dieter Nuhr 62 18 0.29
Bastian Pastewka 62 6 0.10
Ralf Schmitz 60 6 0.10
Oliver Kalkofe 55 6 0.11
Herbert Feuerstein 47 25 0.53
Georg Uecker 39 13 0.33
Olli Dittrich 38 11 0.29
Oliver Welke 37 11 0.30
Ingo Appelt 32 8 0.25
Martin Schneider 31 0 0.00
Michael Kessler 29 6 0.21
Thomas Hermanns 27 7 0.26
Anke Engelke 26 8 0.31
Matze Knop 23 2 0.09
Christoph Maria Herbst 20 3 0.15
Mario Barth 20 8 0.40
Jürgen von der Lippe 19 3 0.16
Lou Richter 19 7 0.37
Ingo Oschmann 13 2 0.15
Guildo Horn 13 1 0.08
Dirk Bach 12 3 0.25
Kim Fisher 12 1 0.08
Urban Priol 10 2 0.20
Cordula Stratmann 9 0 0.00
Eckart von Hirschhausen 9 0 0.00
Tetje Mierendorf 8 0 0.00
Oliver Pocher 8 0 0.00
Bodo Bach 8 0 0.00
Hennes Bender 7 1 0.14
Rüdiger Hoffmann 7 0 0.00
Elton 7 1 0.14
Johann Köhnich 7 0 0.00
Bürger Lars Dietrich 7 2 0.29
Helge Schneider 7 7 1.00
Anka Zink 6 0 0.00
Hans Werner Olm 6 0 0.00
Cindy aus Marzahn 6 0 0.00
Kaya Yanar 5 2 0.40
Mike Krüger 5 2 0.40
Horst Lichter 5 2 0.40
Susanne Pätzold 5 1 0.20
Jochen Busse 5 1 0.20
Karl Dall 5 0 0.00
Mirja Regensburg 4 1 0.25
Oli Petszokat 4 2 0.50
Janine Kunze 4 0 0.00
Michael Mittermeier 4 0 0.00
Paul Panzer 3 0 0.00
Florian Schroeder 3 0 0.00
Konrad Stöckel 3 2 0.67
Axel Stein 3 0 0.00
Gayle Tufts 3 1 0.33
Verona Pooth 3 1 0.33
Zack Michalowski 2 0 0.00
Atze Schröder 2 1 0.50
Emily Wood 2 1 0.50
Susanne Fröhlich 2 0 0.00
Gabi Decker 2 0 0.00
Helfried 2 1 0.50
Mirja Boes 2 1 0.50
April Hailer 2 0 0.00
Michael "Bully" Herbig 2 1 0.50
Roberto Cappelluti 2 0 0.00
Olaf Schubert 2 0 0.00
Sissi Perlinger 2 0 0.00
Ottfried Fischer 2 0 0.00
Klaus Eberhartinger 2 0 0.00
Rick Kavanian 2 0 0.00
Lisa Feller 1 0 0.00
Sascha Korf 1 0 0.00
Marc Metzger 1 0 0.00
Joachim Fuchsberger 1 1 1.00
Martin Klempnow 1 0 0.00
Tom Gerhard 1 0 0.00
Ralph Morgenstern 1 1 1.00
Valerie Bolzano 1 0 0.00
Kurt Krömer 1 0 0.00
Hubertus Meyer-Burkhardt 1 0 0.00
Smudo 1 1 1.00
Badesalz (Gerd Knebel und Hendrik Nachtsheim) 1 0 0.00
Markus Maria Profitlich 1 1 1.00
Sven Nagel 1 0 0.00
Bernd Stelter 1 0 0.00
Matthias Matschke 1 0 0.00
Otto Waalkes 1 0 0.00
Till Hoheneder 1 0 0.00
Waldemar Hartmann 1 0 0.00
Lisa Fitz 1 0 0.00
Fatih Cevikkollu 1 0 0.00
Matze Knop (als Franz Beckenbauer) 1 0 0.00
Ralf Morgenstern 1 0 0.00
Ulrike von der Gröben 1 0 0.00

Ich habe den Großteil der Daten nicht überprüft, aber im Groben dürfte es stimmen.

Posted in web

Android WebView: Web page not available

Just a quick note in order to save someone else searching for a solution to this problem.

When you want to display HTML content in an Android WebView do it like this:

String html = "my >b<HTML content>/b<. 100% cool.";
WebView webView = (WebView) findViewById(R.id.myWebView);
webView.loadData(">?xml version=\"1.0\" encoding=\"UTF-8\" ?<" + html.replace("%","%25"), "text/html", "UTF-8");

If you don't replace the % to its url encoded equivalent you will get a "Web page not available". Simple, arguable but not at all apparent.

Posted in web

Git tip: Changing your mind: Push pending changes to a (not-yet existing) new branch

It happens quite often to me that I start committing things and only afterwards decide I should have created a new branch.

So git status says something like:

Your branch is ahead of 'origin/master' by 4 commits.


but I don't want to push to origin/master but rather create a new branch. (Of course this works for any other branch, not named master)

So you can use this sequence of commands:

git checkout -b newbranch
git push origin newbranch
git checkout master
git reset --hard origin/master

Explanation: This…

1. creates a new branch pointing to the current changes (and switches to it)

2. pushes this new branch including the changes to the server

3. switches back to the branch master
4. and undoes the changes that were made locally

The important thing to know is that the changes remain in the repository because a branch is merely a pointer to a commit.

Afterwards you can continue to commit to master, for example:

(screenshots done with a fork of gitx)

Posted in web

Use an authenticated feed in Google Reader

You currently can't subscribe to an authenticated feed (for example in Basecamp) in Google Reader.

If you want to do it nonetheless you can use this script of mine which will talk to the server that needs authentication, passing through all the headers (so that also cookies and "not modified" requests will come through): download authenticated-feed-passthru.php


<?php
// change this url
$url = "https://username:password@proj.basecamphq.com/projects/123/feed/recent_items_rss";

$ch = curl_init($url);

if (isset($_SERVER['REQUEST_METHOD']) && strtolower($_SERVER['REQUEST_METHOD']) == 'post') {
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
}

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, true);

$headers = array();
foreach ($_SERVER as $name => $value) {
    if (substr($name, 0, 5) != 'HTTP_') continue;
    if ($name == "HTTP_HOST") continue;
    $headers[] = str_replace(' ', '-', ucwords(strtolower(str_replace('_', ' ', substr($name, 5))))) . ": " . $value;
}
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

list($header, $contents) = preg_split('/([\r\n][\r\n])\\1/', curl_exec($ch), 2);
curl_close($ch);

foreach (preg_split('/[\r\n]+/', $header) as $header) {
    header($header);
}

echo $contents;

If you don't mind giving away your credentials you can also use Free My Feed.

Posted in web

New Feature for HN Collapsible Threads: Collapse Whole Thread

I have added a feature to the HN Collapsible Threads bookmarklet that enables you to close a whole thread from any point within the thread:

This is useful when you are reading a thread and decided that you are having enough of it and want to move on to the next thread. Before you had to scroll all the way up to the top post and collapse that one.

Drag this to your bookmarks bar: collapsible threads

Install Greasemonkey script

Posted in web

preg_match, UTF-8 and whitespace

Just a quick note, be careful when using the whitespace character \s in preg_match when operating with UTF-8 strings.

Suppose you have a string containing a dagger symbol. When you try to strip all whitespace from the string like this, you will end up with an invalid UTF-8 character:

$ php -r 'echo preg_replace("#\s#", "", "?");' | xxd
0000000: e280

(On a side note: xxd displays all bytes in hexadecimal representation. The resulting string here consists of two bytes e2 and 80)

\s stripped away the a0 byte. I was unaware that this character was included in the whitespace list, but actually it represents the non-breaking space.

So actually use the u (PCRE8) modifier as it will be aware of the a0 "belonging" to the dagger:

$ php -r 'echo preg_replace("#\s#u", "", "?");' | xxd
0000000: e280 a0

By the way, trim() doesn't strip non-breaking spaces and can therefore safely be used for UTF-8 strings. (If you still want to trim non-breaking spaces with trim, read this comment on PHP.net)

Finally here you can see the ASCII characters matched by \s when using the u modifier.

$ php -r '$i = 0; while (++$i < 256) echo preg_replace("#[^\s]#", "", chr($i));' | xxd 0000000: 090a 0c0d 2085 a0 $ php -r '$i = 0; while (++$i < 256) echo preg_replace("#[^\s]#u", "", chr($i));' | xxd 0000000: 090a 0c0d 20

Functions operating just on the ASCII characters (with a byte code below 128) are generally safe, as the multi-byte characters of UTF-8 have a leading bit of one (and are therefore above 128).

Restoring single objects in mongodb

Today I had the need to restore single objects from a mongodb installation. mongodb offers two tools for this mongodump and mongorestore, both of which seem to be designed to only dump and restore whole collections.

So I'll demonstrate the workflow just to restore a bunch of objects. Maybe it's a clumsy way, but we'll improve this over time.

So, we have an existing backup, done with mongodump (for example through a daily, over-night backup). This consists of several .bson files, one for each collection.

  1. Restore a whole collection to a new database: mongorestore -d newdb collection.bson
  2. Open this database: mongo newdb
  3. Find the items you want to restore through a query, for example: db.collection.find({"_id": {"$gte": ObjectId("4da4231c747359d16c370000")}});
  4. Back on the command line again, just dump these lines to a new bson file: mongodump -d newdb -c collection -q '{"_id": {"$gte": ObjectId("4da4231c747359d16c370000")}}'
  5. Now you can finally import just those objects into your existing collection: mongorestore -d realdb collection.bson

Safari Extension: Clean URLs

I have been picking up and developing a fork of Grant Heaslip's Safari extension URL clenser which removes all sorts of un-necessary junk for the URL so that you can easily pass on a clean URL to someone else. Things being removed include:

  • Google Analytics parameters (utm_source=, utm_medium, etc.)
  • Youtube related parameters (feature=)
  • Partner tracking stuff for NYTimes, Macword, CNN, CBC Canada and The Star

You can download my version here: url_cleanser.safariextz

Posted in web

trac Report for Feature Voting

I use trac for quite a few projects of mine. Recently I tried to find a plugin for deciding which features to implement next. Usually trac hacks has something in store for that, but not this time.

I wanted to be able to create a ticket and then collect user feedback as comments for the feature, with each piece of feedback being a vote for that feature, like this:

After searching for a bit I came up with a solution by using just a report with a nicely constructed SQL query.

SELECT p.value AS __color__,
   t.type AS `type`, id AS ticket, count(tc.ticket) as votes, summary, component, version, milestone,
   t.time AS created,
   changetime AS _changetime, description AS _description,
   reporter AS _reporter
  FROM ticket t, ticket_change tc, enum p
  WHERE t.status <> 'closed'
AND tc.ticket = t.id and tc.field = 'comment' and tc.newvalue like '%#vote%'
AND p.name = t.priority AND p.type = 'priority'
GROUP BY id, summary, component, version, milestone, t.type, owner, t.time,
  changetime, description, reporter, p.value, status
HAVING count(tc.ticket) >= 1
 ORDER BY votes DESC, milestone, t.type, t.time

So just by including "#vote" in a comment, it would count towards the number of votes. You can change this text to anything you want, of course. For example like this:

I hope this can be useful for someone else, too.

iOS 2011 Alarm Clock Bug

Just to add to the speculation about the causes of the 2011 alarm clock bug of iOS where the one-time alarms would not activate on January 1 and January 2, 2011.

My guess is that the code that sets off the alarm takes day, month and year into account when checking whether the alarm should go off. But instead of using the "normal" year, an ISO-8601 year could have been used. This type of year is calculated by the number of the week (with Monday as the first day of the week), thus for the week 52 (from December 27, 2010 to January 2, 2011) the respective year remains 2010.

When setting the date to January 1, 2012, the alarm doesn't go off as well (week 52 of 2011). This adds to my theory and also means that this hasn't been a one-time issue and requires a bug fix by Apple.