Old News but still true as ever: Facebook and Instagram are harming the Open Web

A big motivation for building the Friends Plugin is to be able to follow what my friends and family are up to. Probably the most important “follow technology” is good old RSS.

While mostly used for public posts, there are many examples of how it can deliver posts not meant for the public: the solution are individual RSS feeds per subscriber, for example paid podcasts, or like the Friends plugin gives specific feeds to friends that can contain private posts.

Now reality is that (no longer) every platform has RSS feeds. It’d be easier if they just provided themt but alas. There are plugins for the Friends plugin itself that make use of projects that have built custom parsers for specific services: Fraidyscrape and RSS Bridge.

Now unfortunately, a lot of friends and family post to Facebook or Instagram. It’s just impossible to follow these friends outside of the platforms.

  • They don’t provide (authenticated) RSS feeds. That would solve all of this.
  • They rate limit and quickly block IPs (especially Instagram) that try to read a few posts unauthenticated.
  • It’s unintuitive, developer centric and needs approval to get an API key, so it’s not something every user of the plugin can be asked.
  • Their HTML is scrambled with random class names, preventing any classic scraping.

So in the end, the content of my friends and family is locked in. It would be a solution for them to create their own blog (IndieWeb style) and share their content that way. But I am realistic enough to understand that the hurdles are great. They are already connected to their friends inside Facebook or Instagram. They get lots of likes and responses there. They are used to the UI and the apps. It’s just not good for anyone who doesn’t want to join the Facebook ecosystem and be exposed to their ads.

We’re all using RSS Readers all the time…

… we just don’t realize it. If you think about it, most of the social network services are specialized RSS Readers that combine consuming with creating. (Just to be clear, these social networks don’t provide RSS feeds, what I’m saying is that what they do could be done with RSS feeds.)

Take Facebook, for example:

  • If you follow someone, you’re subscribing to their RSS feed.
  • If you like a page, you’re subscribing to their RSS feed.
  • If you become friends someone, you’re subscribing to their RSS feed and vice versa.
  • If you view your timeline, you’re looking at all posts by your subscriptions (usually modified/filtered by Facebook).
  • If you post something, it’s added to your RSS feed and the people who are subscribed to you (via follow or friendship) get your post in their feed.

Twitter is similar, they just limit the post format to “status”. Instagram limits the post format to “image”. Tiktok to “video”.

The problem that I see with this is that this happens in a centralized manner. With RSS readers, we already have the decentralized version of this technology. The benefit of social networks is that they solve a few problems well:

  • Easily reachable and searchable user base (= audience).
  • Easy account creation (just sign up here).
  • They combine consuming with creation.

I see a possible solution in WordPress. There are many options to get WordPress hosting with your own domain. With the Friends Plugin you can then combine consuming and creating. The Friends plugin allows you to consume the web your way, WordPress is a well established way of creating content.

By using post formats, you can also create different types of content and consume the same kind of incoming content combined. See the screenshots above, showing the main, combined feed, an image feed and a status feed.

Also, commenting on posts is built in to WordPress. With a friendship based approach you can even open up comments just to friends, which can reduce the usual “spam problem.”

What’s still missing for is the convenience of single-purpose apps. The cool thing is that these apps would talk to your own server which handles retrieving of followed content vs. this being relayed to a third party (who then knows the follow patterns of everyone on their platform). With Sunlit, micro.blog has implemented something like this for photo sharing inside their micro.blog platform but I believe this could be done for “RSS” as well.

The Indieweb movement follows very similar ideas to what I described above. The concept of webmentions allows you to respond to a post by posting to your own server, sending a ping to the original article. While the WordPress + Friends approach might be a bit more integrated, I see webmentions as a reasonable bridge between the worlds.

All in all, I believe having a personal blog is good for the health of internet. It’s unfortunate that in their history, personal blogs have this huge focus on “being public.” I believe it is equally well suited to be a place for more private communication, and especially a place where you can consume the web the way you like.

Using Jetpack Blocks without a connection to WordPress.com

Recently I wanted to use a Jetpack block for the WordPress Gutenberg editor, specifically the Slideshow Block, which I think is very slim and elegant compared to other Slideshow blocks on the WordPress.org plugin directory.

The one thing I didn’t want to do, though, is to connect my site to WordPress.com. For the most part because I didn’t intend to use any of the functionality that requires the WordPress.com connection.

The difficulty in this: normally, you can’t actually use Jetpack until you connect the site to your WordPress.com account. This behavior is in place so you can enjoy a friction-less experience after the connection is established (i.e. you can activate any of its features whether or not it needs the connection).

In my case, the connection is not needed for using the Slideshow block since it works with the local media library.

So, I present not a hack but an official way to use Jetpack without a WordPress.com connection: it’s called Offline Mode.

You activate it by adding a constant or filter to your wp-config.php, for example:

define( 'JETPACK_DEV_DEBUG', true );

While it is intended for local installs and actually development, it is useful if you just want to use a single feature of Jetpack that doesn’t require the connection.

Many people say that Jetpack is slow because it’s so big. It is indeed huge from a file size and feature perspective, but also it’s very optimized, so that it will only load the PHP or JS code it needs. The other files lay dormant in your file system and don’t contribute to the loading or rendering time of your site.

Posted in web

Reducing Our Dependency On Third-Party Platforms For Our Online Activity

There is something that I feel is not right with today’s web structure. We, as the population of the web, create so much content that ends up on servers of large companies. We could own our data.

Therefore, I’d like to help reduce everyone’s dependency on third party platforms for their online activity.

Vision

You use your own server for publishing and interact with other people’s content on their server.

Since everybody follows this method, your content only reaches the people it is meant for, without involving a third party in the distribution.

You interact using the same means of today: web UI, mobile apps; the difference is that they don’t talk to a third-party server, neither for publishing nor getting and interacting with others’ content.

I am trying to create a solution for this with WordPress and the Friends plugin.

Note: in the following text I use the word “server” somewhat synonymous for the WordPress+Friends setup but it there could be alternative implementations. Also, server doesn’t mean “a dedicated machine somewhere,” you just need web space where you can install WordPress, which doesn’t mean it has to be expensive. For example, wpfriends.at is hosted for €1,90/month incl. domain.

Separation of Content

Today, separate social networks exist for different types of content; for example short content has its home on Twitter, Instagram is for photos. Facebook is a mix of this but allows for private content. Instant messengers like WhatsApp (or Facebook groups) are very clear about who the content or conversation is for.

So, separating content makes for a more homogeneous experience when consuming the content, also you have a good grasp of where (and for whom) you want to publish your content.

On your own server, you can separate out different types of content as well (in WordPress this is called “post formats”). You can also post something privately, only giving access to authenticated users.

Adding a friend and subscribing to multiple platforms on which they publish

The key is to connect the different types appropriately. By fetching the content from your friends and placing them in such “buckets,” you now have the option to view everyone’s content from multiple perspectives: everything from a single friend (or a group) across content types, or everything across friends of just one content type.

By continuing to use specialized apps for different types of content, you keep the benefits but exclude the third-party having full access to the content since they now talk to your own server.

Platforms

The possibility to publish on the Internet by using your own server (most of the time: rented web space) is well-established since blogging was invented. It can become clunky when you interact with others: how do I respond?

That’s why there are several reasons why platforms are appealing:

  • It’s easy to connect with others on the same platform.
  • Leveraging network effects is easier since the platform knows who is connected to whom.
  • Spam is usually under control.

Thinking back to the era of blogging, we had lots of interactions with others using comments and pingbacks. What mainly led to its demise was automated spam which reduced everybody’s willingness to be open for interaction. Commenting meant moderating spam.

Core

So one of the core features of the Friends plugin is to solve the authentication problem between people you know and trust.

When you decide “to become friends” with someone, it means that you both get (low privilege) accounts on each other’s servers.

You can give them permission to see your privately published content, or not. When you want to respond to their post, you are automatically authenticated on their side.

This means that you could close comments for unauthenticated users (thus eliminating spam) but keep the discussion open for your friends.

Transitioning

Using your own server still has a learning curve. Getting started with a self-hosted WordPress and installing the Friends plugin is not as trivial as creating an account on a social network.

While it has become considerably easier to register a domain, get webspace connected, and a WordPress installed, there are still many further steps until you really can get started.

So, reality is that your friends likely won’t be migrating off third-party networks. Possibly never.

To still allow yourself to disconnect, the Friends plugin allows you to follow your friends across different (possibly third-party) channels including popular social networks like Twitter.

You can subscribe to someone on Twitter and their messages will be aggregated on your server under their user, in the respective post format.

This means that you can either view all the content (that you follow) for a single friend, or you can view the aggregate type of content, e.g. all the short messages your friends posted across services.

Not only for “real” Friends

Many use social networks to not only follow people they know and trust, but also to follow the news or celebrities. Despite the name of the Friends plugin, it can also do that.

You can subscribe any supported content (some out of the box, for others you can extend it with plugins) but not go into the friendship realm.

Your server takes care of fetching the content from various third-party sources and you can then consume the content in the way you see fit:

  • The local “Friends UI” on your own server,
  • an RSS reader,
  • or more specialized clients like mobile apps.

Ads

One side-effect of this could possibly be that this removes our content from being wrapped in ads and websites trying to have us spend as much time as possible on their site.

We’d all spend a little money for domain and server every month or year for our own gain. We don’t need to put ads to our content.

For journalistic content it is already not uncommon for tech publications to provide an ad-free RSS feed as part of their paid subscription.

More Ideas

You can use your server to store more things, for example:

Your personal bookmark collection and todo lists (I have a work-in-progress for transitioning my previous project called thinkery.me to a WordPress plugin thinkery to host this yourself and use the Android app as a client).

Leverage your aggregated content as your own browser start page (my WordPress plugin called Startpage).

Right now, this is only implemented for WordPress, and the authentication is only leveraged for consuming private content and posting comments. But it doesn’t need to end there.

The authentication could be used for further actions, for example you could give posting permissions to your friends to create a (private) forum, hosted by you or a friend.

The Friendship protocol is a REST API that can be implemented in other software as well.

The Friends plugin is open source and GPL.

Work in Progress

How far along the way is the Friends plugin? It’s already well usuable for your own purposes. Like I said above, you can establish friendships between servers and consume your friends’ content, even from third-party social networks.

There is still a lot to do, especially around commenting and notifications. Better tools are needed for leveraging network effects, like search your friends+ posts and explore their friends (if they allow you to do so).

Posted in web

Pocketbook Color

I read a lot of web articles on my e-reader (often using Push to Kindle which is fantastic). I left the Kindle ecosystem a while ago and Pocketbook (a TouchHD 3) has been a good home so far.

Since my content is often a mix of text and non-text, I was appealed by a color eInk screen. The Pocketbook Color recently came out and I purchased one to test it.

Here are my conclusions after about two weeks of usage:

Pro

  • While limited in their range, colors work well for diagrams, illustrations and screenshots. Photos can look a bit awkward but it’s definitely better than greyscale.
  • Before this e-reader, I haven’t read comics on a device but with color it’s quite fun. The clipping tools in the UI are useful to get rid of white borders around the actual comics.

Con

  • It’s not very suitable for night time reading (which is my main use case), the minimum brightness is high, there is only white backlight.
  • With the strong white backlight it feels like an LCD screen which kind of diminishes the idea behind eInk.
  • The technology seems to use two layers: one “common” 300dpi greyscale, and one around 150dpi color layer. This results in a pattern overlaying the whole screen making the screen less crips when reading text.
  • Pocketbook readers are rather slow. I always wonder how they do scrolling and inertia so much better than Kindles but feel so. slow. navigating through the UI.

Overall, I think color eInk is technology well worth exploring, especially when no backlight is necessary and it doesn’t need any battery power (think photo walls)

I am torn whether I’ll keep the Pocketbook Color as my main reading device because using the reader in the dark is like turning on the light in the room.

Posted in web

Decentralized Social Networking with WordPress

Over the past year, I’ve been working on the side on a WordPress plugin that implements an idea that has been growing in me over the last couple of years. Decentralized Social Networking. The plugin that does it is called Friends.

Starting with the frustration that there are few alternatives for people who use Facebook: if you don’t want them to own your data but still want to privately keep your friends and family up to date about your live and discuss what interests you, where do you go?

I realize that many people just switched to instant messaging (like WhatApp) which does allow exchanging private messages (and photos) with your friends, but overall I do like the idea of having a more structured publishing platform. I just don’t like a single entity to control it all.

So I realized: We actually had an alternative all along: blogging.

Blogging is decentralized: you decide where you host, you decide which blogs you read and nobody really knows which ones you have subscribed to.

What disqualifies it as an alternative to my wish to keep friends and family up to date, is that it public by default. While there is the option to publish something as “private,” there is not a lot you can then do with a privately published post.

So what if you were easily able to give your friends access to your private posts?

Here comes the Friends plugin. And with it, the downside to the solution:

You need to have your own blog to become friends with each other.

Right now, this is only implemented on WordPress, the technology is framework agnostic, though.

Your friends get their own user on your blog. Here my request is still pending.

When everyone involved has their own blogging platform and they’d decide where they want to host, we automatically get a decentralized platform.

The technological ingredients to this are actually pretty old:

  • RSS
  • REST API
  • Authentication via keys

After friendship has been established (this involves both parties accepting friendship and exchange private keys in the background), your server will use that key when requesting your friend’s RSS feed which will in turn (since you are friends) contain private posts. And vice versa.

For commenting, you’ll go to your friend’s blog, it’s a one-click-authentication away. This can also eliminate spam if you only allow friends to comment on your posts.

All of this is highly compatible with standard WordPress: if you want to accept a friend request on your mobile phone, use the WordPress iOS or Android app and change that user’s role to “Friend.”

Chicken-and-egg problem: I use a social network because all my friends are there.

So why use a social network that not all of your friends use (yet)? Because the Friends plugin is actually a pretty decent way to consume RSS feeds.

Since it is based on RSS, it means you can also subscribe any blog or website that offers this well-established way of distributing content.

As you are in control of the server, you can also decide what you’re interested in and tailor the feeds to your liking: you can define rules for incoming feed items and ignore posts you know you won’t be interested in.

I personally like to consume notifications via e-mail as it provides read/unread functionality and I can sort and categorize e-mails to my needs. You can now also read your friends’ posts (or subscriptions) via e-mail. But you don’t need to.

Another method to view your friend posts and subscriptions is the “Friends Page,” a timeline of friend posts and subscriptions. You can therefore scroll down the list of your friend’s posts and subcriptions, just as you see it people doing it all the time on Facebook.

Read your friends’ posts on the Friends Page

There is quite a bit more to this, you can Emoji react to a post, recommend it specifically to friends, have sections in your posts for friends/not-friends, and more.

Overall, I see this as a way to take blogging to the next level: Choose your private audience. Choose where you host. Publish publically if it’s meant to be public.

It’s clear that setting up and having your own blog is not (yet) for everyone. It’s more work than just signing up for some social network. Often you’ll need to pay for hosting (and domain). But it also gives you the freedom to take your data somewhere else if you want to. Or delete it.

We’ll have to see who will use this. As of now, it’s for a technical audience but maybe someday there will be dedicated Friends-WordPress hosting?

The Friends Plugin is open source under GPL2. If you have a WordPress blog, try it out, and if you think this could be better, different, enhanced: create an issue, or better: create a patch and send a pull request.

Probably it’s not ready for prime time yet, we’re at version 0.14. But it’s getting there, at least I am already using it daily :) There are many ideas left to be implmented, and these are only mine so far.

Oh, if you happen to be in Vienna coming week, I’m going to talk about Decentralized Social Networking with WordPress at our WordPress meetup on November 7, 2018.

November WordPress Meetup Vienna

Wednesday, Nov 7, 2018, 6:30 PM

CodeFactory Vienna
Kettenbrückengasse 23/2/12 Wien, AT

29 People Attending

Schedule 18:30 | Arrival, Registration 19:00 | Welcome & Introduction 19:15 | Alex Kirk – Decentralized Social Networking with WordPress 19:45 | Break 20:00 | Harry Martin – Hello to 5.0, a first look at the next major release and the new theme Twenty Nineteen 20:30 | Socialising! 21:00 | Leaving CodeFactory, maybe for drinks somewhere close by __…

Check out this Meetup →

Posted in web

Fixing WhatsApp image dates after Android Migration

Recently I've had the issue to have a completely unsorted Photo Library in Android after migrating to a new phone. The reason is that WhatsApp images are copied into internal storage and end up with the last modification date when they were copied, thus conglomerating together when they should be spread out over time.

The problem is not that trivially to solve because you cannot mount internal storage into a computer and then modify the file dates. Thankfully, I was still able to create a viable solution using a bash script.

It all revolves around the Android Terminal Emulator Termux which allows you to execute scripts on your phone.

  1. Install Termux .
  2. Grant Termux access to your storage directories.
  3. Install core tools apt install coreutils
  4. Copy this script into your file tree (for example via Android File Transfer):
for f in IMG-20* VID-20* AUD-20*; do
    [ -e "$f" ] || continue
    NEWDATE=`echo $f | cut -c5-8`-`echo $f | cut -c9-10`-`echo $f | cut -c10-11`
    echo touch -d $NEWDATE "$f"
    touch -d $NEWDATE "$f"
done
  1. Run the script in the directories you need to fix file dates (for example in /storage/emulated/0/WhatsApp/Media/WhatsApp Images)
  2. Delete the data of MediaStorage (and maybe reboot) to make Android re-index the files with the new file dates.
Posted in web

Buchhaltung für EPUs: Bookamat

Eine Sache, die man als EPU nicht gerne macht (zumindest alle, die ich kenne), ist die Buchhaltung. Auch wenn man den ganzen “Rechnungskram” an einen Buchhalter auslagern könnte, so finde ich es für den alltäglichen Betrieb wichtig, die Finanzen im Blick zu haben und die Erfassung der Belege selbst zu übernehmen.

Eigentlich könnte man seine ganzen Einnahmen und Ausgaben auch in einer Excel-Tabelle erfassen. Aber das ist in vielerlei Hinsicht mühsam und unübersichtlich. Also soll eine Software her. Der Markt ist nicht klein, es gibt zahlreiche Software, die Kleinunternehmen damit unter die Arme greifen wollen, aber da gibt es gleich die ersten Fallstricke:

1. Die Software soll für Österreich geeignet sein
2. Das Ganze soll bitte auch am Mac laufen

Die einfache Lösung: eine Web Applikation. Auf dem österreichischen Markt gibt es hier 2 große Player: freefinance und bookamat

bookamat

Mit freefinance sind vielleicht Leute vom Fach zufrieden, für einen Halblaien ist das Interface eine (visuelle) Katastrophe, die ganze Applikation ist voll mit Fachbegriffen frisch aus dem Finanzamt.

freefinance interface

Dagegen ist Bookamat eine Offenbarung. Beim Interface sieht man, dass das jemand macht, der etwas von Webseiten versteht: Patrick Kranzlmüller und Axel Swoboda sind mit ihrer Agentur vonautomatisch vom Fach.

bookamat-neue-buchung

Ich verwende schon mehrere Jahre Bookamat und bei jedem Update sieht man, dass die beiden ihre Software selbst verwenden und immer dort schrauben, wo es noch etwas hakt. Kleines Beispiel aus dem Screenshot oben: Man legt mit der Auswahl des Steuerkontos fest, ob es sich um eine Eingabe oder Ausgabe handelt.

Bei freefinance z.B. muss man erst das richtige Formular aufrufen (also Einnahme oder Ausgabe), und bekommt dann das hier präsentiert:

freefinance-neue-buchung


Die Alarmglocken schrillen schon, wenn man die Hälfte des Bildschirms für Hilfetexte braucht. Bei den Zahlen im Dropdown vermute ich, dass es sich um die Kennzahlen des Finanzamts handelt. Das mag hilfreich sein, aber zeigt wie sehr man sich an der Finanz und nicht am Unternehmer orientiert.

bookamat-steuerkonten

Bookamat hingegen zeigt es auch an, aber dort, wo man sich aussucht, welche Konten man im Alltag überhaupt braucht.

Letztlich bleibt mir nur zu sagen: verwendet Bookamat! Es nimmt der Buchhaltung zwar nicht vollkommen den Schrecken, aber es macht sie verständlicher und erfreulicher. Und das ganze nur für 100 Euro im Jahr, inkl MwSt. (dh für alle die nicht unter die Kleinunternehmerregelung fallen nur 83,33 Euro an).

Little Snitch und Freewave

Inspiriert von der Anleitung von Freewave, die zeigt, wie man in einem unverschlüsselten WLAN (wie Freewave es an vielen Orten in ganz Österreich anbietet) dafür sorgen kann, dass nicht unabsichtlich Apps mit dem Internet sprechen, habe ich hier ein paar Anpassungen zusammengestellt, um das Gleiche mit SSH und SOCKS zu nutzen.

Wenn man Zugang zu einem Server im Internet per SSH hat (eigener Webserver, Uni-Account), kann man sich einfach eine sichere Verbindung (zumindest bis zu ebendiesem Server, aber es geht ja eigentlich nur darum die WLAN-Signale zu verschlüsseln) basteln.

Man startet ein Terminal-Fenster und gibt Folgendes ein:

ssh -D9999 username@mein.server

Mit dem Schalter -D<port> wird ein SOCKS-Proxy gestartet, der auf localhost:<port>, in diesem Fall auf localhost:9999, lauscht. Man lässt dann das Terminal-Fenster sowie die Verbindung so lange offen, solange man den Proxy verwenden will.

Dann konfiguriert man sein OSX so (in den Systemeinstellungen unter “Netzwerk”), dass es den SOCKS-Proxy verwendet:

socks-proxy

und letztlich passt man dann noch die Regeln an, wie sie in der Anleitung von Freewave gezeigt werden:

little-snitch-proxy

Der Unterschied besteht in der Regel für “Terminal”, die pppd und racoon ersetzt. Am einfachsten erstellt man die, indem man die Regel erstmal einfach weg lässt und die “alle ausgehenden”-Regel auf “Nachfragen” stellt. Dann das ssh Kommando ausführen, Little Snitch fragt nach, was es tun soll. Daraufhin erlaubt man die Verbindung für immer und stellt danach die “alle ausgehenden”-Regel wieder auf “Verbieten” zurück. (bei mir gibt es hier 2 Regeln, weil ich je nach Anwendungsfall 2 verschiedene Server verwende)

Übrigens: Ich fand die Bezeichnungen “Ohne VPN” und “Mit VPN” verwirrend, deswegen hab ich die bei mir “Potenziell Unsicher” bzw. “Vertrauenswürdig” genannt.

Insgesamt ist leider schade, dass mit einer solchen Konstruktion die ganzen Regeln, die man mehr oder weniger mühsam unter “Vertrauenswürdig” erstellt hat, nicht gelten, während man “Potenziell Unsicher” aktiviert hat: es geht einfach alles durch SSH, auch was man unter “Vertrauenswürdig” verboten hat.

SSL Certificate Expiry Warning Script

With the increasing trend of SSL on the web, where Google values SSL sites higher and you can have your site be added to the HSTS preload list (the browser will first try HTTPS before trying HTTP), it is a good idea to start using SSL yourself.

The downside: you need to get a certificate through a (pre-trusted by the browser) CA, or certificate authority. This usually costs money, though there are some services that give you a certificate for free. The free certificates only last for one year or less, this means you need to request and install a new certificate frequently, especially when you have multiple domains.

Now it can happen to anyone, even Microsoft (Windows Azure Service Disruption from Expired Certificate), that you forget to renew (and update) your certificate in time.

There is a nice service called certalert.me (interestingly enough not over HTTPS) that will send you an e-mail when a certificate is due to be updated. But as with any web service, unfortunately you can never be sure how long it’s going to live.

So, I have created a script that I run through a cronjob every day that will send me a notification e-mail several times in advance (1 day and 2 7 14 30 60 days ahead), so that you are not dependent on a third party to get notified about expiries. As it is supposed to be with cronjobs, there is no output when there is nothing to report (thus no e-mail).

Here is the script (download warn_about_certificate_expiry.sh):


#!/bin/sh 

CertExpiries=$(mktemp)
for i in /etc/certificates/*.pem; do
	echo $(basename $i): $(openssl x509 -in $i -inform PEM -text -noout -enddate | grep "Not After" | tail -1 | awk '{print $4, $5, $7}') >> $CertExpiries
done

Date=$(date -ud "+1 day" | awk '{print $2, $3, $6}')
Expiries=$(grep "$Date" $CertExpiries)
if [ $? -eq 0 ]; then 
	echo These Certificates expire TOMORROW!
	echo $Expiries
	echo
fi
for i in 2 7 14 30 60; do
	Date=$(date -ud "+$i day" | awk '{print $2, $3, $6}')
	Expiries=$(grep "$Date" $CertExpiries)
	if [ $? -eq 0 ]; then 
		echo These Certificates expire in $i days:
		echo $Expiries
		echo
	fi
done
rm $CertExpiries;

Stack Overflow: Ways out of the negativity

This is in response to the Stack Overflow Meta question: Why is Stack Overflow so negative of late?

In my opinion the problem that Stack Overflow is currently facing is caused by a lot of new users that are characterized by user Mysticial as "help vampires". They care nothing for the site and just want their code fixed. They don’t research (or very little) and provide less than the minimum information needed. Most of the times the questions are very basic and can be answered by an intermediate programmer in a few minutes.

In a normal forum, users would not yield any responses. Not so on Stack Overflow: you get reputation for answering questions and therefore even theses badly researched questions get answers within under a minute. Mystical calls these users "reputation whores".

The problem is that "help vampires" and "reputation whores" create a vicious circle: they both need each other and therefore the circle continues to spin.

The outcome of this situation: the site is flooded with a high number of low quality questions, experienced programmers who are interested in learning something don’t see the forrest for the trees. Even though questions can be voted up, they don’t stand out enough to gain momentum.

Proposed Solutions

a) Create a "beginners test"

This would create a higher burden for low reputation users before they can ask their question. They need to invest more time and rethink their action before they get to post something.

A few ideas what that could be:

  • The user needs to give 3 search queries that he used either on Google or on Stack Overflow that didn’t yield results.
  • If they don’t include any code, they must confirm that they are asking a non-code question. See this proposal on Stack Exchange Meta.
  • Specify the time that they took to research the problem (while this can be easily faked, it makes the user reconsider if they had taken enough time for the problem)

b) Have experienced users review a question, before it goes online

There would be a process where a new user asks his or her question, but it doesn’t go online. Higher reputation users read the question but are unable to answer it, and give feedback if the question has enough information or has been researched enough. Finally, the question get’s thrown into the shark tank.

It would be fine to give these reviewing higher reputation users even more reputation for reviewing this: they are helping to improve the site, this is actually what the reputation system has been designed for: to make the site interesting, not for feeding the "help vampires".

All in all it is remarkable that despite the current situation, Stack Overflow has reached the quality it has. The reputation and badge system has for sure been a very big factor in this but it is very appalling that in order to reach a certain reputation level, you really have to feed the "help vampires".

You can find me on Stack Overflow as akirk.

Posted in web

Fix qTranslate with WordPress 3.9

When updating a blog of mine to WordPress 3.9 the page wouldn’t load anymore because of qTranslate not able to cope with the update. In the error log it says:

#eee; padding: .5em;">PHP Catchable fatal error: Object of class WP_Post could not be converted to string in ../wp-content/plugins/qtranslate/qtranslate_core.php on line 455

The error is caused by this change: get_the_date() to accept optional $post argument

There is a proposed quick fix by Saverio Proto, but it doesn’t take the problem at its root:

qTranslate registers the function qtrans_dateFromPostForCurrentLanguage($old_date, $format ='', $before = '', $after = '') for the hook get_the_date but it actually only accepts one parameter. With the new update it accepts a second parameter $post, which now wrongly fills the variable $before that is in the process being converted to a string.

So the solution simply is to delete the two parameters that were assigned the wrong meaning and have defaults anyway.

Posted in web

Add a Rate Limit to Your Website

Suppose you have a ressource on the web (for example an API) that either generates a lot of load, or that is prone to be abused by excessive use, you want to rate-limit it. That is, only a certain number of requests is allowed per time-period.

A possible way to do this is to use Memcache to record the number of requests received per a certain time period.

Task: Only allow 1000 requests per 5 minutes

First attempt:
The naive approach would be to have a key rate-limit-1.2.3.4 (where 1.2.3.4 would be the client’s IP address) with a expiration time of 5 minutes (aka 300 seconds) and increment it with every request. But consider this:

10:00: 250 reqs -> value 250
10:02: 500 reqs -> value 750
10:04: 250 reqs -> value 1000
10:06: 100 reqs -> value 1250 -> fails! (though there were only 850 requests in the last 5 minutes)

Whats the problem?

Memcache renews the expiration time with every set.

Second attempt:
Have a new key every 5 minutes: rate-limit-1.2.3.4-${minutes modulo 5}. This circumvents the problem that the key expiration but creates another one:

10:00: 250 reqs -> value 250
10:02: 500 reqs -> value 750
10:04: 250 reqs -> value 1000
10:06: 300 reqs -> value 300 -> doesn’t fail! (though there were 1050 requests in the last 5 minutes)

Solution:
Store the value for each minute separately: rate-limit-1.2.3.4-$hour$minute. When checking, query all the keys in the last 5 minutes to calculate the requests in the last 5 minutes.

Sample code:


foreach ($this->getKeys($minutes) as $key) {
    $requests += $this->memcache->get($key);
}

$this->memcache->increment($key, 1);

if ($requests > $allowedRequests) throw new RateExceededException;

For your convenience I have open sourced my code at github: php-ratelimiter.

Thinkery API launched

Just a quick note, we made the Thinkery API public.

If you don’t know thinkery.me, it is a simple yet powerful tool for storing both notes and bookmarks. The contents of the saved page is stored in your Thinkery which you can keep even if the webpage goes down. With #hashtags you can easily categorize everything.

Check it out!

Posted in web

munin smart plugin: ignore error in the past

As a hard drive in my server failed, my hosting provider exchanged the drive with another one which obviously had some sort of error in its past, but now seems to be fully ok again. I would have wished to receive a drive without any problems but as my server is RAID 1, I can live with that.

I do my monitoring with Munin and for monitoring my hard drives I use the smart plugin. Now this plugin also monitors the exit code of smartctl, where smartctl sets bit no 6 if there was an error in the past, so now while everything is alright, the exit code is always numeric 64.

Now the smart plugin reports this as an error, if the exit code is > 0, i.e. now it always reports a problem.

I could set the threshold to 65, but then I wouldn’t be notified of other errors which essentially makes the plugin useless.

I asked at Serverfault but no one seems to have a solution for that.

So I attacked the problem on my own and patched the plugin. In the source code the important line is here:


if exit_status!=None :
# smartctl exit code is a bitmask, check man page.
num_exit_status=int(exit_status/256)

which I have modified to look like this:

if exit_status!=None :
# smartctl exit code is a bitmask, check man page.
num_exit_status=int(exit_status/256)
# filter out bit 6
num_exit_status &= 191
if num_exit_status<=2 : exit_status=None if exit_status!=None :

Now it doesn't bug me anymore when bit 6 is set, but if any other bit goes on again, I will still be notified. The most interesting part is the line where there is a bitwise operation with 191: this is 0x11011111 in binary, so doing an AND operation with the current value it will just set bit no 6 to 0 while letting the other values untouched.

Therefore a value of 64 (as mine does) will be reported as 0 while a value of 8 would remain at 8. But also, very importantly, a value of 72 (bit 6 set as always and bit 3 set because the disk is failing) it would also report 8.

And there we have another reason, why it is good to be firm with knowledge about how bits and bytes behave in a computer. Saved me from a warning message every 5 minutes :-)

Genial Daneben Analyse

Zur Abwechslung mal ein Post auf deutsch. Ich bin Fan der (inzwischen abgesetzten) Fernsehsendung Genial daneben. Es gibt da eine Genial Daneben Datenbank mit (nahezu) allen Fragen, die in den Sendungen vorkamen. Ohne konkreten Nutzen habe ich diese Daten aus der Text-Form in eine echte Datenbank konviertiert (Script hier) und bin zu folgender Tabelle gekommen. Für alle die es interessiert:

Fragen: 2732 (1087 beantwortet, das sind fast 40%)

Teilnehmer Teilnahmen gelöst gelöst pro Teilnahme
Bernhard Hoecker 399 223 0.56
Hella von Sinnen 390 202 0.52
Wigald Boning 120 56 0.47
Guido Cantz 96 51 0.53
Barbara Schöneberger 62 14 0.23
Dieter Nuhr 62 18 0.29
Bastian Pastewka 62 6 0.10
Ralf Schmitz 60 6 0.10
Oliver Kalkofe 55 6 0.11
Herbert Feuerstein 47 25 0.53
Georg Uecker 39 13 0.33
Olli Dittrich 38 11 0.29
Oliver Welke 37 11 0.30
Ingo Appelt 32 8 0.25
Martin Schneider 31 0 0.00
Michael Kessler 29 6 0.21
Thomas Hermanns 27 7 0.26
Anke Engelke 26 8 0.31
Matze Knop 23 2 0.09
Christoph Maria Herbst 20 3 0.15
Mario Barth 20 8 0.40
Jürgen von der Lippe 19 3 0.16
Lou Richter 19 7 0.37
Ingo Oschmann 13 2 0.15
Guildo Horn 13 1 0.08
Dirk Bach 12 3 0.25
Kim Fisher 12 1 0.08
Urban Priol 10 2 0.20
Cordula Stratmann 9 0 0.00
Eckart von Hirschhausen 9 0 0.00
Tetje Mierendorf 8 0 0.00
Oliver Pocher 8 0 0.00
Bodo Bach 8 0 0.00
Hennes Bender 7 1 0.14
Rüdiger Hoffmann 7 0 0.00
Elton 7 1 0.14
Johann Köhnich 7 0 0.00
Bürger Lars Dietrich 7 2 0.29
Helge Schneider 7 7 1.00
Anka Zink 6 0 0.00
Hans Werner Olm 6 0 0.00
Cindy aus Marzahn 6 0 0.00
Kaya Yanar 5 2 0.40
Mike Krüger 5 2 0.40
Horst Lichter 5 2 0.40
Susanne Pätzold 5 1 0.20
Jochen Busse 5 1 0.20
Karl Dall 5 0 0.00
Mirja Regensburg 4 1 0.25
Oli Petszokat 4 2 0.50
Janine Kunze 4 0 0.00
Michael Mittermeier 4 0 0.00
Paul Panzer 3 0 0.00
Florian Schroeder 3 0 0.00
Konrad Stöckel 3 2 0.67
Axel Stein 3 0 0.00
Gayle Tufts 3 1 0.33
Verona Pooth 3 1 0.33
Zack Michalowski 2 0 0.00
Atze Schröder 2 1 0.50
Emily Wood 2 1 0.50
Susanne Fröhlich 2 0 0.00
Gabi Decker 2 0 0.00
Helfried 2 1 0.50
Mirja Boes 2 1 0.50
April Hailer 2 0 0.00
Michael “Bully” Herbig 2 1 0.50
Roberto Cappelluti 2 0 0.00
Olaf Schubert 2 0 0.00
Sissi Perlinger 2 0 0.00
Ottfried Fischer 2 0 0.00
Klaus Eberhartinger 2 0 0.00
Rick Kavanian 2 0 0.00
Lisa Feller 1 0 0.00
Sascha Korf 1 0 0.00
Marc Metzger 1 0 0.00
Joachim Fuchsberger 1 1 1.00
Martin Klempnow 1 0 0.00
Tom Gerhard 1 0 0.00
Ralph Morgenstern 1 1 1.00
Valerie Bolzano 1 0 0.00
Kurt Krömer 1 0 0.00
Hubertus Meyer-Burkhardt 1 0 0.00
Smudo 1 1 1.00
Badesalz (Gerd Knebel und Hendrik Nachtsheim) 1 0 0.00
Markus Maria Profitlich 1 1 1.00
Sven Nagel 1 0 0.00
Bernd Stelter 1 0 0.00
Matthias Matschke 1 0 0.00
Otto Waalkes 1 0 0.00
Till Hoheneder 1 0 0.00
Waldemar Hartmann 1 0 0.00
Lisa Fitz 1 0 0.00
Fatih Cevikkollu 1 0 0.00
Matze Knop (als Franz Beckenbauer) 1 0 0.00
Ralf Morgenstern 1 0 0.00
Ulrike von der Gröben 1 0 0.00

Ich habe den Großteil der Daten nicht überprüft, aber im Groben dürfte es stimmen.

Posted in web

Android WebView: Web page not available

Just a quick note in order to save someone else searching for a solution to this problem.

When you want to display HTML content in an Android WebView do it like this:

String html = "my >b<HTML content>/b<. 100% cool.";
WebView webView = (WebView) findViewById(R.id.myWebView);
webView.loadData(">?xml version=\"1.0\" encoding=\"UTF-8\" ?<" + html.replace("%","%25"), "text/html", "UTF-8");

If you don’t replace the % to its url encoded equivalent you will get a “Web page not available”. Simple, arguable but not at all apparent.

Posted in web

Git tip: Changing your mind: Push pending changes to a (not-yet existing) new branch

It happens quite often to me that I start committing things and only afterwards decide I should have created a new branch.

So git status says something like:

Your branch is ahead of 'origin/master' by 4 commits.


but I don’t want to push to origin/master but rather create a new branch. (Of course this works for any other branch, not named master)

So you can use this sequence of commands:

git checkout -b newbranch
git push origin newbranch
git checkout master
git reset --hard origin/master

Explanation: This…

1. creates a new branch pointing to the current changes (and switches to it)

2. pushes this new branch including the changes to the server

3. switches back to the branch master
4. and undoes the changes that were made locally

The important thing to know is that the changes remain in the repository because a branch is merely a pointer to a commit.

Afterwards you can continue to commit to master, for example:

(screenshots done with a fork of gitx)

Posted in web

Use an authenticated feed in Google Reader

You currently can’t subscribe to an authenticated feed (for example in Basecamp) in Google Reader.

If you want to do it nonetheless you can use this script of mine which will talk to the server that needs authentication, passing through all the headers (so that also cookies and “not modified” requests will come through): download authenticated-feed-passthru.php


<?php
// change this url
$url = "https://username:password@proj.basecamphq.com/projects/123/feed/recent_items_rss";

$ch = curl_init($url);

if (isset($_SERVER['REQUEST_METHOD']) && strtolower($_SERVER['REQUEST_METHOD']) == 'post') {
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
}

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, true);

$headers = array();
foreach ($_SERVER as $name => $value) {
    if (substr($name, 0, 5) != 'HTTP_') continue;
    if ($name == "HTTP_HOST") continue;
    $headers[] = str_replace(' ', '-', ucwords(strtolower(str_replace('_', ' ', substr($name, 5))))) . ": " . $value;
}
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

list($header, $contents) = preg_split('/([\r\n][\r\n])\\1/', curl_exec($ch), 2);
curl_close($ch);

foreach (preg_split('/[\r\n]+/', $header) as $header) {
    header($header);
}

echo $contents;

If you don’t mind giving away your credentials you can also use Free My Feed.

Posted in web