Squid’s HTTP Acceleration Mode

I have recently configured a server of mine to use the Squid Cache in HTTP Acceleration mode. So what’s this anyway?

A typical request to a webserver looks like this: Client browser opens connection to server port 80, server sends back the data through that connection. For the time of the transfer the server “loses” one child process. So if a client with a slow connection requests a large file this can take some minutes. If many slow clients block child processes, eventually too few will be left for “ordinary” clients.

A solution for this is to prepend a proxy server to the HTTP server. The proxy server is lightweight and does the communication with the client browser. The communication with the web server is done via a high speed interface (either loopback when it’s just one server or an lan with 100(0) mbit), so almost no time is spent waiting for a transfer to finish.

Setup is easy, and I’ve covered this in my thesis already.

But I’ve got some more real-life info for you.

There are two usual ways for setting this up.

  1. Set the web server to listen on port 81, Squid on 80.
  2. Web server still listens on port 80 but just for 127.0.0.1, the loopback interface. Squid listens on port 80 on the external interface.

What makes number two the favourable is that you are not haveing a server process listening on an unconventional port, and, for redirects (Location: /somewhereelse) the port number is correct (see the corresponding question in the Squid FAQ). For existing configurations with virtual hosts there is no need to change a < VirtualHost *:80> to < VirtualHost *:81>.

So in ports.conf of Apache, for example, you change this:

# Listen 80
Listen 127.0.0.1:80

In squid.conf you do these changes (apart from those listed in my thesis):

# http_port 3128
http_port ip.add.re.ss:80

So this works nice already, but there is one more thing. Now the source address for a http request is 127.0.0.1. So if you want to do some processing with the REMOTE_ADDR, for example in PHP, you’d have to insert something like this before you’d could use the address again.

if (isset($_SERVER["HTTP_VIA"])) {
// squid http accel
$_SERVER["REMOTE_ADDR"] = $_SERVER["HTTP_X_FORWARDED_FOR"];
}

Also in the log files there is now a 127.0.0.1 as source instead of the real ip address. The following changes things back to normal (in apache2.conf):

# LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" combined

This should be all for now. Happy speed-boosting ;)

Favicon caching

Hello there,

You may have noticed that your blummy now opens a bit faster. This is due to a new feature:

  • [Feature] Favicons are cached on the blummy server.

So now your blummy will not hesitate to open because some server is not ready to serve his icon.

I hope this causes some speed-ups ;)

PS: In the first few hours there could be some hick-ups with caching, as some servers might be down while filling up the cache.

Using a Feedback Form for Spam

Have you ever received weird spam via the feedback form of your site? Something with your own address as sender or with some Mime stuff in the body? Your form is likely to be misused for spamming.

How does it work?

For PHP, for example, there is the mail function that can be used to easily send an e-mail. Most probably you’d use some code like this to send the message from your feedback form.

< ?php $msg .= "Name: " . $_POST["name"] . "\n"; $msg .= "E-Mail: " . $_POST["email"] . "\n"; $msg .= $_POST["msg"]; mail("my.e.mail@addr.es", "feedback from my site", $msg); ?>

That’s simple and works well, but it’s a little annoying if you want to answer that e-mail. You click the e-mail address to open a new message and have to paste the whole message into the new window for quoting. There’s an easy solution: Pretend that the e-mail comes from the customer requesting some info. This can be simply done via the additional_headers parameter of mail.

< ?php $sender = "From: " . $_POST["email"] . "\r\n"; $sender = "From: " . $_POST["name"] . " <" . $_POST["email"] . ">\r\n"; // even nicer, shows the name, not the address
mail("my.e.mail@addr.es", "feedback from my site", $msg, $sender);
?>

Well. We’ve just introduced 2 potential spamming opportunities. Why? Let’s see. For mail transport we use SMTP. Our outgoing mail might look like this (generated by mail).

From: tester < test@test.com>
To: my.e.mail@addr.es
Subject: feedback from my site

this is my message

(Before, the From would have looked something like From: webserver@mydomain.com)
So if the spammer manages to insert another field (like To, CC, or BCC), not only we would receive that e-mail but also the guy entered as CC. This works by inserting a line break into the name or e-mail address. For example, for a given name such as

Alex
CC: other@e.mail.addr.es

that would be the case.
Although this is usually not possible through a normal textbox () a post request can easily be constructed containing that linebreak and the malicious CC.

So be sure to strip out at least the characters \r and \n from name or e-mail address or just strip out any non-latin characters (people with german umlauts in their names, for example, will have to live with that).

So a quite good method would be to use this piece of code:

$name = preg_replace("|[^a-z0-9 \-.,]|i", "", $_POST["name"]);
$email = preg_replace("|[^a-z0-9@.]|i", "", $_POST["email"]);
$sender = "From: " . $name . " < " . $email . ">\r\n";
mail("my.e.mail@addr.es", "feedback from my site", $msg, $sender);

The conclusion is simple (and always the same one): Never trust any data you receive from a user.
Verify all data you receive and strip potentially harmful characters. Common bad characters are:

  • for mails: \r, \n,
  • for HTML: < , > (you could use htmlspecialchars for that),
  • for URLs: &, =,
  • complete the list in the comments ;)

Ah, the conclusion. Never trust any data you receive from a user.

spam, form, e-mail, smtp

Moderation in place

The latest update to blummy covers moderation.
I have cleaned up the blummlets so that there should not be too many blummlets doing the same thing left. While cleaning up I had to merge a few blummlets; if everything went okay you wouldn’t have noticed anything, if not, a blummlet might have disappeared from your blummy. Sorry for that.

  • [GUI] A check next to a blummlet means that it has been verified by a moderator.
  • [Feature] In preferences you can check whether you want to see unverified blummlets, too (default is no)
  • [Feature] You can now choose to open a blummlet in a new Window or Tab (Preferences under advanced). This is actually a not seen feature for bookmarklets. It is established by using the Blummy library function Blummy.href() which resembles location.href = url as Blummy.href(url).
  • [GUI] Cleaned up the preferences window a bit.

With cleaning up I moved almost each blummlet using location.href to use Blummy.href so that for most of them the “open in tab” feature should work. Additionally I have converted all the document.selection/window.getSelection etc. to Blummy.getSelection so that using the selection should work in almost any browser now.

blummy, changes

Now with Preferences

So the latest release is online. The additions this time:

  • [GUI] There is now a simple and advanced mode.
  • [Feature] A Preferences page is now available.
  • [App] You can now change blummy’s opacity in the Preferences.
  • [App] You can now change the close button position in blummy in the Preferences.
  • [Feature] There is now a Save button if XHR does not work for you in configuration.
  • [GUI] A small Flash movie on the Start Page shows how blummy works.

blummy forum

I have now setup a forum for blummy.
It is integrated with the blummy authentication system so that you don’t have to register once again.

blummy, forum

del.icio.us Bookmarks

I have created a new blummlet called del.icio.us Bookmarks.
It’s just another demonstration of what is possible with blummy but it might be useful, too ;)

Using it you have access to your latest bookmarks on del.icio.us from within blummy. You can also specify a certain tag so that only bookmarks having that tag will be displayed. Give it a try!

blummy, del.icio.us, bookmark