Introduction

I was recently requested to document common apache rewrite pitfalls and examples and crafted the following document as a response.  It is intended as a two page document (Rewrite Cheat Sheet) where the first page is a reference guide of commonly used rewrite variables and flags and the second page is a short list of examples, gotchas, and troubleshooting advice.

Rewrite Cheat Sheet

Common Variables

HTTP Headers

• HTTP_USER_AGENT
• HTTP_REFERER
• HTTP_COOKIE
• HTTP_FORWARDED
• HTTP_HOST
• HTTP_PROXY_CONNECTION
• HTTP_ACCEPT

connection & request

• REMOTE_ADDR
• REMOTE_HOST
• REMOTE_PORT
• REMOTE_USER
• REMOTE_IDENT
• REQUEST_METHOD
• SCRIPT_FILENAME
• PATH_INFO
• QUERY_STRING
• AUTH_TYPE

server internals

• DOCUMENT_ROOT
• SERVER_ADMIN
• SERVER_NAME
• SERVER_ADDR
• SERVER_PORT
• SERVER_PROTOCOL
• SERVER_SOFTWARE

date and time

• TIME_YEAR
• TIME_MON
• TIME_DAY
• TIME_HOUR
• TIME_MIN
• TIME_SEC
• TIME_WDAY
• TIME

specials

• API_VERSION
• THE_REQUEST
• REQUEST_URI
• REQUEST_FILENAME
• IS_SUBREQ
• HTTPS

Variable Descriptions

• REQUEST_FILENAME
  • The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as when used in virtual host context, the same value as REQUEST_URI.
• HTTPS
  • Will contain the text “on” if the connection is using SSL/TLS, or “off” otherwise. (This variable can be safely used regardless of whether or not mod_ssl is loaded).

Flag Descriptions

• ‘nocase|NC‘ (no case) :: This makes the test case-insensitive – differences between ‘A-Z’ and ‘a-z’ are ignored, both in the expanded TestString and the CondPattern.
• ‘ornext|OR‘ (or next condition) (RewriteCond Only) :: Use this to combine rule conditions with a local OR instead of the implicit AND.
• ‘last|L’ :: Stop the rewriting process immediately and don’t apply any more rules.
• ‘proxy|P’ :: Force the substitution URL to be internally sent as a proxy request.
• ‘qsappend|QSA‘ :: Appends any query string created in the rewrite target to any query string that was in the original request URL.
• ‘redirect|R[=code]‘ :: Forces an external redirect, optionally with the specified HTTP status code.
• ‘forbidden|F‘ :: Returns a 403 FORBIDDEN response to the client browser.

Good Examples

• Adding www to all requests
  • RewriteCond %{HTTP_HOST} !^www [NC]
  • RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]
• Forcing all requests to HTTPS
  • RewriteCond %{HTTPS} off
  • RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]
• Redirect a specific subweb to another domain
  • RewriteRule ^/?subweb/(.*) http://other.example.com/$1 [R,L,QSA]
• Block specific IPs from access
  • RewriteCond %{REMOTE_ADDR} ^127\.0\.0
  • RewriteRule ^ – [F,L]
• Creating a filesystem alias with modrewrite
  • RewriteRule ^/?alias/(.*) /var/www/vhosts/$1/httpdocs/$1 [L,R]
• A condition to stop CMS software from over-riding fullstatus (added before the offending rewriterule)
  • RewriteCond ${REQUEST_URI} !server-status [NC]

Bad Examples

• Recursive rewrite
  • RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]

Gotchas

• Some rewrites may conflict with existing rewrites provided by many CMS packages (wordpress, drupal, joomla, etc). Check for any existing rewrites in a .htaccess file.
• Rewriterule and rewritecond can only be used in the following contexts: server config, virtual host, directory, .htaccess

Common Troubleshooting

• Enable the rewrite logs with RewriteLog and RewriteLevel
  • RewriteLog <file path>
  • RewriteLogLevel 3 # ranges from 0 to 9
• Check your regular expressions against a PCRE checker (they are very bountiful on the Internet [http://tinyurl.com/3hop7xu]).
• Utilize curl to test redirects (R), `curl -I example.com`

As I mentioned last time, you want to be comfortable with the existing documentation on CFLAGS before going crazy trying to play with. It also helps to have a good understanding of what you’re doing to the code when you modify these “sacred” parameters.

Alright, now that the CYA is out of the way let’s take this one step further. Last time we talked about figuring out which instruction sets your processor understood and how to figure out what `-m` flags would get those instruction sets into the binaries on your system. This time we’ll be talking about making sure those same flags are in your use flags (just to be sure they’re picked up by the system).

Finding Flag Names

So how do we find the flags that do what we want? Well, as always BASH is our friend and can be used to find this answer in a mostly automated fashion:

. /etc/make.conf && gcc -Q -c -v ${CFLAGS} --help=target | grep enabled

This displays the currently enabled flags based on your CFLAGS parameter and allows us to find which flags have use flags with the following one liner:

gawk '/-m.*/ { print $1 }' | cut -d 'm' --complement -f 1 | xargs -I{} equery h "{}"

Conclusion

Using a little scripting we can extract the necessary information to quickly determine if there are any use flags we should be adding for particular compiler flags that our system might support.  With this last level of optimization beyond the previous time’s we should be ready to move on to -O3 (for the daring) and watch our machine’s nose bleed.

It seems that disk snapshots have become a hot topic and a confusing topic.  I intend to simply outline what snapshots look like in terms of the lower layers of abstraction and nothing more.  Snapshots are built into things like LVM, SAN, etc but I will not be covering those technologies.  Instead, what I intend to cover is the abstractions ranging from the disk device to the snapshot.

The Disk

A hard disk is typically chopped up into pieces called partitions (and this is the unit I’ll use as my foundation).  These logical separations of the blocks on a disk allow us to make sure that disks are not over-allocated for one particular purpose or lose access to a system if one partition fills up, etc.  Partitioning is only the beginning.   We are able to take the idea of partitions and extend the idea of a contiguous region of blocks to the strange concept of snapshots as well.

Writing to Disk

Using our simplified view of a disk (a contiguous region of blocks) we can see that the simplest way to write data (ignoring filesystems) is to simply take our data one block at a time and place it on the disk.  This is great until we need to remove blocks or update blocks (which is probably why we only use this type of writing for tapes).  We’ll have to add some logic to this view of disks to handle the complexities of files but for all intensive purposes, disks are simply groups of blocks available for writing and groups of blocks that are already used.  If we want a consistent view of the data at any point it’s simply a matter of making sure nothing is writing to the disk (partition) we’re interested in.  This is where snapshots can help us out.

Snapshots

A snapshot creates another region of blocks we can write to.  Let me start that again … when you create a snapshot of some disk (snapshots are not standalone by any means) the following happens:

• It reserves space for a changelog (or writes that would happen to the disk) in the difference disk
• It creates a new way to access the original disk via the snapshot name (which refers to the base disk)
• It begins sending all writes to the reserved space for the changelog (depicted in the figure below)

Thus, we have the following situation on the disk (again very simplified):

Deleting Snapshots

Since a snapshot isn’t a true partition what happens when one of these gets removed?  The obvious is that the snapshot name gets removed so it’s no longer accessible, but that’s not going to help rectify the two regions of data we now have on the disk.  This is handled by making the difference disk a literal list of differences.  I find it best to think of it (even though it may not actually be implemented in this fashion) as a queue of block changes or a transaction log for writes to the disk.  With this visualization of the process it makes sense as to what happens when the snapshot is removed:

• The reference to the base disk is removed
• The difference disk is replayed onto the base disk (applying all recorded changes)

That’s all there is to it.  Everything continues as if the snapshot exists until it has been fully replayed and then disk access resumes as normal without the snapshot.

Conclusion

Snapshots provide a convenient way to access a frozen image of a disk which is perfect for backups or point in time restores of data.  This is typically used via LVM, virtualbox, SANs, etc and has far-reaching impacts that allow system administration to be easier.

Backups are a subject I return to semi-frequently with a passion to never be in an “oh shit” scenario.  Last time I built my backup system, bacula with a postgresql DB backend, I determined that I would move to a common database backup script for all of my databases.  Holland fit this bill perfectly with support for postgres, sqlite and mysql.  This allows one command to backup all of my databases on all of my servers and subsequently creates a much simpler bacula configuration (the database job is defined the same as the catalog job).

The Solution

The problem I had when configuring holland to backup postgresql is that there was no example configuration file.  It wasn’t hard to craft a working default postgres configuration and the following is what I came up with (/etc/holland/backupsets/default.conf:

[holland:backup]
plugin = pgdump
backups-to-keep = 1
auto-purge-failures = yes
purge-policy = after-backup
estimated-size-factor = 1.0

[pgdump]
role = postgres

[pgauth]
username = postgres

Conclusion

Setting up holland to backup databases is incredibly easy and flexible.  By having a common backup solution for all databases other configurations become easier and processes can be streamlined.

The bindings usually are read from the file `/etc/inputrc`by bash but zsh does not do this by default.  There are probably more elegant solutions but a quick brute force solution is to create a bindkeys file out of inputrc:

gawk '$1 ~ /.*:/ { print "bindkey",$1,$2 }' /etc/inputrc | \
sed -e 's/://g' > ~/.zshrc-bindkeys

Once this file has been crafted it’s simply a matter of invoking it from your .zshrc with `source ~/.zshrc-bindkeys`.

Portage is an amazingly simple and complex piece of technology.  The simplicity in each piece’s ability to do a specific function comes together in a complex package management system that rivals all other forms of package management (at least in my opinion).  Automating updates is something that admins everywhere do out of necessity.  Heck, automating everything is an admin’s life.  Automating portage’s updates is a bit more harrowing than other package management systems but it isn’t impossible.

Problem

As admins we attempt to simplify the work we actually do by writing scripts and programs to do most of our job for us.  It’s often been said that systems admins are the only people whose job description is to remove their job responsibilities.

Portage doesn’t have any default automation for doing nightly or even weekly portage updates but that doesn’t stop the creative from coming up with their own solution.  A simple but elegant solution is to create a small cron script that runs every day.  The problem comes when you want to read the wonderful output of portage (sometimes these messages can guide you when problems are about to occur) to avert disasters.  If the updates are performed from cron, the output will be preserved in an e-mail to the appropriate user but then we have to sift through all of the output at once.  This also doesn’t solve the issue if the updates are performed by another utility such as puppet.  These annoying little changes to the problem require a slightly more elegant solution.

Solution

The solution is to take advantage of portage’s logging specifications.  From the make.conf man file:

• PORTAGE_ELOG_CLASSES
• PORTAGE_ELOG_SYSTEM
• PORTAGE_ELOG_COMMAND
• PORTAGE_ELOG_MAILURI
• PORTAGE_ELOG_MAILFROM
• PORTAGE_ELOG_MAILSUBJECT

Using a combination of these directives in the make.conf file allows us to log the reports from portage to a large number of locations.  If we wanted to simply add mailing output (not the full build output just the messages) we would add the following directives to make.conf:

PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_MAILFROM="portage@alunduil.com"

This simply adds the mailing log utility to portage and specifies that the e-mails come from the address portage@alunduil.com.  Of course, much more complex configurations can be crafted to suit any admins’ needs.

Conclusion

Letting your servers notify you of possible actions is one way of automating maintenance tasks; making maintenance eventually disappear from your task list. By starting with the tasks that are repeated the most frequently, you can quickly free up time for higher level automation and organization which leads to a cleaner and sturdier infrastructure.

The utility, pclean, does all of this and only has one major problem (so far) before I shall call it good enough for a 1.0 release. If you would like to try this little utility; it’s available in my overlay and if you notice any odd behavior please report it to my bugzilla.

It appears the output of ping has changed in this release of iputils from referring to the icmp sequence numbers as icmp_seq to icmp_req which obliterates the ping.pl script that cacti uses to do pings of servers it watches.

The Fix

The fix is quite simple: change the seq to req in the grep line of ping.pl but the following fix is probably more versatile (and will be checked for upstream).

*** ping.pl     2010-07-09 17:33:46.000000000 -0500
--- ping.pl.new 2010-10-24 18:22:16.325881546 -0500
***************
*** 4,10 ****
  $host = $ARGV[0];
  $host =~ s/tcp:/$1/gis;

! open(PROCESS, "ping -c 1 $host | grep icmp_seq | grep time |");
  $ping = ;
  close(PROCESS);
  $ping =~ m/(.*time=)(.*) (ms|usec)/;
--- 4,10 ----
  $host = $ARGV[0];
  $host =~ s/tcp:/$1/gis;

! open(PROCESS, "ping -c 1 $host | grep -E icmp_\(r\|s\)eq | grep time |");
  $ping = ;
  close(PROCESS);
  $ping =~ m/(.*time=)(.*) (ms|usec)/;

Conclusion

Sometimes things break just because of small changes. This is a simple example of that and the quick fix for the annoyance of not recording your ping times.

Hamachi is a lightweight personal VPN connector that is a breeze to setup but there can be some pain if you don’t know what to expect. As always Gentoo provides us with an ebuild that simplifies the installation process but getting up and running is still a little confusing.

Installation and Setup

User Specific Configurations

The obvious first step is `emerge -av hamachi` (this only available to ~arch right now so add to `package.keywords` as necessary). The following are the typical instructions to install hamachi on Gentoo from portage:

1. Add to `packages.keywords` if necessary
2. `emerge -av hamachi`
3. `rc-update add tuncfg default`

After these steps have been taken you can run hamachi as any user on the system for ad-hoc VPN creation.

Server Wide Configuration

If you prefer to do a system wide on boot VPN with hamachi this is also possible but requires a slightly different setup:

1. Add to `packages.keywords` if necessary
2. `emerge -av hamachi`
3. rc-update add hamachi default

Now all configuration should be placed inside `/etc/hamachi` for this setup so the system will automatically start routing traffic correctly.

Kernel Configuration

For hamachi to work correctly you do need the tun parameter in your kernel or loaded as a module. This parameter is located in Device Drivers->Network device support->Universal TUN/TAP device driver support.

Using Hamachi

Now that hamachi is on the system we need to start using it. The server-wide installation doesn’t require this (but I’m sure you can use this method to create a configuration usable by the server-wide instance) but the user specific usage does.

Starting hamachi is as simple as the following:

1. `hamachi-init`
2. `hamachi start`
3. `hamachi login`
4. `hamachi create [ ]`
5. `hamachi join [ ]`
6. `hamachi go-online `

That’s it. You’re now connected to a private network named . You can view who else is connected to your network with `hamachi list` and `hamachi get-nicks`.

Conclusion

Setting up a VPN can be daunting (see the OpenVPN configuration documentation) or it can be a breeze with hamachi. Need a quick VPN for LAN gaming or a VPN for performing maintenance over the internet on a device behind a firewall? Hamachi may be the quick solution you’re looking for.

Not overly recently, I was asked to configure a chroot jail by setting up bind mounts instead of copying over the correct binaries. The incident caused a really interesting system failure, a wake-up call to myself, and an ambitious project to create an autochroot command (more later).

The Back Story

When setting up a chroot jail it’s well known that all the binaries and references that will be used in that jail need to be inside that jail. This creates quite a complicated configuration for something even as simple as creating a chroot jail that only has bash and bash internals in it. Not only do we need to create a clean directory structure and duplicate our file system hierarchy but we also need to create all the dependencies (and for bash this quickly grows):

ldd /bin/bash
linux-vdso.so.1 => (0x00007fff189ff000)
libncurses.so.5 => /lib/libncurses.so.5 (0x00007f6ca8604000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f6ca8400000)
libc.so.6 => /lib/libc.so.6 (0x00007f6ca809a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6ca8854000)

Once all the dependencies are in place, the proper items to do work in this minimal shell environment also need to be installed. This may include device nodes or system environment files (e.g. /etc/passwd, /etc/shadow). Which make some commands a real pain to install in a chroot by hand (e.g. /usr/bin/ssh, /usr/bin/scp, etc).

The Problem

So, it’s hard; that’s not a big deal. There are ways to make this easier. Things like bind mounts:

mount -o bind

These handy features allow you to take any directory in the system and mount it somewhere else. These become extremely useful when setting up a chroot environment for system recovery via a livecd but can be a ticking time bomb if used incorrectly.

Alright, let’s solve our chroot problem with bind mounts, but I bet you can guess by this point how it’s going to be accomplished. We’re simply going to mount all of the necessary areas of the external filesystem inside the chroot:

mount -o bind /dev /chroot/dev
mount -o bind /lib /chroot/lib
mount -o bind /usr/lib /chroot/usr/lib

Now, that chroot was much easier to set up and work with, but what happens if an unsuspecting party does the unthinkable: `rm -fr /chroot`. They’re simply cleaning up a chroot environment that doesn’t need to be used anymore, right? Unfortunately, that’s not the case. This will remove all files in /chroot including /dev, /lib, /usr/lib, etc.

The Lesson

Don’t ever use bind mounts in chroot environments unless you know exactly what you are doing! Even then, it may be best practice to simply not use bind mounts. You may be thinking, why can’t we simply mount these points readonly? Unfortunately, bind mounts aren’t that smart. This is what happens when you attempt a bind mount readonly:

mount -o bind,or /proc /mnt
mount: warning: /mnt seems to be mounted read-write.
mount
/proc on /mnt type none (rw,bind)

Not quite what we were looking for and unfortunate for our quick solution.

Conclusion

Never use bind mounts for a chroot environment (except maybe livecd chroot environments). It may take more time but copying the necessary binaries to the chroot is much safer and will keep your system from any inadvertent harm.

Now, I mentioned that I came up with a much more ambitious solution to this problem, autoroot. This small utility (when finished) will be able to take a directory and list of binaries and automatically create an appropriate chroot environment. The system should be smart enough to take into consideration different distributions and installation points (a current limitation of existing solutions).