Introduction

I was recently requested to document common apache rewrite pitfalls and examples and crafted the following document as a response.  It is intended as a two page document (Rewrite Cheat Sheet) where the first page is a reference guide of commonly used rewrite variables and flags and the second page is a short list of examples, gotchas, and troubleshooting advice.

Rewrite Cheat Sheet

Common Variables

HTTP Headers

• HTTP_USER_AGENT
• HTTP_REFERER
• HTTP_COOKIE
• HTTP_FORWARDED
• HTTP_HOST
• HTTP_PROXY_CONNECTION
• HTTP_ACCEPT

connection & request

• REMOTE_ADDR
• REMOTE_HOST
• REMOTE_PORT
• REMOTE_USER
• REMOTE_IDENT
• REQUEST_METHOD
• SCRIPT_FILENAME
• PATH_INFO
• QUERY_STRING
• AUTH_TYPE

server internals

• DOCUMENT_ROOT
• SERVER_ADMIN
• SERVER_NAME
• SERVER_ADDR
• SERVER_PORT
• SERVER_PROTOCOL
• SERVER_SOFTWARE

date and time

• TIME_YEAR
• TIME_MON
• TIME_DAY
• TIME_HOUR
• TIME_MIN
• TIME_SEC
• TIME_WDAY
• TIME

specials

• API_VERSION
• THE_REQUEST
• REQUEST_URI
• REQUEST_FILENAME
• IS_SUBREQ
• HTTPS

Variable Descriptions

• REQUEST_FILENAME
  • The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as when used in virtual host context, the same value as REQUEST_URI.
• HTTPS
  • Will contain the text “on” if the connection is using SSL/TLS, or “off” otherwise. (This variable can be safely used regardless of whether or not mod_ssl is loaded).

Flag Descriptions

• ‘nocase|NC‘ (no case) :: This makes the test case-insensitive – differences between ‘A-Z’ and ‘a-z’ are ignored, both in the expanded TestString and the CondPattern.
• ‘ornext|OR‘ (or next condition) (RewriteCond Only) :: Use this to combine rule conditions with a local OR instead of the implicit AND.
• ‘last|L’ :: Stop the rewriting process immediately and don’t apply any more rules.
• ‘proxy|P’ :: Force the substitution URL to be internally sent as a proxy request.
• ‘qsappend|QSA‘ :: Appends any query string created in the rewrite target to any query string that was in the original request URL.
• ‘redirect|R[=code]‘ :: Forces an external redirect, optionally with the specified HTTP status code.
• ‘forbidden|F‘ :: Returns a 403 FORBIDDEN response to the client browser.

Good Examples

• Adding www to all requests
  • RewriteCond %{HTTP_HOST} !^www [NC]
  • RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]
• Forcing all requests to HTTPS
  • RewriteCond %{HTTPS} off
  • RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]
• Redirect a specific subweb to another domain
  • RewriteRule ^/?subweb/(.*) http://other.example.com/$1 [R,L,QSA]
• Block specific IPs from access
  • RewriteCond %{REMOTE_ADDR} ^127\.0\.0
  • RewriteRule ^ – [F,L]
• Creating a filesystem alias with modrewrite
  • RewriteRule ^/?alias/(.*) /var/www/vhosts/$1/httpdocs/$1 [L,R]
• A condition to stop CMS software from over-riding fullstatus (added before the offending rewriterule)
  • RewriteCond ${REQUEST_URI} !server-status [NC]

Bad Examples

• Recursive rewrite
  • RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]

Gotchas

• Some rewrites may conflict with existing rewrites provided by many CMS packages (wordpress, drupal, joomla, etc). Check for any existing rewrites in a .htaccess file.
• Rewriterule and rewritecond can only be used in the following contexts: server config, virtual host, directory, .htaccess

Common Troubleshooting

• Enable the rewrite logs with RewriteLog and RewriteLevel
  • RewriteLog <file path>
  • RewriteLogLevel 3 # ranges from 0 to 9
• Check your regular expressions against a PCRE checker (they are very bountiful on the Internet [http://tinyurl.com/3hop7xu]).
• Utilize curl to test redirects (R), `curl -I example.com`

Memcached is a simple key/value memory store that allows values from pretty much any application to be stored in memory for quick retrieval. Most languages have simple APIs that make this very easy from a programming standpoint. This also means that it’s not quite as simple to configure as APC (but similar for modern applications).

Installation and Configuration

To install memcached I recommend using your choice of installation (mine being emerge) and then checking the configuration (again in the case of Gentoo) located at /etc/conf.d/memcached.

Update the file to listen on the interface you want, use the memory you want, etc and then start the daemon up. Now if you’re using memcached on the same server as your apache server most webapps will be able to just turn on memcached and connect and work, but if you’re using memcached and apache on separate servers it’s a little more work.

WordPress and memcached

I’ve already touted the wordpress plugin: W3 Total Cache but unfortunately this plugin doesn’t allow you to easily configure memcached usage through the web interface. It defaults to using the memcached server located at 127.0.0.1:11211 which if we’re using separate servers is less than ideal. The file we need to modify with a simply sed (or by hand if you prefer knowing what’s going on) is %{DOCROOT}/wp-content/w3-total-cache-config.php.

We just need to change the memcached servers to point at our server’s location before enabling it in the interface. To easily accomplish this we can use sed:

sed -i -e ‘s/127.0.0.1:11211/: /g’ %{DOCROOT}/wp-content/w3-total-cache-config.php

Mediawiki and memcached

The other application we can easily setup for memcached is mediawiki. Mediawiki wants us to modify its LocalSettings.php with the following additions:

## Shared memory settings
$wgMainCacheType = CACHE_MEMCACHED;
$wgParserCacheType = CACHE_MEMCACHED;
$wgMessageCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array(“giskard.alunduil.com:11211″);

$wgSessionsInMemcached = true;

These settings simply turn on memcached (CACHE_MEMCACHED) for the various caching areas that mediawiki uses and specifies an array of memcached servers to store that caching information in.

If you do have more than one memcached server configured and available you can use the following syntax to include all of them with various weights:

$wgMemCachedServers = array(array(“host:port”, weight), array(“host2:port2″, weight));

Conclusion

Using memcached can greatly improve the performance of webapps or other applications that don’t need to present the most up to date information. This can reduce traffic to an overloaded database or configuration file or whatever other store you want to place a cache in front of.

First things first …

A few common things to possibly turn on, install, or enable:

• APC (Another PHP Cache)
• Memcached

MySQL Optimization

The easiest way (and the quickest) to determine what can be done to optimize MySQL is to use a script called mysqltuner.pl. The really interesting thing about this script is that to get it you simply run the following:

wget mysqltuner.pl

After you’ve gotten this script run it with your perl interpreter:

perl mysqltuner.pl

It will prompt you for your username and password for mysql and then print out a nice report outlining how your mysql daemon has been running. Here’s an example of this output:

>> MySQLTuner 1.0.1 – Major Hayden
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with ‘–help’ for additional options and output filtering
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password:

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.90-log
[OK] Operating on 32-bit architecture with less than 2GB RAM

——– Storage Engine Statistics ——————————————-
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 28M (Tables: 96)
[--] Data in InnoDB tables: 1M (Tables: 55)
[!!] Total fragmented tables: 2

——– Performance Metrics ————————————————-
[--] Up for: 1h 39m 49s (24K q [4.163 qps], 354 conn, TX: 9M, RX: 34M)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 148.6M global + 960.0K per thread (100 max threads)
[OK] Maximum possible memory usage: 242.3M (11% of installed RAM)
[OK] Slow queries: 0% (0/24K)
[OK] Highest usage of available connections: 28% (28/100)
[OK] Key buffer size / total MyISAM indexes: 64.0K/6.3M
[OK] Key buffer hit rate: 99.6% (164K cached / 682 reads)
[OK] Query cache efficiency: 81.3% (14K cached / 17K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 463 sorts)
[!!] Joins performed without indexes: 42
[OK] Temporary tables created on disk: 12% (54 on disk / 417 total)
[OK] Thread cache hit rate: 70% (106 created / 354 connections)
[!!] Table cache hit rate: 2% (48 open / 2K opened)
[OK] Open file limit used: 9% (95/1K)
[OK] Table locks acquired immediately: 98% (7K immediate / 7K locks)
[!!] Connections aborted: 7%
[OK] InnoDB data size / buffer pool: 1.0M/4.0M

——– Recommendations —————————————————–
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours – recommendations may be inaccurate
Adjust your join queries to always utilize indexes
Increase table_cache gradually to avoid file descriptor limits
Your applications are not closing MySQL connections properly
Variables to adjust:
join_buffer_size (> 128.0K, or always use indexes with joins)
table_cache (> 48)

The first thing to notice is that mysql should be running for 24 to 48 hours before you trust the results of this script. The second thing to acknowledge is that it isn’t omniscient. Understanding how these parameters affect your memory footprint and your runtime efficiency are what we’re most interested in though.

Glossary of Parameters

The best source of my.cnf parameters is of course the MySQL documentation, but I’ll clarify a few of the more pertinent items from the above output.

• Maximum Possible Memory – This is not the actual hard limit on your memory usage but a calculated theoretical maximum based on the parameters in your my.cnf and if this is over ~90% you may use your swap more than you want to. After making any changes I suggest you check this to make sure you haven’t over-allocated yourself.
• Highest Usage of Available Connections – This is the historical max connections used by mysql and if it’s at 100% this isn’t a guarantee that you need to increase max_connections but may just indicate that your connections spiked once. That being said I recommend following its advice if you have the memory for it.
• Query Cache Efficiency – The hit rate for the cache on your queries (I hope you have enabled). As with any hash table if you’re not hitting what’s in the top bucket you need to increase the number of buckets.
• Temporary Tables Created on Disk – This can be a finicky parameter to get right and is the first place I would recommend cutting if you don’t have the memory to run mysql like you would want, but if you do have the memory then up this as far as you can within reason.
• Thread Cache Hit Rate – Just like any other cache.
• Table Cache Hit Rate – Just like any other cache.

Fragmented Tables

If you noticed the fragmented tables along with the recommendation to run OPTIMIZE TABLE that’s not something that will typically slow mysql down noticeably with today’s disks, but if you want to defragment them I recommend reading this article on the topic.

Apache Tweaking

The following parameters are usually in httpd.conf unless you run a distribution that organizes apache in an easier to work with manner (my preference of course is Gentoo which stores these parameters in: 00_mpm.conf).

The parameters for tweaking apache’s processes depend on which worker module you have in use. The following is the pertinent sections from my configuration:

#prefork MPM
# This is the default MPM if USE=-threads
#
# MinSpareServers: Minimum number of idle child server processes
# MaxSpareServers: Maximum number of idle child server processes

StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 50
MaxRequestsPerChild 500

# worker MPM
# This is the default MPM if USE=threads
#
# MinSpareThreads: Minimum number of idle threads available to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# ThreadsPerChild: Number of threads created by each child process

StartServers 2
MinSpareThreads 15
MaxSpareThreads 50
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 1000

These parameters are pretty self explanatory but just for completeness’ sake I’ll do a small recap on these here:

* StartServers – The number of servers to have running and handling connections at first launch.
* MinSpareServers – The minimum number of servers to have running not currently handling connections.
* MaxSpareServers – The maximum number of servers to have running not currently handling connections.
* MaxClients – The maximum number of clients that can simultaneously connect to Apache.
* MaxRequestsPerChild – The maximum number of requests that a child will respond to before terminating.

Using these parameters we can control the amount of memory that Apache will consume at the cost of other things we may want to have like lots of connections or many sequential requests.

Conclusion

There are a lot of different things that can be done for a server that seems to be performing less than ideally and I’ve only covered a fraction of things that can happen. As always with this topic the situation will change based on your needs and how your server runs, but at least now you may have a starting point for what to modify after you’ve troubleshooted every other thing that could go wrong.

I’m assuming for this quick guide that you already have a website installed and consequently apache.

emerge -av awstats

A quick breakdown on the use flags from gentookit’s equery:

• apache2 : Add Apache2 support
• geoip : Add geoip support for country and city lookup based on IPs
• vhosts : Adds support for installing web-based applications into a virtual-hosting environment

Configuration

Lots of configuration needs to occur for awstats to work correctly:

First, we need to setup a configuration file for the web site so we can update the statistics.
cp /etc/awstats/awstats.model.conf /etc/awstats/awstats..conf

Where fqdn is the fully qualified domain name of your website you’ll be monitoriing. After you’ve copied the default configuration customize it for your particular website.

Second, you need to enable awstats in your apache vhost configuration:

CustomLog /var/www/localhost/log/apache/production.log combined

Alias /awstats/classes “/usr/share/webapps/awstats/6.9-r1/htdocs/classes/”
Alias /awstats/css “/usr/share/webapps/awstats/6.9-r1/htdocs/css/”
Alias /awstats/icons “/usr/share/webapps/awstats/6.9-r1/htdocs/icon/”
ScriptAlias /awstats/ “/usr/share/webapps/awstats/6.9-r1/hostroot/cgi-bin/”
ScriptAlias /awstats “/usr/share/webapps/awstats/6.9-r1/hostroot/cgi-bin/awstats.pl”
ScriptAlias /awstats.pl “/usr/share/webapps/awstats/6.9-r1/hostroot/cgi-bin/awstats.pl”
Options None
AllowOverride None

Order allow,deny
Allow from all

Options ExecCGI
AllowOverride None

Order allow,deny
Allow from all

Options None
AllowOverride None
Order allow,deny
Allow from all
Options None
AllowOverride None
Order allow,deny
Allow from all

And verify the logging output in /etc/apache2/modules.d/00_mod_log_config.conf:
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%h %l %u %t \”%r\” %>s %b” common

LogFormat “%{Referer}i -> %U” referer
LogFormat “%{User-Agent}i” agent
LogFormat “%v %h %l %u %t \”%r\” %>s %b %T” script
LogFormat “%v %h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” VLOG=%{VLOG}e” vhost
# You need to enable mod_logio.c to use %I and %O
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %I %O” combinedio
LogFormat “%v %h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %I %O” vhostio

# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a
# container, they will be logged here. Contrariwise, if you *do*
# define per- access logfiles, transactions will be
# logged therein and *not* in this file.
CustomLog /var/log/apache2/access_log common

# If you would like to have agent and referer logfiles,
# uncomment the following directives.
#CustomLog /var/log/apache2/referer_log referer
#CustomLog /var/log/apache2/agent_logs agent

# If you prefer a logfile with access, agent, and referer information
# (Combined Logfile Format) you can use the following directive.
#CustomLog /var/log/apache2/access_log combined

And Lastly, you need to add a cron entry to update the statistics on a regular basis:
# AWStats
*/15 * * * * perl /usr/share/webapps/awstats/6.9-r1/hostroot/cgi-bin/awstats.pl -config=www.alunduil.com -update > /dev/null

Everything should be running smoothly but give your installation some time to begin collecting statistics.

!Ridiculous System Load