As I mentioned last time, you want to be comfortable with the existing documentation on CFLAGS before going crazy trying to play with. It also helps to have a good understanding of what you’re doing to the code when you modify these “sacred” parameters.

Alright, now that the CYA is out of the way let’s take this one step further. Last time we talked about figuring out which instruction sets your processor understood and how to figure out what `-m` flags would get those instruction sets into the binaries on your system. This time we’ll be talking about making sure those same flags are in your use flags (just to be sure they’re picked up by the system).

Finding Flag Names

So how do we find the flags that do what we want? Well, as always BASH is our friend and can be used to find this answer in a mostly automated fashion:

. /etc/make.conf && gcc -Q -c -v ${CFLAGS} --help=target | grep enabled

This displays the currently enabled flags based on your CFLAGS parameter and allows us to find which flags have use flags with the following one liner:

gawk '/-m.*/ { print $1 }' | cut -d 'm' --complement -f 1 | xargs -I{} equery h "{}"

Conclusion

Using a little scripting we can extract the necessary information to quickly determine if there are any use flags we should be adding for particular compiler flags that our system might support.  With this last level of optimization beyond the previous time’s we should be ready to move on to -O3 (for the daring) and watch our machine’s nose bleed.

Portage is an amazingly simple and complex piece of technology.  The simplicity in each piece’s ability to do a specific function comes together in a complex package management system that rivals all other forms of package management (at least in my opinion).  Automating updates is something that admins everywhere do out of necessity.  Heck, automating everything is an admin’s life.  Automating portage’s updates is a bit more harrowing than other package management systems but it isn’t impossible.

Problem

As admins we attempt to simplify the work we actually do by writing scripts and programs to do most of our job for us.  It’s often been said that systems admins are the only people whose job description is to remove their job responsibilities.

Portage doesn’t have any default automation for doing nightly or even weekly portage updates but that doesn’t stop the creative from coming up with their own solution.  A simple but elegant solution is to create a small cron script that runs every day.  The problem comes when you want to read the wonderful output of portage (sometimes these messages can guide you when problems are about to occur) to avert disasters.  If the updates are performed from cron, the output will be preserved in an e-mail to the appropriate user but then we have to sift through all of the output at once.  This also doesn’t solve the issue if the updates are performed by another utility such as puppet.  These annoying little changes to the problem require a slightly more elegant solution.

Solution

The solution is to take advantage of portage’s logging specifications.  From the make.conf man file:

• PORTAGE_ELOG_CLASSES
• PORTAGE_ELOG_SYSTEM
• PORTAGE_ELOG_COMMAND
• PORTAGE_ELOG_MAILURI
• PORTAGE_ELOG_MAILFROM
• PORTAGE_ELOG_MAILSUBJECT

Using a combination of these directives in the make.conf file allows us to log the reports from portage to a large number of locations.  If we wanted to simply add mailing output (not the full build output just the messages) we would add the following directives to make.conf:

PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_MAILFROM="portage@alunduil.com"

This simply adds the mailing log utility to portage and specifies that the e-mails come from the address portage@alunduil.com.  Of course, much more complex configurations can be crafted to suit any admins’ needs.

Conclusion

Letting your servers notify you of possible actions is one way of automating maintenance tasks; making maintenance eventually disappear from your task list. By starting with the tasks that are repeated the most frequently, you can quickly free up time for higher level automation and organization which leads to a cleaner and sturdier infrastructure.

Often in administration people make reference to a magic number known as load average but it’s not always clear what this number means (besides being a magical indicator of whether or not the server is going to barf in a terrible fashion soon). This number doesn’t have a whole lot of detail in and of itself (it’s meant to be a quick-glance overall health check afterall).

What is it already?

The load average is an exponentially damped/weighted moving average that is similar to a running n^2 average. This number is calculated not on every clock tick but in accordance with the jiffies the kernel is tracking. Every time slice (which can although shouldn’t be tweaked in the kernel) the average is calculated based on the previous values. This rolling average allows us to keep a minimal amount of information on hand and still have an average since boot.

The load that gets placed into this average algorithm is simply a count of the number of processes in the run queue at that instant. Thus, since processes who are waiting on I/O (those in the D state) need to periodically check in or wait to be woken up by the kernel; these processes can contribute to the count of processes in the run queue. Since these processes aren’t taking CPU time but are taking space in the run queue they can increase the apparent load on the server (thus bringing in the I/O wait of the system into the load average) without raising the CPU usage time.

Conclusion

The load average doesn’t tell you a whole lot of information but coupled with information from `iostat` or your CPU usage you can quickly use it to gauge whether your server is falling over itself or not. Since it reports the number of processes in the run queue it is safe to assume a reasonably efficient use of hardware would dictate you want n+1 as your load average where n is the number of cores in the system.

References

More information (including source code for this calculation) can be found here: http://en.wikipedia.org/wiki/Load_%28computing%29

An excellent reference on how the Linux process life cycle works is Robert Love’s Linux Kernel Development (which recently had a third edition released).

Another reference on the Linux process life cycle is http://wiki.kldp.org/wiki.php/ProcessManagement

Memcached is a simple key/value memory store that allows values from pretty much any application to be stored in memory for quick retrieval. Most languages have simple APIs that make this very easy from a programming standpoint. This also means that it’s not quite as simple to configure as APC (but similar for modern applications).

Installation and Configuration

To install memcached I recommend using your choice of installation (mine being emerge) and then checking the configuration (again in the case of Gentoo) located at /etc/conf.d/memcached.

Update the file to listen on the interface you want, use the memory you want, etc and then start the daemon up. Now if you’re using memcached on the same server as your apache server most webapps will be able to just turn on memcached and connect and work, but if you’re using memcached and apache on separate servers it’s a little more work.

WordPress and memcached

I’ve already touted the wordpress plugin: W3 Total Cache but unfortunately this plugin doesn’t allow you to easily configure memcached usage through the web interface. It defaults to using the memcached server located at 127.0.0.1:11211 which if we’re using separate servers is less than ideal. The file we need to modify with a simply sed (or by hand if you prefer knowing what’s going on) is %{DOCROOT}/wp-content/w3-total-cache-config.php.

We just need to change the memcached servers to point at our server’s location before enabling it in the interface. To easily accomplish this we can use sed:

sed -i -e ‘s/127.0.0.1:11211/: /g’ %{DOCROOT}/wp-content/w3-total-cache-config.php

Mediawiki and memcached

The other application we can easily setup for memcached is mediawiki. Mediawiki wants us to modify its LocalSettings.php with the following additions:

## Shared memory settings
$wgMainCacheType = CACHE_MEMCACHED;
$wgParserCacheType = CACHE_MEMCACHED;
$wgMessageCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array(“giskard.alunduil.com:11211″);

$wgSessionsInMemcached = true;

These settings simply turn on memcached (CACHE_MEMCACHED) for the various caching areas that mediawiki uses and specifies an array of memcached servers to store that caching information in.

If you do have more than one memcached server configured and available you can use the following syntax to include all of them with various weights:

$wgMemCachedServers = array(array(“host:port”, weight), array(“host2:port2″, weight));

Conclusion

Using memcached can greatly improve the performance of webapps or other applications that don’t need to present the most up to date information. This can reduce traffic to an overloaded database or configuration file or whatever other store you want to place a cache in front of.

Making PHP run faster is usually pretty easy when you see some of the code that people can write (myself included) but what if we either don’t want to look at their code or don’t want to fix the code? What else can we do to improve the runtime and the load time of PHP web sites (this website included)?

We’ll cover how to use APC (Alternative PHP Cache) to help alleviate some of the issues of a slow web site. Just like last time we optimized MySQL and talked about optimizing Apache we’ll continue on the path to getting PHP applications as fast as we can (without touching a line of code).

Enter APC

APC is PEAR project and a PHP module that after installed and a restart of Apache is already doing some work to make life better. By default APC turns on op-code caching which saves on the compile and execution times of PHP. The configuration for this module is pretty straight forward, but let’s look at a copy of the one I’m using:

; /etc/php/apache2-php5/ext-active/apc.ini (On Gentoo anyways …)
extension=apc.so
apc.enabled=”1″
apc.shm_segments=”4″
apc.shm_size=”128″
apc.num_files_hint=”1024″
apc.ttl=”7200″
apc.user_ttl=”7200″
apc.gc_ttl=”3600″
apc.cache_by_default=”1″
;apc.filters=”"
;apc.mmap_file_mask=”/tmp/apcphp5.XXXXXX”
apc.slam_defense=”0″
apc.file_update_protection=”2″
apc.enable_cli=”0″
apc.max_file_size=”1M”
apc.stat=”1″
apc.write_lock=”1″
apc.report_autofilter=”0″
apc.include_once_override=”0″
apc.rfc1867=”0″
apc.rfc1867_prefix=”upload_”
apc.rfc1867_name=”APC_UPLOAD_PROGRESS”
apc.rfc1867_freq=”0″
apc.localcache=”0″
apc.localcache.size=”512″
apc.coredump_unmap=”0″

Zooming In

These options are documented fairly thoroughly in the APC PHP Manual.

Most of these options can stay at their defaults and provide a pleasant experience but if we have the memory we should probably tweak these to get more out of them:

• apc.shm_segments – The number of chunks to use from /dev/shm.
• apc.shm_size – The size of aforementioned chunks.

These two parameters are the heart of APC and dictate the memory usage you’ll get out of it. If you take segments*size you’ll get the maximum amount of shm space used by APC for the cache. A limitation on shm_size is usually in place through the kernel and to determine what this is you can simply `cat /proc/sys/kernel/shmmax` (remember this command prints out B and APC expects MB).

Further Caching

APC also has an object cache available which can be used for just about anything but the application has to be modified to support this. If you’re using WordPress to host your site an excellent plugin for doing this is W3 Total Cache.

Conclusion

APC can help you get an edge out of your server but, without careful tuning, won’t get you more than the last inch of performance. Play with the memory settings until it isn’t over-utilizing memory (have a larger cache than your code base needs) or under-utilizing memory (having a smaller cache than your code base needs) but don’t forget you don’t want it to run into your swap space.

First things first …

A few common things to possibly turn on, install, or enable:

• APC (Another PHP Cache)
• Memcached

MySQL Optimization

The easiest way (and the quickest) to determine what can be done to optimize MySQL is to use a script called mysqltuner.pl. The really interesting thing about this script is that to get it you simply run the following:

wget mysqltuner.pl

After you’ve gotten this script run it with your perl interpreter:

perl mysqltuner.pl

It will prompt you for your username and password for mysql and then print out a nice report outlining how your mysql daemon has been running. Here’s an example of this output:

>> MySQLTuner 1.0.1 – Major Hayden
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with ‘–help’ for additional options and output filtering
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password:

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.90-log
[OK] Operating on 32-bit architecture with less than 2GB RAM

——– Storage Engine Statistics ——————————————-
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 28M (Tables: 96)
[--] Data in InnoDB tables: 1M (Tables: 55)
[!!] Total fragmented tables: 2

——– Performance Metrics ————————————————-
[--] Up for: 1h 39m 49s (24K q [4.163 qps], 354 conn, TX: 9M, RX: 34M)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 148.6M global + 960.0K per thread (100 max threads)
[OK] Maximum possible memory usage: 242.3M (11% of installed RAM)
[OK] Slow queries: 0% (0/24K)
[OK] Highest usage of available connections: 28% (28/100)
[OK] Key buffer size / total MyISAM indexes: 64.0K/6.3M
[OK] Key buffer hit rate: 99.6% (164K cached / 682 reads)
[OK] Query cache efficiency: 81.3% (14K cached / 17K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 463 sorts)
[!!] Joins performed without indexes: 42
[OK] Temporary tables created on disk: 12% (54 on disk / 417 total)
[OK] Thread cache hit rate: 70% (106 created / 354 connections)
[!!] Table cache hit rate: 2% (48 open / 2K opened)
[OK] Open file limit used: 9% (95/1K)
[OK] Table locks acquired immediately: 98% (7K immediate / 7K locks)
[!!] Connections aborted: 7%
[OK] InnoDB data size / buffer pool: 1.0M/4.0M

——– Recommendations —————————————————–
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours – recommendations may be inaccurate
Adjust your join queries to always utilize indexes
Increase table_cache gradually to avoid file descriptor limits
Your applications are not closing MySQL connections properly
Variables to adjust:
join_buffer_size (> 128.0K, or always use indexes with joins)
table_cache (> 48)

The first thing to notice is that mysql should be running for 24 to 48 hours before you trust the results of this script. The second thing to acknowledge is that it isn’t omniscient. Understanding how these parameters affect your memory footprint and your runtime efficiency are what we’re most interested in though.

Glossary of Parameters

The best source of my.cnf parameters is of course the MySQL documentation, but I’ll clarify a few of the more pertinent items from the above output.

• Maximum Possible Memory – This is not the actual hard limit on your memory usage but a calculated theoretical maximum based on the parameters in your my.cnf and if this is over ~90% you may use your swap more than you want to. After making any changes I suggest you check this to make sure you haven’t over-allocated yourself.
• Highest Usage of Available Connections – This is the historical max connections used by mysql and if it’s at 100% this isn’t a guarantee that you need to increase max_connections but may just indicate that your connections spiked once. That being said I recommend following its advice if you have the memory for it.
• Query Cache Efficiency – The hit rate for the cache on your queries (I hope you have enabled). As with any hash table if you’re not hitting what’s in the top bucket you need to increase the number of buckets.
• Temporary Tables Created on Disk – This can be a finicky parameter to get right and is the first place I would recommend cutting if you don’t have the memory to run mysql like you would want, but if you do have the memory then up this as far as you can within reason.
• Thread Cache Hit Rate – Just like any other cache.
• Table Cache Hit Rate – Just like any other cache.

Fragmented Tables

If you noticed the fragmented tables along with the recommendation to run OPTIMIZE TABLE that’s not something that will typically slow mysql down noticeably with today’s disks, but if you want to defragment them I recommend reading this article on the topic.

Apache Tweaking

The following parameters are usually in httpd.conf unless you run a distribution that organizes apache in an easier to work with manner (my preference of course is Gentoo which stores these parameters in: 00_mpm.conf).

The parameters for tweaking apache’s processes depend on which worker module you have in use. The following is the pertinent sections from my configuration:

#prefork MPM
# This is the default MPM if USE=-threads
#
# MinSpareServers: Minimum number of idle child server processes
# MaxSpareServers: Maximum number of idle child server processes

StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 50
MaxRequestsPerChild 500

# worker MPM
# This is the default MPM if USE=threads
#
# MinSpareThreads: Minimum number of idle threads available to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# ThreadsPerChild: Number of threads created by each child process

StartServers 2
MinSpareThreads 15
MaxSpareThreads 50
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 1000

These parameters are pretty self explanatory but just for completeness’ sake I’ll do a small recap on these here:

* StartServers – The number of servers to have running and handling connections at first launch.
* MinSpareServers – The minimum number of servers to have running not currently handling connections.
* MaxSpareServers – The maximum number of servers to have running not currently handling connections.
* MaxClients – The maximum number of clients that can simultaneously connect to Apache.
* MaxRequestsPerChild – The maximum number of requests that a child will respond to before terminating.

Using these parameters we can control the amount of memory that Apache will consume at the cost of other things we may want to have like lots of connections or many sequential requests.

Conclusion

There are a lot of different things that can be done for a server that seems to be performing less than ideally and I’ve only covered a fraction of things that can happen. As always with this topic the situation will change based on your needs and how your server runs, but at least now you may have a starting point for what to modify after you’ve troubleshooted every other thing that could go wrong.

Make sure you’ve at least looked at the following document: Gentoo Optimization Guide.

Checking Flags

The only real change we want to make to our system are simple things we know will not break it. Thus, we only look at making sure that we have the correct flags for enabling all of the instruction sets that our processor has available.

Checking Processor

grep flags /proc/cpuinfo | grep uniq

This pulls out all of the features of the processor as detected by the kernel. Example output is shown (line breaks added for readability):

alunduil@elijah ~ $ grep flags /proc/cpuinfo | uniq
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall
nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good
extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic
cr8_legacy 3dnowprefetch

Checking Default Flags

gcc does a great job of determining what flags should be set by using the new -march=native flag (which you should have set by this point). Using the following command we can double check that all of the instructions we want enabled are enabled:

gcc -Q -c -v -march=native –help=target | grep disabled

If anything appears in the resulting list, it’s not enabled and should be enabled by adding the appropriate -m flag. For my cpu this results in: -msse3 -m3dnow.

Putting it Together

By adding the original guide with a check for instructions, we allow gcc to utilize the instructions that were specifically created for uses we may have (multimedia, extended math, etc).

The resulting CFLAG variable that I placed in my make.conf from the above discussion:

CFLAGS=”-march=native -O2 -pipe -msse3 -m3dnow”