Optimizing Gentoo CFLAGS (Part II)

Linux Guides Comments Off

Feb 202011

Introduction

As I mentioned last time, you want to be comfortable with the existing documentation on CFLAGS before going crazy trying to play with. It also helps to have a good understanding of what you’re doing to the code when you modify these “sacred” parameters.

Alright, now that the CYA is out of the way let’s take this one step further. Last time we talked about figuring out which instruction sets your processor understood and how to figure out what `-m` flags would get those instruction sets into the binaries on your system. This time we’ll be talking about making sure those same flags are in your use flags (just to be sure they’re picked up by the system).

Finding Flag Names

So how do we find the flags that do what we want? Well, as always BASH is our friend and can be used to find this answer in a mostly automated fashion:

. /etc/make.conf && gcc -Q -c -v ${CFLAGS} --help=target | grep enabled

This displays the currently enabled flags based on your CFLAGS parameter and allows us to find which flags have use flags with the following one liner:

gawk '/-m.*/ { print $1 }' | cut -d 'm' --complement -f 1 | xargs -I{} equery h "{}"

Conclusion

Using a little scripting we can extract the necessary information to quickly determine if there are any use flags we should be adding for particular compiler flags that our system might support. With this last level of optimization beyond the previous time’s we should be ready to move on to -O3 (for the daring) and watch our machine’s nose bleed.

Mailing Portage Output

Linux Guides Comments Off

Jan 182011

Introduction

Portage is an amazingly simple and complex piece of technology. The simplicity in each piece’s ability to do a specific function comes together in a complex package management system that rivals all other forms of package management (at least in my opinion). Automating updates is something that admins everywhere do out of necessity. Heck, automating everything is an admin’s life. Automating portage’s updates is a bit more harrowing than other package management systems but it isn’t impossible.

Problem

As admins we attempt to simplify the work we actually do by writing scripts and programs to do most of our job for us. It’s often been said that systems admins are the only people whose job description is to remove their job responsibilities.

Portage doesn’t have any default automation for doing nightly or even weekly portage updates but that doesn’t stop the creative from coming up with their own solution. A simple but elegant solution is to create a small cron script that runs every day. The problem comes when you want to read the wonderful output of portage (sometimes these messages can guide you when problems are about to occur) to avert disasters. If the updates are performed from cron, the output will be preserved in an e-mail to the appropriate user but then we have to sift through all of the output at once. This also doesn’t solve the issue if the updates are performed by another utility such as puppet. These annoying little changes to the problem require a slightly more elegant solution.

Solution

The solution is to take advantage of portage’s logging specifications. From the make.conf man file:

PORTAGE_ELOG_CLASSES
PORTAGE_ELOG_SYSTEM
PORTAGE_ELOG_COMMAND
PORTAGE_ELOG_MAILURI
PORTAGE_ELOG_MAILFROM
PORTAGE_ELOG_MAILSUBJECT

Using a combination of these directives in the make.conf file allows us to log the reports from portage to a large number of locations. If we wanted to simply add mailing output (not the full build output just the messages) we would add the following directives to make.conf:

PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_MAILFROM="portage@alunduil.com"

This simply adds the mailing log utility to portage and specifies that the e-mails come from the address portage@alunduil.com. Of course, much more complex configurations can be crafted to suit any admins’ needs.

Conclusion

Letting your servers notify you of possible actions is one way of automating maintenance tasks; making maintenance eventually disappear from your task list. By starting with the tasks that are repeated the most frequently, you can quickly free up time for higher level automation and organization which leads to a cleaner and sturdier infrastructure.

What is Load Average?

Linux Guides 2 Responses »

Sep 092010

Introduction

Often in administration people make reference to a magic number known as load average but it’s not always clear what this number means (besides being a magical indicator of whether or not the server is going to barf in a terrible fashion soon). This number doesn’t have a whole lot of detail in and of itself (it’s meant to be a quick-glance overall health check afterall).

What is it already?

The load average is an exponentially damped/weighted moving average that is similar to a running n^2 average. This number is calculated not on every clock tick but in accordance with the jiffies the kernel is tracking. Every time slice (which can although shouldn’t be tweaked in the kernel) the average is calculated based on the previous values. This rolling average allows us to keep a minimal amount of information on hand and still have an average since boot.

The load that gets placed into this average algorithm is simply a count of the number of processes in the run queue at that instant. Thus, since processes who are waiting on I/O (those in the D state) need to periodically check in or wait to be woken up by the kernel; these processes can contribute to the count of processes in the run queue. Since these processes aren’t taking CPU time but are taking space in the run queue they can increase the apparent load on the server (thus bringing in the I/O wait of the system into the load average) without raising the CPU usage time.

Conclusion

The load average doesn’t tell you a whole lot of information but coupled with information from `iostat` or your CPU usage you can quickly use it to gauge whether your server is falling over itself or not. Since it reports the number of processes in the run queue it is safe to assume a reasonably efficient use of hardware would dictate you want n+1 as your load average where n is the number of cores in the system.

References

More information (including source code for this calculation) can be found here: http://en.wikipedia.org/wiki/Load_%28computing%29

An excellent reference on how the Linux process life cycle works is Robert Love’s Linux Kernel Development (which recently had a third edition released).

Another reference on the Linux process life cycle is http://wiki.kldp.org/wiki.php/ProcessManagement

Memcached Remote Memory Sharing

Linux Guides 1 Response »

Jun 242010

Introduction

Memcached is a simple key/value memory store that allows values from pretty much any application to be stored in memory for quick retrieval. Most languages have simple APIs that make this very easy from a programming standpoint. This also means that it’s not quite as simple to configure as APC (but similar for modern applications).

Installation and Configuration

To install memcached I recommend using your choice of installation (mine being emerge) and then checking the configuration (again in the case of Gentoo) located at /etc/conf.d/memcached.

Update the file to listen on the interface you want, use the memory you want, etc and then start the daemon up. Now if you’re using memcached on the same server as your apache server most webapps will be able to just turn on memcached and connect and work, but if you’re using memcached and apache on separate servers it’s a little more work.

WordPress and memcached

I’ve already touted the wordpress plugin: W3 Total Cache but unfortunately this plugin doesn’t allow you to easily configure memcached usage through the web interface. It defaults to using the memcached server located at 127.0.0.1:11211 which if we’re using separate servers is less than ideal. The file we need to modify with a simply sed (or by hand if you prefer knowing what’s going on) is %{DOCROOT}/wp-content/w3-total-cache-config.php.

We just need to change the memcached servers to point at our server’s location before enabling it in the interface. To easily accomplish this we can use sed:

sed -i -e ‘s/127.0.0.1:11211/: /g’ %{DOCROOT}/wp-content/w3-total-cache-config.php

Mediawiki and memcached

The other application we can easily setup for memcached is mediawiki. Mediawiki wants us to modify its LocalSettings.php with the following additions:

## Shared memory settings
$wgMainCacheType = CACHE_MEMCACHED;
$wgParserCacheType = CACHE_MEMCACHED;
$wgMessageCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array(“giskard.alunduil.com:11211″);

$wgSessionsInMemcached = true;

These settings simply turn on memcached (CACHE_MEMCACHED) for the various caching areas that mediawiki uses and specifies an array of memcached servers to store that caching information in.

If you do have more than one memcached server configured and available you can use the following syntax to include all of them with various weights:

$wgMemCachedServers = array(array(“host:port”, weight), array(“host2:port2″, weight));

Conclusion

Using memcached can greatly improve the performance of webapps or other applications that don’t need to present the most up to date information. This can reduce traffic to an overloaded database or configuration file or whatever other store you want to place a cache in front of.

APC Caching PHP

Linux Guides Comments Off

May 262010

Introduction

Making PHP run faster is usually pretty easy when you see some of the code that people can write (myself included) but what if we either don’t want to look at their code or don’t want to fix the code? What else can we do to improve the runtime and the load time of PHP web sites (this website included)?

We’ll cover how to use APC (Alternative PHP Cache) to help alleviate some of the issues of a slow web site. Just like last time we optimized MySQL and talked about optimizing Apache we’ll continue on the path to getting PHP applications as fast as we can (without touching a line of code).

Enter APC

APC is PEAR project and a PHP module that after installed and a restart of Apache is already doing some work to make life better. By default APC turns on op-code caching which saves on the compile and execution times of PHP. The configuration for this module is pretty straight forward, but let’s look at a copy of the one I’m using:

; /etc/php/apache2-php5/ext-active/apc.ini (On Gentoo anyways …)
extension=apc.so
apc.enabled=”1″
apc.shm_segments=”4″
apc.shm_size=”128″
apc.num_files_hint=”1024″
apc.ttl=”7200″
apc.user_ttl=”7200″
apc.gc_ttl=”3600″
apc.cache_by_default=”1″
;apc.filters=”"
;apc.mmap_file_mask=”/tmp/apcphp5.XXXXXX”
apc.slam_defense=”0″
apc.file_update_protection=”2″
apc.enable_cli=”0″
apc.max_file_size=”1M”
apc.stat=”1″
apc.write_lock=”1″
apc.report_autofilter=”0″
apc.include_once_override=”0″
apc.rfc1867=”0″
apc.rfc1867_prefix=”upload_”
apc.rfc1867_name=”APC_UPLOAD_PROGRESS”
apc.rfc1867_freq=”0″
apc.localcache=”0″
apc.localcache.size=”512″
apc.coredump_unmap=”0″

Zooming In

These options are documented fairly thoroughly in the APC PHP Manual.

Most of these options can stay at their defaults and provide a pleasant experience but if we have the memory we should probably tweak these to get more out of them:

apc.shm_segments – The number of chunks to use from /dev/shm.
apc.shm_size – The size of aforementioned chunks.

These two parameters are the heart of APC and dictate the memory usage you’ll get out of it. If you take segments*size you’ll get the maximum amount of shm space used by APC for the cache. A limitation on shm_size is usually in place through the kernel and to determine what this is you can simply `cat /proc/sys/kernel/shmmax` (remember this command prints out B and APC expects MB).

Further Caching

APC also has an object cache available which can be used for just about anything but the application has to be modified to support this. If you’re using WordPress to host your site an excellent plugin for doing this is W3 Total Cache.

Conclusion

APC can help you get an edge out of your server but, without careful tuning, won’t get you more than the last inch of performance. Play with the memory settings until it isn’t over-utilizing memory (have a larger cache than your code base needs) or under-utilizing memory (having a smaller cache than your code base needs) but don’t forget you don’t want it to run into your swap space.

Older Entries

February 2012
S	M	T	W	T	F	S
« May
	1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29

Suffusion theme by Sayontan Sinha

Alunduil's Hosting

Optimizing Gentoo CFLAGS (Part II)

Introduction

Finding Flag Names

Conclusion

Mailing Portage Output

Introduction

Problem

Solution

Conclusion

What is Load Average?

Introduction

What is it already?

Conclusion

References

Memcached Remote Memory Sharing

Introduction

Installation and Configuration

WordPress and memcached

Mediawiki and memcached

Conclusion

APC Caching PHP

Introduction

Enter APC

Zooming In

Further Caching

Conclusion

Archives

Alunduil's Hosting

Optimizing Gentoo CFLAGS (Part II)

Introduction

Finding Flag Names

Conclusion

Mailing Portage Output

Introduction

Problem

Solution

Conclusion

What is Load Average?

Introduction

What is it already?

Conclusion

References

Memcached Remote Memory Sharing

Introduction

Installation and Configuration

WordPress and memcached

Mediawiki and memcached

Conclusion

APC Caching PHP

Introduction

Enter APC

Zooming In

Further Caching

Conclusion

Common Tags

Archives