Alunduil's Hosting http://www.alunduil.com Gentoo Hackery and Other Fun ... Sun, 01 May 2011 22:00:34 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Apache Rewrite Cheat Sheet http://www.alunduil.com/2011/05/01/apache-rewrite-cheat-sheet/ http://www.alunduil.com/2011/05/01/apache-rewrite-cheat-sheet/#comments Sun, 01 May 2011 22:00:34 +0000 Alex Brandt http://www.alunduil.com/?p=410 [...]]]>

Introduction

I was recently requested to document common apache rewrite pitfalls and examples and crafted the following document as a response.  It is intended as a two page document (Rewrite Cheat Sheet) where the first page is a reference guide of commonly used rewrite variables and flags and the second page is a short list of examples, gotchas, and troubleshooting advice.

Rewrite Cheat Sheet

Common Variables

HTTP Headers

  • HTTP_USER_AGENT
  • HTTP_REFERER
  • HTTP_COOKIE
  • HTTP_FORWARDED
  • HTTP_HOST
  • HTTP_PROXY_CONNECTION
  • HTTP_ACCEPT

connection & request

  • REMOTE_ADDR
  • REMOTE_HOST
  • REMOTE_PORT
  • REMOTE_USER
  • REMOTE_IDENT
  • REQUEST_METHOD
  • SCRIPT_FILENAME
  • PATH_INFO
  • QUERY_STRING
  • AUTH_TYPE

server internals

  • DOCUMENT_ROOT
  • SERVER_ADMIN
  • SERVER_NAME
  • SERVER_ADDR
  • SERVER_PORT
  • SERVER_PROTOCOL
  • SERVER_SOFTWARE

date and time

  • TIME_YEAR
  • TIME_MON
  • TIME_DAY
  • TIME_HOUR
  • TIME_MIN
  • TIME_SEC
  • TIME_WDAY
  • TIME

specials

  • API_VERSION
  • THE_REQUEST
  • REQUEST_URI
  • REQUEST_FILENAME
  • IS_SUBREQ
  • HTTPS

Variable Descriptions

  • REQUEST_FILENAME
    • The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as when used in virtual host context, the same value as REQUEST_URI.
  • HTTPS
    • Will contain the text “on” if the connection is using SSL/TLS, or “off” otherwise. (This variable can be safely used regardless of whether or not mod_ssl is loaded).

Flag Descriptions

  • ‘nocase|NC‘ (no case) :: This makes the test case-insensitive – differences between ‘A-Z’ and ‘a-z’ are ignored, both in the expanded TestString and the CondPattern.
  • ‘ornext|OR‘ (or next condition) (RewriteCond Only) :: Use this to combine rule conditions with a local OR instead of the implicit AND.
  • ‘last|L’ :: Stop the rewriting process immediately and don’t apply any more rules.
  • ‘proxy|P’ :: Force the substitution URL to be internally sent as a proxy request.
  • ‘qsappend|QSA‘ :: Appends any query string created in the rewrite target to any query string that was in the original request URL.
  • ‘redirect|R[=code]‘ :: Forces an external redirect, optionally with the specified HTTP status code.
  • ‘forbidden|F‘ :: Returns a 403 FORBIDDEN response to the client browser.

Good Examples

  • Adding www to all requests
    • RewriteCond %{HTTP_HOST} !^www [NC]
    • RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]
  • Forcing all requests to HTTPS
    • RewriteCond %{HTTPS} off
    • RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]
  • Redirect a specific subweb to another domain
    • RewriteRule ^/?subweb/(.*) http://other.example.com/$1 [R,L,QSA]
  • Block specific IPs from access
    • RewriteCond %{REMOTE_ADDR} ^127\.0\.0
    • RewriteRule ^ – [F,L]
  • Creating a filesystem alias with modrewrite
    • RewriteRule ^/?alias/(.*) /var/www/vhosts/$1/httpdocs/$1 [L,R]
  • A condition to stop CMS software from over-riding fullstatus (added before the offending rewriterule)
    • RewriteCond ${REQUEST_URI} !server-status [NC]

Bad Examples

  • Recursive rewrite
    • RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [R,L,QSA]

Gotchas

  • Some rewrites may conflict with existing rewrites provided by many CMS packages (wordpress, drupal, joomla, etc). Check for any existing rewrites in a .htaccess file.
  • Rewriterule and rewritecond can only be used in the following contexts: server config, virtual host, directory, .htaccess

Common Troubleshooting

  • Enable the rewrite logs with RewriteLog and RewriteLevel
    • RewriteLog <file path>
    • RewriteLogLevel 3 # ranges from 0 to 9
  • Check your regular expressions against a PCRE checker (they are very bountiful on the Internet [http://tinyurl.com/3hop7xu]).
  • Utilize curl to test redirects (R), `curl -I example.com`
]]> http://www.alunduil.com/2011/05/01/apache-rewrite-cheat-sheet/feed/ 0 Android Battery Life http://www.alunduil.com/2011/03/06/android-battery-life/ http://www.alunduil.com/2011/03/06/android-battery-life/#comments Sun, 06 Mar 2011 23:00:25 +0000 Alex Brandt http://www.alunduil.com/?p=400 [...]]]> Recently, I’ve stumbled across an application in the android market, juice defender, that does exactly what it advertises. The application extends battery life through automated micromanaging of peripherals and software. This combined with another application, juice plotter, allows one to actually experiment with different battery preserving techniques (or different ROMs for the rooted phones).

More specifically, the application behaves much like any other power management software in other Linux installations. It controls the frequency scaling of the processor (if the phone is rooted), it controls networking interfaces and screens to help conserve power. All of this functionality can easily be configured and modified for your desired power settings, but the defaults will provide a significant boost to battery life. This is one more useful utility in a long list that allows us to get the most out of our phones.

]]> http://www.alunduil.com/2011/03/06/android-battery-life/feed/ 0 Android Automation http://www.alunduil.com/2011/02/27/android-automation/ http://www.alunduil.com/2011/02/27/android-automation/#comments Sun, 27 Feb 2011 23:00:41 +0000 Alex Brandt http://www.alunduil.com/?p=394 [...]]]> Recently I’ve found the wonderful Android application, profile valet.  This little application handles location and time based profiles for various settings on your Android device.  It can control things like sound, wifi and bluetooth based on time schedules or location (GPS backed).  Using a utility like this can help prevent the common, “I forgot to unsilence my phone,” resulting in missed calls, texts or emails.  It can also be used to control silencing your phone for certain events.  Simply setup a profile for various movie theaters to silence your phone automatically when you enter that area.

By automating the little things in life we can open up our minds to less stress and turn our attention to more fun or important matters.  I know offloading thought has been pushed by the “Getting Things Done” paradigm but it can be expanded with the smartphone.  It can be expanded to allow our devices to be the tool they were designed to be.

]]> http://www.alunduil.com/2011/02/27/android-automation/feed/ 0 Optimizing Gentoo CFLAGS (Part II) http://www.alunduil.com/2011/02/20/optimizing-gentoo-cflags-part-ii/ http://www.alunduil.com/2011/02/20/optimizing-gentoo-cflags-part-ii/#comments Sun, 20 Feb 2011 23:00:46 +0000 Alex Brandt http://www.alunduil.com/?p=205 [...]]]> Introduction

As I mentioned last time, you want to be comfortable with the existing documentation on CFLAGS before going crazy trying to play with. It also helps to have a good understanding of what you’re doing to the code when you modify these “sacred” parameters.

Alright, now that the CYA is out of the way let’s take this one step further. Last time we talked about figuring out which instruction sets your processor understood and how to figure out what `-m` flags would get those instruction sets into the binaries on your system. This time we’ll be talking about making sure those same flags are in your use flags (just to be sure they’re picked up by the system).

Finding Flag Names

So how do we find the flags that do what we want? Well, as always BASH is our friend and can be used to find this answer in a mostly automated fashion:

. /etc/make.conf && gcc -Q -c -v ${CFLAGS} --help=target | grep enabled

This displays the currently enabled flags based on your CFLAGS parameter and allows us to find which flags have use flags with the following one liner:

gawk '/-m.*/ { print $1 }' | cut -d 'm' --complement -f 1 | xargs -I{} equery h "{}"

Conclusion

Using a little scripting we can extract the necessary information to quickly determine if there are any use flags we should be adding for particular compiler flags that our system might support.  With this last level of optimization beyond the previous time’s we should be ready to move on to -O3 (for the daring) and watch our machine’s nose bleed.

]]> http://www.alunduil.com/2011/02/20/optimizing-gentoo-cflags-part-ii/feed/ 0 What are Snapshots? http://www.alunduil.com/2011/02/15/what-are-snapshots/ http://www.alunduil.com/2011/02/15/what-are-snapshots/#comments Tue, 15 Feb 2011 22:41:27 +0000 Alex Brandt http://www.alunduil.com/?p=373 [...]]]> Introduction

It seems that disk snapshots have become a hot topic and a confusing topic.  I intend to simply outline what snapshots look like in terms of the lower layers of abstraction and nothing more.  Snapshots are built into things like LVM, SAN, etc but I will not be covering those technologies.  Instead, what I intend to cover is the abstractions ranging from the disk device to the snapshot.

The Disk

A hard disk is typically chopped up into pieces called partitions (and this is the unit I’ll use as my foundation).  These logical separations of the blocks on a disk allow us to make sure that disks are not over-allocated for one particular purpose or lose access to a system if one partition fills up, etc.  Partitioning is only the beginning.   We are able to take the idea of partitions and extend the idea of a contiguous region of blocks to the strange concept of snapshots as well.

Writing to Disk

Using our simplified view of a disk (a contiguous region of blocks) we can see that the simplest way to write data (ignoring filesystems) is to simply take our data one block at a time and place it on the disk.  This is great until we need to remove blocks or update blocks (which is probably why we only use this type of writing for tapes).  We’ll have to add some logic to this view of disks to handle the complexities of files but for all intensive purposes, disks are simply groups of blocks available for writing and groups of blocks that are already used.  If we want a consistent view of the data at any point it’s simply a matter of making sure nothing is writing to the disk (partition) we’re interested in.  This is where snapshots can help us out.

Snapshots

A snapshot creates another region of blocks we can write to.  Let me start that again … when you create a snapshot of some disk (snapshots are not standalone by any means) the following happens:

  • It reserves space for a changelog (or writes that would happen to the disk) in the difference disk
  • It creates a new way to access the original disk via the snapshot name (which refers to the base disk)
  • It begins sending all writes to the reserved space for the changelog (depicted in the figure below)

Thus, we have the following situation on the disk (again very simplified):

Deleting Snapshots

Since a snapshot isn’t a true partition what happens when one of these gets removed?  The obvious is that the snapshot name gets removed so it’s no longer accessible, but that’s not going to help rectify the two regions of data we now have on the disk.  This is handled by making the difference disk a literal list of differences.  I find it best to think of it (even though it may not actually be implemented in this fashion) as a queue of block changes or a transaction log for writes to the disk.  With this visualization of the process it makes sense as to what happens when the snapshot is removed:

  • The reference to the base disk is removed
  • The difference disk is replayed onto the base disk (applying all recorded changes)

That’s all there is to it.  Everything continues as if the snapshot exists until it has been fully replayed and then disk access resumes as normal without the snapshot.

Conclusion

Snapshots provide a convenient way to access a frozen image of a disk which is perfect for backups or point in time restores of data.  This is typically used via LVM, virtualbox, SANs, etc and has far-reaching impacts that allow system administration to be easier.

]]> http://www.alunduil.com/2011/02/15/what-are-snapshots/feed/ 0 Backing up PostgreSQL with Holland http://www.alunduil.com/2011/02/02/backing-up-postgresql-with-holland/ http://www.alunduil.com/2011/02/02/backing-up-postgresql-with-holland/#comments Thu, 03 Feb 2011 01:10:40 +0000 Alex Brandt http://www.alunduil.com/?p=294 [...]]]> Introduction

Backups are a subject I return to semi-frequently with a passion to never be in an “oh shit” scenario.  Last time I built my backup system, bacula with a postgresql DB backend, I determined that I would move to a common database backup script for all of my databases.  Holland fit this bill perfectly with support for postgres, sqlite and mysql.  This allows one command to backup all of my databases on all of my servers and subsequently creates a much simpler bacula configuration (the database job is defined the same as the catalog job).

The Solution

The problem I had when configuring holland to backup postgresql is that there was no example configuration file.  It wasn’t hard to craft a working default postgres configuration and the following is what I came up with (/etc/holland/backupsets/default.conf:

[holland:backup]
plugin = pgdump
backups-to-keep = 1
auto-purge-failures = yes
purge-policy = after-backup
estimated-size-factor = 1.0

[pgdump]
role = postgres

[pgauth]
username = postgres

Conclusion

Setting up holland to backup databases is incredibly easy and flexible.  By having a common backup solution for all databases other configurations become easier and processes can be streamlined.

]]> http://www.alunduil.com/2011/02/02/backing-up-postgresql-with-holland/feed/ 0 Paging Problem Plague http://www.alunduil.com/2011/02/01/paging-problem-plague/ http://www.alunduil.com/2011/02/01/paging-problem-plague/#comments Tue, 01 Feb 2011 21:21:00 +0000 Alex Brandt http://www.alunduil.com/?p=300 [...]]]> In the last couple of months I have been plagued by a problem with the system that hosts this website.  A nice kernel panic followed by a random uptime and a crash.  This is a bit difficult to swallow when the server is around 1200 miles away but a reboot and away it goes again.

The actual error is show here:

Feb 01 14:13:27 [kernel] kernel BUG at mm/filemap.c:128!
Feb 01 14:13:27 [kernel] Modules linked in:
Feb 01 14:13:27 [kernel] Pid: 13604, comm: apache2 Not tainted 2.6.36-hardened-r6 #1 0K8980/Dimension 3000
Feb 01 14:13:27 [kernel] EIP: 0060:[<c105af4f>] EFLAGS: 00010046 CPU: 0
Feb 01 14:13:27 [kernel] EIP is at __remove_from_page_cache+0x44/0x91
Feb 01 14:13:27 [kernel] EAX: 00000000 EBX: c16acf40 ECX: 00000015 EDX: 00000015
Feb 01 14:13:27 [kernel] ESI: de877d7c EDI: ffffffff EBP: 00000015 ESP: dc4abe28
Feb 01 14:13:27 [kernel]  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
Feb 01 14:13:27 [kernel]  de877d7c c16acf40 c105afbc c16acf40 de877d7c c1061079 c16acf40 00000000
Feb 01 14:13:27 [kernel] <0> c1061123 de877d7c 00000015 00000006 ffffffff 00000000 0000000e 00000000
Feb 01 14:13:27 [kernel] <0> c1641f40 c1641f60 c1641f80 c1641fa0 c173ecc0 c1695de0 c16acf40 c15d2f00
Feb 01 14:13:27 [kernel]  [<c105afbc>] ? remove_from_page_cache+0x20/0x27
Feb 01 14:13:27 [kernel]  [<c1061079>] ? truncate_inode_page+0x6c/0x7d
Feb 01 14:13:27 [kernel]  [<c1061123>] ? truncate_inode_pages_range+0x99/0x23a
Feb 01 14:13:27 [kernel]  [<c10612cd>] ? truncate_inode_pages+0x9/0xc
Feb 01 14:13:27 [kernel]  [<c10fa452>] ? ext4_evict_inode+0x83/0x265
Feb 01 14:13:27 [kernel]  [<c108dab2>] ? evict+0x17/0x7b
Feb 01 14:13:27 [kernel]  [<c108e211>] ? iput+0x182/0x1df
Feb 01 14:13:27 [kernel]  [<c108b58e>] ? d_kill+0x2a/0x43
Feb 01 14:13:27 [kernel]  [<c108c292>] ? dput+0xf3/0xfb
Feb 01 14:13:27 [kernel]  [<c107e289>] ? fput+0x191/0x1b3
Feb 01 14:13:27 [kernel]  [<c106e14f>] ? remove_vma+0x34/0x52
Feb 01 14:13:27 [kernel]  [<c106f26a>] ? __do_munmap+0x257/0x2a8
Feb 01 14:13:27 [kernel]  [<c106f335>] ? sys_munmap+0x49/0x60
Feb 01 14:13:27 [kernel]  [<c1378005>] ? syscall_call+0x7/0xb
Feb 01 14:13:27 [kernel] ---[ end trace 4598df0f375c22c4 ]---
Feb 01 14:13:27 [kernel] kernel BUG at mm/filemap.c:128!Feb 01 14:13:27 [kernel] Modules linked in:Feb 01 14:13:27 [kernel] Pid: 13604, comm: apache2 Not tainted 2.6.36-hardened-r6 #1 0K8980/Dimension 3000               Feb 01 14:13:27 [kernel] EIP: 0060:[<c105af4f>] EFLAGS: 00010046 CPU: 0Feb 01 14:13:27 [kernel] EIP is at __remove_from_page_cache+0x44/0x91Feb 01 14:13:27 [kernel] EAX: 00000000 EBX: c16acf40 ECX: 00000015 EDX: 00000015Feb 01 14:13:27 [kernel] ESI: de877d7c EDI: ffffffff EBP: 00000015 ESP: dc4abe28Feb 01 14:13:27 [kernel]  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068Feb 01 14:13:27 [kernel]  de877d7c c16acf40 c105afbc c16acf40 de877d7c c1061079 c16acf40 00000000Feb 01 14:13:27 [kernel] <0> c1061123 de877d7c 00000015 00000006 ffffffff 00000000 0000000e 00000000Feb 01 14:13:27 [kernel] <0> c1641f40 c1641f60 c1641f80 c1641fa0 c173ecc0 c1695de0 c16acf40 c15d2f00Feb 01 14:13:27 [kernel]  [<c105afbc>] ? remove_from_page_cache+0x20/0x27Feb 01 14:13:27 [kernel]  [<c1061079>] ? truncate_inode_page+0x6c/0x7dFeb 01 14:13:27 [kernel]  [<c1061123>] ? truncate_inode_pages_range+0x99/0x23aFeb 01 14:13:27 [kernel]  [<c10612cd>] ? truncate_inode_pages+0x9/0xcFeb 01 14:13:27 [kernel]  [<c10fa452>] ? ext4_evict_inode+0x83/0x265Feb 01 14:13:27 [kernel]  [<c108dab2>] ? evict+0x17/0x7bFeb 01 14:13:27 [kernel]  [<c108e211>] ? iput+0x182/0x1dfFeb 01 14:13:27 [kernel]  [<c108b58e>] ? d_kill+0x2a/0x43Feb 01 14:13:27 [kernel]  [<c108c292>] ? dput+0xf3/0xfbFeb 01 14:13:27 [kernel]  [<c107e289>] ? fput+0x191/0x1b3Feb 01 14:13:27 [kernel]  [<c106e14f>] ? remove_vma+0x34/0x52Feb 01 14:13:27 [kernel]  [<c106f26a>] ? __do_munmap+0x257/0x2a8Feb 01 14:13:27 [kernel]  [<c106f335>] ? sys_munmap+0x49/0x60Feb 01 14:13:27 [kernel]  [<c1378005>] ? syscall_call+0x7/0xbFeb 01 14:13:27 [kernel] ---[ end trace 4598df0f375c22c4 ]---

The uprecords for this machine:

     #               Uptime | System                                     Boot up
----------------------------+---------------------------------------------------
     1    24 days, 20:31:52 | Linux 2.6.32-hardened-r2  Thu Nov 25 20:58:34 2010
     2    18 days, 09:10:28 | Linux 2.6.36-hardened-r6  Sat Jan  8 12:31:38 2011
     3    15 days, 12:41:25 | Linux 2.6.32-hardened-r9  Fri Oct  8 09:48:07 2010
     4    14 days, 03:43:24 | Linux 2.6.32-hardened-r2  Wed Nov  3 11:29:55 2010
     5     9 days, 19:44:59 | Linux 2.6.32-hardened-r2  Sun Oct 24 15:14:10 2010
     6     7 days, 21:49:22 | Linux 2.6.32-hardened-r2  Mon Dec 20 17:31:43 2010
     7     7 days, 17:34:24 | Linux 2.6.36-hardened-r6  Tue Dec 28 15:58:45 2010
     8     7 days, 08:53:26 | Linux 2.6.32-hardened-r2  Thu Nov 18 11:55:57 2010
     9     5 days, 16:58:46 | Linux 2.6.36-hardened-r6  Wed Jan 26 21:43:51 2011
    10     3 days, 00:09:43 | Linux 2.6.36-hardened-r6  Wed Jan  5 12:17:33 2011
----------------------------+---------------------------------------------------
->  13     0 days, 00:14:46 | Linux 2.6.36-hardened-r6  Tue Feb  1 15:02:26 2011
----------------------------+---------------------------------------------------
1up in     0 days, 00:40:31 | at                        Tue Feb  1 15:57:42 2011
t10 in     2 days, 23:54:58 | at                        Fri Feb  4 15:12:09 2011
no1 in    24 days, 20:17:07 | at                        Sat Feb 26 11:34:18 2011
    up   115 days, 09:00:18 | since                     Fri Oct  8 09:48:07 2010
  down     0 days, 21:28:47 | since                     Fri Oct  8 09:48:07 2010
   %up               99.230 | since                     Fri Oct  8 09:48:07 2010

In conclusion, I’m suspecting a hardware issue but if anyone has any ideas as to what might cause this type of problem I’d be more than happy to hear possible causes and solutions until I get a replacement server built out.

]]> http://www.alunduil.com/2011/02/01/paging-problem-plague/feed/ 0 Special Keys in zsh http://www.alunduil.com/2011/01/27/special-keys-in-zsh/ http://www.alunduil.com/2011/01/27/special-keys-in-zsh/#comments Fri, 28 Jan 2011 02:26:40 +0000 Alex Brandt http://www.alunduil.com/?p=288 [...]]]> I recently made the plunge and began using zsh in lieu of bash.  I’ve not regretted the decision in the slightest but there have been minor annoyances that needed to be dealt with.  The simplest annoyance was the special keys (delete, home, page up, page down, etc).  The solution was quite simple and elegant but not completely obvious.

The bindings usually are read from the file `/etc/inputrc`by bash but zsh does not do this by default.  There are probably more elegant solutions but a quick brute force solution is to create a bindkeys file out of inputrc:

gawk '$1 ~ /.*:/ { print "bindkey",$1,$2 }' /etc/inputrc | \
sed -e 's/://g' > ~/.zshrc-bindkeys

Once this file has been crafted it’s simply a matter of invoking it from your .zshrc with `source ~/.zshrc-bindkeys`.

]]> http://www.alunduil.com/2011/01/27/special-keys-in-zsh/feed/ 0 Mailing Portage Output http://www.alunduil.com/2011/01/18/mailing-portage-output/ http://www.alunduil.com/2011/01/18/mailing-portage-output/#comments Tue, 18 Jan 2011 23:49:06 +0000 Alex Brandt http://www.alunduil.com/?p=282 [...]]]> Introduction

Portage is an amazingly simple and complex piece of technology.  The simplicity in each piece’s ability to do a specific function comes together in a complex package management system that rivals all other forms of package management (at least in my opinion).  Automating updates is something that admins everywhere do out of necessity.  Heck, automating everything is an admin’s life.  Automating portage’s updates is a bit more harrowing than other package management systems but it isn’t impossible.

Problem

As admins we attempt to simplify the work we actually do by writing scripts and programs to do most of our job for us.  It’s often been said that systems admins are the only people whose job description is to remove their job responsibilities.

Portage doesn’t have any default automation for doing nightly or even weekly portage updates but that doesn’t stop the creative from coming up with their own solution.  A simple but elegant solution is to create a small cron script that runs every day.  The problem comes when you want to read the wonderful output of portage (sometimes these messages can guide you when problems are about to occur) to avert disasters.  If the updates are performed from cron, the output will be preserved in an e-mail to the appropriate user but then we have to sift through all of the output at once.  This also doesn’t solve the issue if the updates are performed by another utility such as puppet.  These annoying little changes to the problem require a slightly more elegant solution.

Solution

The solution is to take advantage of portage’s logging specifications.  From the make.conf man file:

  • PORTAGE_ELOG_CLASSES
  • PORTAGE_ELOG_SYSTEM
  • PORTAGE_ELOG_COMMAND
  • PORTAGE_ELOG_MAILURI
  • PORTAGE_ELOG_MAILFROM
  • PORTAGE_ELOG_MAILSUBJECT

Using a combination of these directives in the make.conf file allows us to log the reports from portage to a large number of locations.  If we wanted to simply add mailing output (not the full build output just the messages) we would add the following directives to make.conf:

PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_MAILFROM="portage@alunduil.com"

This simply adds the mailing log utility to portage and specifies that the e-mails come from the address portage@alunduil.com.  Of course, much more complex configurations can be crafted to suit any admins’ needs.

Conclusion

Letting your servers notify you of possible actions is one way of automating maintenance tasks; making maintenance eventually disappear from your task list. By starting with the tasks that are repeated the most frequently, you can quickly free up time for higher level automation and organization which leads to a cleaner and sturdier infrastructure.

]]> http://www.alunduil.com/2011/01/18/mailing-portage-output/feed/ 0 Small Interpreters Using Python http://www.alunduil.com/2010/12/15/small-interpreters-using-python/ http://www.alunduil.com/2010/12/15/small-interpreters-using-python/#comments Thu, 16 Dec 2010 01:59:41 +0000 Alex Brandt http://www.alunduil.com/?p=254 [...]]]> Introduction

I know this doesn’t come up nearly as often as other topics but writing an interpreter (a.k.a. a compiler) can be a very useful concept depending on what exactly you need to do with the input to your program. For example, what if we had a fairly strict data store but we for some reason wanted to access it using something like SQL? We’d have to parse the SQL statements and then find a way to mogrify it until we had something that would allow us to get the data we wanted. There is an easier way involving a little parsing theory. For the purposes of this discussion I’m assuming you are at least somewhat familiar with Context Free Grammars.

Creating the Grammar

The first step to creating a parser (especially when using a parser generator) is to find or craft a definition for the grammar. The grammar we’ll use as an example is the expression grammar from tiny basic. This simple grammar is safe for LR or LL parsing (which is important if you look at a common definition of a language like SQL).

Tiny Basic Expression Grammar

expression ::= (+|-|ε) term ((+|-) term)*
term ::= factor ((*|/) factor)*
factor ::= number | (expression)

Enter pyparsing

To create a very simple and extensible LL parser I’ve recently stumbled upon pyparsing. This simple four production grammar expands to the following pyparsing implementation:

[sourcecode language="python" wraplines="false"]
expr = Forward()
factor = ( Word(nums) | Group(Suppress(‘(‘) + expr + Suppress(‘)’)) )
term = Group(factor + ZeroOrMore((Literal(‘*’)|Literal(‘/’)) + factor))
expr << Group(Optional(Literal(‘-’)|Literal(‘+’)) + term + ZeroOrMore((Literal(‘-’)|Literal(‘+’)) + term))
[/sourcecode]

This allows us to turn sentences such as ‘5+5*6/3-(47+56)*34‘ into an easy to work with list such as: ‘[[['5'], ‘+’, ['5', '*', '6', '/', '3'], ‘-’, [[[['47'], ‘+’, ['56']]], ‘*’, ’34′]]]‘. There are probably improvements that could be made to this parser so that it auto-collapses expressions and other fun handlers but for the purposes of a simple grammar this will suffice.

Calling the grammar after defining it is a very simple process: `expr.parseString(’5+5*6/3-(47+56)*34′)`.

Testing Parsers an Easier Way

Sure unit testing should be done (and parsers lend themselves to unit tests very well) but there’s something satisfying about seeing your sentences get parsed out in real time. The obvious answer is, “create a mini-shell like environment.” Python also makes this process extremely simple and only requires a few lines of code to get a basically functional shell for your parser (complete with history):

[sourcecode language="python" wraplines="false"]
import rlcompleter
import readline
import so

if not os.access(".history", os.F_OK): open(".history", "w").close()
readline.read_history_file(".history")
buffer = ""

while True:
try: line = raw_input(pycolorize.light_blue("BASIC$ "))
except EOFError:
readline.write_history_file(".history")
print
break

if line.lower() == "exit" or line.lower() == "quit":
readline.write_history_file(".history")
break

buffer += line
result = ACTION_ON_BUFFER
buffer = ""
[/sourcecode]

Putting It Together

The complete script for reference purposes:

[sourcecode language="python" wraplines="false"]
import rlcompleter
import readline
import os

from pyparsing import *
import pprint

if not os.access(".history", os.F_OK): open(".history", "w").close()
readline.read_history_file(".history")

try:
import pycolorize
except:
sys.path.append(os.path.join(os.path.dirname(__file__), "vendor", "pycolorize"))
import pycolorize

class ExpressionParser(object):
def __init__(self):
self._expr = Forward()
factor = ( Word(nums) | Group(Suppress(‘(‘) + self._expr + Suppress(‘)’)) )
term = Group(factor + ZeroOrMore((Literal(‘*’)|Literal(‘/’)) + factor))
self._expr << Group(Optional(Literal(‘-’)|Literal(‘+’)) + term + ZeroOrMore((Literal(‘-’)|Literal(‘+’)) + term))

def _calculate(self, l):
while any([ isinstance(x, list) for x in l]):
for n,i in enumerate(l):
if isinstance(i, list): l[n] = self._calculate(i)
return str(eval(" ".join(l)))

def __call__(self, string):
return self._calculate(self._expr.parseString(string).asList())

buffer = ""

print pycolorize.green("Enter your SQL commands to tokenize:")
print pycolorize.green("Enter a blank line to exit.")

while True:
try: line = raw_input(pycolorize.light_blue("BASIC$ "))
except EOFError:
readline.write_history_file(".history")
print
break

if line.lower() == "exit" or line.lower() == "quit":
readline.write_history_file(".history")
break

buffer += line
result = None
try: result = ExpressionParser()(buffer)
except ParseBaseException, e:
buffer = ""
pycolorize.error(e.line)
pycolorize.error(" "*(e.col – 1) + "^")
pycolorize.error(str(e))
continue
pycolorize.status("Result: %s", result)
buffer = ""
[/sourcecode]

Conclusion

Writing LL parsers is a breeze with pyparsing but it must be kept in mind that any grammar that has any left recursion will cause errors that may take some time to find or remove. Other parser generators (for C and C++) include bison and lemon. These parser generators are LR parser generators.

By coupling the parser with a small CLI quick checks on new features to the grammar (and by extension the parser) can be a breeze. Putting this all together with unit tests and proper grammar analysis can lead to a well written and extensible language to be used for whatever purpose you may have in mind.

]]> http://www.alunduil.com/2010/12/15/small-interpreters-using-python/feed/ 0