Jan 182011
 

Introduction

Portage is an amazingly simple and complex piece of technology.  The simplicity in each piece’s ability to do a specific function comes together in a complex package management system that rivals all other forms of package management (at least in my opinion).  Automating updates is something that admins everywhere do out of necessity.  Heck, automating everything is an admin’s life.  Automating portage’s updates is a bit more harrowing than other package management systems but it isn’t impossible.

Problem

As admins we attempt to simplify the work we actually do by writing scripts and programs to do most of our job for us.  It’s often been said that systems admins are the only people whose job description is to remove their job responsibilities.

Portage doesn’t have any default automation for doing nightly or even weekly portage updates but that doesn’t stop the creative from coming up with their own solution.  A simple but elegant solution is to create a small cron script that runs every day.  The problem comes when you want to read the wonderful output of portage (sometimes these messages can guide you when problems are about to occur) to avert disasters.  If the updates are performed from cron, the output will be preserved in an e-mail to the appropriate user but then we have to sift through all of the output at once.  This also doesn’t solve the issue if the updates are performed by another utility such as puppet.  These annoying little changes to the problem require a slightly more elegant solution.

Solution

The solution is to take advantage of portage’s logging specifications.  From the make.conf man file:

  • PORTAGE_ELOG_CLASSES
  • PORTAGE_ELOG_SYSTEM
  • PORTAGE_ELOG_COMMAND
  • PORTAGE_ELOG_MAILURI
  • PORTAGE_ELOG_MAILFROM
  • PORTAGE_ELOG_MAILSUBJECT

Using a combination of these directives in the make.conf file allows us to log the reports from portage to a large number of locations.  If we wanted to simply add mailing output (not the full build output just the messages) we would add the following directives to make.conf:

PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_MAILFROM="portage@alunduil.com"

This simply adds the mailing log utility to portage and specifies that the e-mails come from the address portage@alunduil.com.  Of course, much more complex configurations can be crafted to suit any admins’ needs.

Conclusion

Letting your servers notify you of possible actions is one way of automating maintenance tasks; making maintenance eventually disappear from your task list. By starting with the tasks that are repeated the most frequently, you can quickly free up time for higher level automation and organization which leads to a cleaner and sturdier infrastructure.

 

I’ve often gotten frustrated with my /etc/portage/package.* files when they become massive and full of crud that I don’t even have installed any longer. Because of this I have crafted a simple little utility to clean out packages that are no longer installed and use flags that are no longer valid from these files. This should help trim the cruft from the Gentoo configuration.

The utility, pclean, does all of this and only has one major problem (so far) before I shall call it good enough for a 1.0 release. If you would like to try this little utility; it’s available in my overlay and if you notice any odd behavior please report it to my bugzilla.

Holland on Gentoo

 Linux Guides  Comments Off
Aug 012010
 

Introduction

There is a new king of backups in town, holland. This little framework written in Python allows one to easily backup anything that might need to be converted to a more flat file style before being backed up. Right now there is support for mysql, sqlite, and postgresql but with a little finesse it could potentially support directories as well as databases. This would make not only mysql backups a breeze but LDAP as well.

Progress Update

I have added a preliminary set of ebuilds to my overlay (which could use some code review if anyone is interested) that allows holland to easily be installed on Gentoo systems. So easy in fact that all it takes is `emerge holland`.

It accepts a set of use flags to bring in the “providers” you want to be able to backup for and makes sure that those packages are installed on the system.

Examples

The holland ebuilds have three providers right now:

  • mysql
  • postgresql
  • sqlite

You can install any of these three you want in any combination; it doesn’t care. It will default to installing the mysql but can easily be told not to by placing -mysql in the use flags for holland. Diego Pettenò — Flameeyes mentioned to me that in EAPI 4 we’ll get the cool option of being able to specify one of a set of use flags must be set without forcing the choice but until then we have this slick solution.

There is also lvm support for snapshotting off the database directory before grabbing the database and a myriad of other features I haven’t had a chance to explore yet.

To perform a rudimentary backup after installing holland simply run `holland bk`. This will read the configurations in `/etc/holland` and backup the databases it finds.

Conclusion

The new kid on the block, holland, will make backups of complex databases and directories a breeze. Simply change that cronjob from using mysqldump to calling holland and you’re finished.

Layman Overlay

 Linux Guides  Comments Off
Jan 112010
 

My Overlay

My overlay has been available via subversion for quite some time now, but not in any easy to use format (integration with layman).  Getting this overlay to work with yours must have been a pain if you wanted to try something I had been working on.

Making My Overlay Available

The first thing I had to do to make my overlay talk with layman, let alone get along with layman, was add an xml definition of the overlay somewhere.  I chose the easy to manipulate path of http://www.alunduil.com/svn/portage/trunk/portage/alunduil-overlay.xml.

Adding My Overlay List to Your Layman

To add my overlay(s) to your layman list, simply add the following path to your overlays variable in /etc/layman/layman.cfg: `http://www.alunduil.com/svn/portage/trunk/portage/alunduil-overlay.xml`.

Jul 232009
 

Inroduction

After a while the files in /etc/portage become cluttered with the common, “Let’s try this . . . whoops that didn’t work . . . let’s try this.” No matter how hard you try to keep this clean, you have probably forgotten something along the way. Installing a testing package then removing it because you found a better one later, etc. Well, what’s an easy way to clean these files up and make sure that we minimize their sizes and keep our Gentoo system crisp (or as crisp as we can by just managing these files)? I’ve written some bash one-liners that assist with this and could easily be adapted into a script that automates a lot of the cleaning for you.

In all of these scripts change the /etc/portage/package.use to the file you are interested in cleaning.

Checking for Multiple Occurrences of an Atom Within a File

for atom in $(gawk ‘{print $1}’ /etc/portage/package.use); do [ "$(grep ${atom} /etc/portage/package.use | wc -l)" -gt "1" ] && echo “${atom}”; done

Checking for N Uses of a Use Flag in /etc/portage/package.use

I use this to move frequently used use flags to /etc/make.conf if it seems appropriate.

for flag in $(gawk ‘{print $2}’ /etc/portage/package.use); do [ "$(grep "${flag}" /etc/portage/package.use | wc -l)" -gt "2" ] && echo “${flag}”; done

Checking for Removed Atoms Within a File

for atom in $(gawk ‘{print $1}’ /etc/portage/package.use); do [ "$(portageq match / ${atom} | wc -l)" -lt "1" ] && echo “${atom}”; done

© 2011 Alunduil's Hosting Suffusion theme by Sayontan Sinha