Pathfinder Development » David Digioia

Little Lost MySQL Lambs

David Digioia — Wed, 31 May 2006 16:00:00 +0000

If you're a DBA or a Sysadmin, your flock is probably larger than you think it is. By this, I mean, your organization, or your department, probably has more database servers than you think it does. Probably a LOT more.

Today's post aims to help you locate these "lost lambs" and bring them back into the fold. I am using the example of MySQL, but many of these strategies apply to other database software.

Like any company that manages infrastructure for clients, we often run into emergency cases where a client database needs to be restored. Sometimes we're not sure how they are being backed up, or even IF they are being backed up at all. Why? Sometimes databases magically drift from experimental to production. This points to governance issues of how and when applications and environments should be promoted to production, but it still leaves the practical question of "what do I do now with all of my lost databases?"

In many cases we end up auditing ALL of the databases at a client's sites. What we find often surprises the client. In one case we found more than 12 undocumented instances of MySQL! Many of these had sprung up in the past few months.

MySQL is a very capable, free, open source database server. These qualities make it very easy to bring up new instances of MySQL quickly. The downside of this is that they are often not tracked, managed, secured, or backed up as carefully as they should be.

If you find yourself in a similar situation, the first thing you should do is audit your database servers. Count them, but also determine the following information about them. (It may be helpful to use a table or spreadsheet.)

hostname

type (standalone, replicated, cluster)

IP address(es)

purpose(s)

criticality of the database server (impact if this server goes down)

criticality of the data (impact if amount x of this data is lost)

applications it depends on

applications that depend on it

security policy regarding the data and the server

privacy policy regarding the data

table engines in use (InnoDB, MyISAM, etc.)

current primary MySQL backup method

any changes proposed for primary MySQL backup method

proposed supplemental MySQL backup method

filesystem backup method

bare-metal (disaster recovery) backup/restore method

date of last MySQL restore test

date of last filesystem backup test

date of last bare-metal restore test

Please notice that we list filesystem and bare-metal backups and MySQL backups separately. All are important, but in this post we will focus on MySQL backups.

Effective MySQL backup methods vary depending on the structure of the server, and the table engines in use. Here are some suggestions:

standalone, MyISAM: mysqldump & binary logging

standalone, innodb: innodb Hot Backup & innobackup Perl script (automates a "sharp" backup of MyISAM & InnoDB tables)

replicated, innodb: stop replication on one slave server, back up that slave server using innobackup and/or mysqldump, re-syncrhronize the slave server

clustered, ndbcluster: "online backup", which is a "hot" or "live" backup initiated using the MySQL cluster management server

Also, some open source and commercial filesystem backup solutions can provide agents or plug-ins that allow "hot" backups of MySQL. One solution we often use is Arkeia. From Arkeia's home page:

"The Arkeia MySQL plug-in is easy to install and configure: databases and table backup/restore can be selected via the convenient navigator in the Arkeia GUI. Backup can be defined up to the table level. Binary logs are also backed up, enabling the restore of databases at an exact state between two backups."

"Arkeia has a distinct advantage over other database backup solutions: a multifl ow system that enables faster backup than other software. This allows back up of multiple tables at the same time."

Some people advocate LVM2 snapshots as a way to backup live MySQL databases. However, this approach can be problematic. That is because the very databases that probably need backups most (large, high traffic databases), are the ones for which the snapshot or the number of locked tables may quickly grow too large to make snapshots practical.

Mike Kruckenberg has an excellent post discussing these issues. He concludes that IF you are using the innodb engine, and IF you prefer not to purchase the innodb hot backup program from Innobase Oy (now owned by Oracle), LVM2 snapshots may be a good choice.

We hope we have given you some food for thought. Happy Shepherding!

Technorati Tags: database administration, system administration

PHP Spam Injection First Aid with Apache ModSecurity

David Digioia — Wed, 17 May 2006 14:34:33 +0000

From time to time we work with clients who would like to upgrade their web sites. Often their site is composed of various one-off applications -- typically PHP-based -- that someone built for them. More often than not, these applications were not developed with security in mind.

Our first reaction is to pull the plug, analyze, and rebuild a secure and scalable solution. But pulling the plug is usually not an option. If a company relies on an application for leads or sales, they probably can't afford to shut it down for any length of time. Under these circumstances, triage is usually the best one can hope for.

Fortunately, there are a few things one can do to stem the bleeding. One of the more common problems with PHP-based applications is that they can allow the injection of malicious content, such as SQL or email spam. In some cases we find that over 95% of a client's ISP traffic is coming from spam injection. The solution? Grab an industrial size helping of Apache mod_security.

What is it? From the ModSecurity home page:

ModSecurityTM is an open source intrusion detection and prevention engine for web applications (or a web application firewall). Operating as an Apache Web server module or standalone, the purpose of ModSecurity is to increase web application security, protecting web applications from known and unknown attacks.

Essentially, it inspects web traffic passing through the web server for suspicious content as well as attempts to trigger buffer overflows, etc. When it finds such content, it can stop the traffic and/or log the incident.

To put mod_security to work for you, first, download and unpack the tarball, build and install the DSO, and update Apache’s httpd.conf file.

cd /usr/local; tar xzf /root/modsecurity-apache-1.9.4.tar.gz
cd /usr/local/modsecurity-apache-1.9.4/apache2
/usr/local/apache2/bin/apxs -cia mod_security.c

Paste the ModSecurity minimal recommended filtering rules into your httpd.conf file. Here are the first few lines from from the online manual's Appendix A: Recommended Configuration:

# Turn ModSecurity On

SecFilterEngine On

# Reject requests with status 403

SecFilterDefaultAction "deny,log,status:403"

# Some sane defaults

SecFilterScanPOST On

SecFilterCheckURLEncoding On

SecFilterCheckUnicodeEncoding Off

If you'd just like to see if someone is trying to exploit your site, you can set up ModSecurity to simply audit your traffic. The lines

# Log rule violations, but allow the requests

SecFilterDefaultAction "log,pass"

will do that for you. However, please note that if you want to merely log rule violations without denying the traffic, you must not include any implicit validations (URL encoding validation, Unicode encoding validation, cookie format validation, and byte range restrictions) in your rules.

When you are satisfied with your rules, you can deny the traffic by changing the default action to this:

# Deny requests and log with status 403

SecFilterDefaultAction "deny,log,status:403"

Once you've got a bunch of traffic in your audit log, you can grep through it to see if you've got visitors with bad intentions:

grep -i 'to|bcc|cc' audit_log | less
or
grep -i 'to|bcc|cc' audit_log | wc -l

You may find lots of suspicious lines. In fact, you may find that some spammers are including portions of books, stories, or other nonsense, presumably to get past the final recipients' Bayesian spam filters.

To block a common PHP mail injection exploit, add a rule like this to your httpd.conf file in the ModSecurity section:

# necessary to stop spammers doing mail injection into PHP mail forms!!!

SecFilterSelective ARGS_VALUES "\n[[:space:]]*(to|bcc|cc)[[:space:]]*:.*@"

The ModSecurity site also conveniently includes a package of rules, including PHP-related rules, grouped by function. Note that there are “SQL Injection Attack” rules in the “general” conf file. You can include the rule groups you want by using an "Include" directive in the ModSecurity section of your httpd.conf file; i.e. “Include conf/modsecurity-php.conf”.

These rules are a good place to start, as are the rules from gotroot.com. You may need to tweak these a little bit, and be selective in which rulesets you apply. For example, often aggregating IP addresses such as AOL proxies are blocked due to the blacklist rules, which may not be what you want.

This is only a brief introduction, but I hope you will try ModSecurity for yourself, and discover how powerful it can be.