You want one entry form a database, selected by a unique ID number. In most cases you have an index, probably even a primary index that will find your record in no time. That is the ideal case though. Now you make lists with many entries, where you already have a much larger set of data to wrangle and again indexes can be your friend, but are less well performing as when you could select them by ID. And worst of all is full text searching. You want all entries that contain a text in multiple fields. Maybe even partial matches and so on. Doing this with just a normal query involves things as LIKE and probably under MySQL the % sign. If you don’t have many entries this might even work well enough. Queue up large databases and you are completely out of luck.

Sphinx helps you with these scenarios.

In Sphinx you run a indexer program that goes into the database and generates it’s own index for words and even partial words. Depending on the configuration you can teach it to not do partial matches, even match in the middle of the words or treat similar words as one and the same word (both on indexing and on search queries). Switching to Sphinx is very simple. You have to install Sphinx on your server (which is very easy), setup the configuration file to reflect your database and search preferences and then just run the indexer script periodically (depending on your content). Inside your application you change the path that leads to those LIKE queries to ask Sphinx over the supplied API. Now Sphinx does not return the results from the database, but gives you a list of id’s you can then use to query the database. Now you can run just a fast fetch for those id’s and you have your result set.

If you just search for all entries from category X or user Z then you are still better off just using the usual ways. But you need full text searching, then Sphinx is your perfect solution.

And now the latest version is out. There are a lot of bugs squished in it, but most not very important, which goes to show how well it has worked now for a long while already. My oldest installation runs for over a year now without a glitch on the Sphinx end.

Currently Sphinx needs to rebuild the whole index each time you want it rebuilt, or use something called delta indexing, which is more a stopgap measure. A real-time updating index is on the menu for the next version though and will make this extremely powerful tool even better.

Feel free to Flattr this post at flattr.com, if you like it.

But there is no simple –csv switch in mysqldump, your weapon of choice for these tasks. So here the command that will allow you to do what you are after:

mysqldump -p -u USER -T DIRECTORY --fields-enclosed-by=\" --fields-terminated-by=, DATABASE

So this is the short version, and here what it all means:

• -p : Asks for a password, as most users have one. If you don’t specify this on a database with a user/password, it will error
• -u USER : Replace USER with your actual username to connect to the database.
• -T DIRECTORY : This creates a tab delimited file. Not what we wanted but it’s the base we need.
• –fields-enclosed-by=\” : Will add ” characters around the fields. This will allow CSV implementations to find everything that fits together. You will need that backslash or it wont run.
• –fields-terminated-by=, : The so much sought after comma. This replaces the tab and puts a comma in its place, which, you guessed it, creates the CSV file.
• DATABASE : Well you know, the thing this is all about…

To actually be able to do it though, you will need the FILE privilege on this database. Armed with this, you should be able to do your CSV exports easy now.

Feel free to Flattr this post at flattr.com, if you like it.

To make life easy, I’ll use Zend Framework for this. You could use anything else that sends queries, but since I already used it for this project, there was little sense in doing something else.  So, with all useless stuff stripped out, here the code in it’s first, rather infancy way.

$db = Zend_Registry::get('db');
$x = 0;
$y = 0;
$z = 0;

while($z < 100)
{
	while($y < 100)
	{
		while($x < 100)
		{
			$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
			$query .= '('.$x.', '.$y.', '.$z.')';
			$db->query($query);
			$x++;
		}
		$y++;
		$x = 0;
	}
	$z++;
	$y = 0;
}

This sample creates rows for a location in a 3D space in case you wonder. So these lines represent single points in a cube with the side length of 100 whatever units you want to imagine here. It starts at 0,0,0 and ends at 99,99,99. So this gives us our nice 1 million entries.

The $db variable up there is a Zend_Db_Adapter instance, so I can now run queries directly. Yes I know you could write the query differently, but I left it like that to make life easier for the next steps. So instead of firing off 1 Million queries, how about grouping 100 queries together, into one large one? I left the top part out, so I will only post the while loop. The result looks like this.

while($z < 100)
{
	while($y < 100)
	{
		$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
		$first = true;
		while($x < 100)
		{
			if ($first == TRUE)
			{
				$first = false;
			}
			else
			{
				$query .= ', ';
			}
			$query .= '('.$x.', '.$y.', '.$z.')';
			$x++;
		}

		$db->query($query);
		$y++;
		$x = 0;
	}
	$z++;
	$y = 0;
}

It’s faster already, but it still does a lot of trashing around on the DB. While we now have “only” 10’000 queries instead of 1 Million, it still is far from “nice”. I could extended this and create larger queries, so I end up with 100 queries, each of them having 10’000 entries, but that might by too large queries and it does not solve much of my problem if I have even more data. Since I’m using MySQL for this, I have transactions at my disposal as well. So instead of writing it all with every query, I start a transaction, run 100 queries then commit that batch. But wait isn’t that the same thing than just writing 100 queries? In theory, yes, practically though this changes a few things. Since those queries are small, the system can keep them in RAM and write them in one big write, combined together, instead of 100 small changes, it makes one big one, lightning fast. Here how this looks like.

while($z < 100)
{
	$db->beginTransaction();
	while($y < 100)
	{
		$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
		$first = true;
		while($x < 100)
		{
			if ($first == TRUE)
			{
				$first = false;
			}
			else
			{
				$query .= ', ';
			}
			$query .= '('.$x.', '.$y.', '.$z.')';
			$x++;
		}

		$db->query($query);
		$y++;
		$x = 0;
	}
	$db->commit();
	$z++;
	$y = 0;
}

Now PHP will write all 1 Million rows in under 30 seconds even on a relative small system.

You could of course optimize this, maybe even wrap the whole process in a transaction instead of single steps, but that’s up to you and your ingenuity. For my needs this did the job so I had no need to go even further with it. Either way you should be able to take it and run with it fast and far for your problem of inserting large amounts of data.

Again, I used Zend Framework, so in your case the beginTransaction and commit function might not exist or named differently.

Feel free to Flattr this post at flattr.com, if you like it.