Taking my little experiment Toreas, the free fantasy book or novel if you like, as a sample, I was in updating hell. The page itself where the book is readable from is an instance of MediaWiki. I chose it for the ease of adding new things, a good and stable code base, but also because I had a lot of experience in extending it. Then I wanted to add notes to each release and of course that looks like a blog. So my number one choice was simply WordPress (which also powers this site), for a lot of the same reasons that MediaWiki was the page software. Twitter is not a software for me, but a web service as such and I wanted to add tweets as soon as there is a new chapter out (even though there probably is nobody reading those). I also have plans to include more services, like Facebook, but I have not yet come around to those.

So, a new chapter is out. I have to edit 4 MediaWiki pages (3 of those templates), one of them I have to calculate newly with a script I have written. So for calculating the statistics I have to open the template (which requires editing the main article first to get the edit link) run from a website a script that calls up each and every page already created and creates the stats out of that in wikitext. Now I copy paste this over from my script to the template and save it. Next I have to login to twitter, post the comment in the right format. And last but not least post the blog post on the page, again logging in, selecting the right category and tags and so on and so forth. All of this made it a horror to update and I frankly found myself thinking a few times to delay an update just because of this horrible (admittedly self induced) process.

Then I stumbled across an artifact in my memory. MediaWiki has an API. Looking at it closely I found that you could login, open pages and edit them. You could do pretty much anything with it, but those were my only requirements. After looking around a bit, I found a class named Snoopy that is supposed to take care of it, but it was still too much “manual labor” on my side. So A quick check on ZendFramework and in the Http Client library I found all the parts that I needed to make this work. I could define one client and use it like a browser. It would even keep the cookies for me and do all that handling.

So I whipped up my own MediaWiki API class, based on Zend_Http_Client. It actually uses the autoloader as well, but that’s because I’m lazy. So now I instantiate it by handing it the api url, a user name and password, and the whole thing is ready for duty. A get function gives me the content (needed for the latest changes, where I kill the last line in the box and add a new one on top) and an edit function that allows me to simply overwrite the current version. All the magic is done behind the scenes and no worries for me. Twitter of course also has an API to which Zend already has Zend_Service_Twitter. Again, three lines and the new status is posted. And lastly, WordPress has an XMLPRC interface, which has had me quiet confused for a while. Now that I understand it (or at least I think I do) I am able to post news in there by just supplying a title and description, while the tags and categories are preset (for this particular thing).

My 20 minutes of monkeying around with everything, plus additional time for writing a blog post if I was so inclined to do, has been reduced to copy pasting the story (then write the blog post) and hit a button. So for the last eight chapters I have lost two and a half hours of time, which no longer happens. Not only do I gain time, but the step to actually publish the next chapter has become pretty much nothing, as opposed to the big process it has been before.

Once I clean up the API class a little, I might even release my code for the MediaWiki class.

Feel free to Flattr this post at flattr.com, if you like it.

Translate uses an adapter to actually do the heavy lifting. Translate itself is just a gateway to the adapter classes, so the magic happens there. Now my solution was to just create a new adapter and have it give back a string that’s fixed every single time. So any request to the translate method would return the same string. With this you can just look on your page, see where it doesn’t display your special string and there you have your missing spots. You could set it to be an empty string, but if you use it for links, then you wont see the link anymore.

Well used, you will only see the data strings, like user names, the configured replacement string and of course any string you missed.

I use the App prefix for such very common bits of code that I can share easy between all my applications and of course with others, but there is nothing preventing you from changing this of course.

First create a file called Testing.php in the folder library/App/Translate/Adapter filled with the content below.

<?php

/** Zend_Translate_Adapter */
require_once 'Zend/Translate/Adapter.php';

/**
 * @category   App
 * @package    App_Translate_Adapter_Testing
 * @copyright  Copyright (c) 2009 Christian Riesen <chris.riesen@gmail.com>
 * @license    Public Domain
 */
class App_Translate_Adapter_Testing extends Zend_Translate_Adapter
{
    private $_data = array();
    private $_teststring = 'XtestX';

    /**
     * Generates the adapter
     *
     * @param  array               $data     Translation data
     * @param  string|Zend_Locale  $locale   OPTIONAL Locale/Language to set, identical with locale identifier,
     *                                       see Zend_Locale for more information
     * @param  array               $options  OPTIONAL Options to set
     */
    public function __construct($data, $locale = null, array $options = array())
    {
        $this->_loadTranslationData($data, $locale, $options);
    }

    /**
     * Load translation data
     *
     * @param  string|array  $data
     * @param  string        $locale  Locale/Language to add data for, identical with locale identifier,
     *                                see Zend_Locale for more information
     * @param  array         $options OPTIONAL Options to use
     * @return array
     */
    protected function _loadTranslationData($data, $locale, array $options = array())
    {
        $this->_data = array();
        if (!is_null($data)) {
        	$this->_teststring = $data;
        }
        return $this->_data;
    }

    /**
     * Translates the given string
     * returns the translation
     *
     * @param  string|array       $messageId Translation string, or Array for plural translations
     * @param  string|Zend_Locale $locale    (optional) Locale/Language to use, identical with
     *                                       locale identifier, @see Zend_Locale for more information
     * @return string
     */
    public function translate($messageId, $locale = null)
    {
        return $this->_teststring;
    }

    /**
     * returns the adapters name
     *
     * @return string
     */
    public function toString()
    {
        return "Testing";
    }
}

So now it’s ready for use. To use it you have a line that starts something like this.

$translate = new Zend_Translate('array', ...

Now this specific one would be for the array adapter. Just comment this line and add this line here.

$translate = new Zend_Translate('App_Translate_Adapter_Testing', 'XtestingX');

All the rest can stay the same as you have used it before.

Load your application and all the translated strings will have changed to XtestingX and whatever is left over points you in the right direction. I checked and could not find anything this class would break, in case you use other functions with translate or the adapter directly. Of course some classes will not return something, like a list of all translations, but they should not break, should you attempt to call them.

Of course once you tested, you have to reverse this, or you could use a configuration option or a switch to enable it. Personally, I prefer to comment it and uncomment when really needed and testing is done. Test on a none live server, or your page will become useless in an instant (or at the least very confusing).

Feel free to use it, anyway you wish. I specified it as Public Domain which I’m trying out as a license now for a while and see how feedback and experiences are with it. Yes I know that public domain is not a license per se, as it’s a free for all thing, but maybe that’s a “really good thing” ™.

Feel free to Flattr this post at flattr.com, if you like it.

To make life easy, I’ll use Zend Framework for this. You could use anything else that sends queries, but since I already used it for this project, there was little sense in doing something else.  So, with all useless stuff stripped out, here the code in it’s first, rather infancy way.

$db = Zend_Registry::get('db');
$x = 0;
$y = 0;
$z = 0;

while($z < 100)
{
	while($y < 100)
	{
		while($x < 100)
		{
			$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
			$query .= '('.$x.', '.$y.', '.$z.')';
			$db->query($query);
			$x++;
		}
		$y++;
		$x = 0;
	}
	$z++;
	$y = 0;
}

This sample creates rows for a location in a 3D space in case you wonder. So these lines represent single points in a cube with the side length of 100 whatever units you want to imagine here. It starts at 0,0,0 and ends at 99,99,99. So this gives us our nice 1 million entries.

The $db variable up there is a Zend_Db_Adapter instance, so I can now run queries directly. Yes I know you could write the query differently, but I left it like that to make life easier for the next steps. So instead of firing off 1 Million queries, how about grouping 100 queries together, into one large one? I left the top part out, so I will only post the while loop. The result looks like this.

while($z < 100)
{
	while($y < 100)
	{
		$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
		$first = true;
		while($x < 100)
		{
			if ($first == TRUE)
			{
				$first = false;
			}
			else
			{
				$query .= ', ';
			}
			$query .= '('.$x.', '.$y.', '.$z.')';
			$x++;
		}

		$db->query($query);
		$y++;
		$x = 0;
	}
	$z++;
	$y = 0;
}

It’s faster already, but it still does a lot of trashing around on the DB. While we now have “only” 10’000 queries instead of 1 Million, it still is far from “nice”. I could extended this and create larger queries, so I end up with 100 queries, each of them having 10’000 entries, but that might by too large queries and it does not solve much of my problem if I have even more data. Since I’m using MySQL for this, I have transactions at my disposal as well. So instead of writing it all with every query, I start a transaction, run 100 queries then commit that batch. But wait isn’t that the same thing than just writing 100 queries? In theory, yes, practically though this changes a few things. Since those queries are small, the system can keep them in RAM and write them in one big write, combined together, instead of 100 small changes, it makes one big one, lightning fast. Here how this looks like.

while($z < 100)
{
	$db->beginTransaction();
	while($y < 100)
	{
		$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
		$first = true;
		while($x < 100)
		{
			if ($first == TRUE)
			{
				$first = false;
			}
			else
			{
				$query .= ', ';
			}
			$query .= '('.$x.', '.$y.', '.$z.')';
			$x++;
		}

		$db->query($query);
		$y++;
		$x = 0;
	}
	$db->commit();
	$z++;
	$y = 0;
}

Now PHP will write all 1 Million rows in under 30 seconds even on a relative small system.

You could of course optimize this, maybe even wrap the whole process in a transaction instead of single steps, but that’s up to you and your ingenuity. For my needs this did the job so I had no need to go even further with it. Either way you should be able to take it and run with it fast and far for your problem of inserting large amounts of data.

Again, I used Zend Framework, so in your case the beginTransaction and commit function might not exist or named differently.

Feel free to Flattr this post at flattr.com, if you like it.