Taking my little experiment Toreas, the free fantasy book or novel if you like, as a sample, I was in updating hell. The page itself where the book is readable from is an instance of MediaWiki. I chose it for the ease of adding new things, a good and stable code base, but also because I had a lot of experience in extending it. Then I wanted to add notes to each release and of course that looks like a blog. So my number one choice was simply WordPress (which also powers this site), for a lot of the same reasons that MediaWiki was the page software. Twitter is not a software for me, but a web service as such and I wanted to add tweets as soon as there is a new chapter out (even though there probably is nobody reading those). I also have plans to include more services, like Facebook, but I have not yet come around to those.

So, a new chapter is out. I have to edit 4 MediaWiki pages (3 of those templates), one of them I have to calculate newly with a script I have written. So for calculating the statistics I have to open the template (which requires editing the main article first to get the edit link) run from a website a script that calls up each and every page already created and creates the stats out of that in wikitext. Now I copy paste this over from my script to the template and save it. Next I have to login to twitter, post the comment in the right format. And last but not least post the blog post on the page, again logging in, selecting the right category and tags and so on and so forth. All of this made it a horror to update and I frankly found myself thinking a few times to delay an update just because of this horrible (admittedly self induced) process.

Then I stumbled across an artifact in my memory. MediaWiki has an API. Looking at it closely I found that you could login, open pages and edit them. You could do pretty much anything with it, but those were my only requirements. After looking around a bit, I found a class named Snoopy that is supposed to take care of it, but it was still too much “manual labor” on my side. So A quick check on ZendFramework and in the Http Client library I found all the parts that I needed to make this work. I could define one client and use it like a browser. It would even keep the cookies for me and do all that handling.

So I whipped up my own MediaWiki API class, based on Zend_Http_Client. It actually uses the autoloader as well, but that’s because I’m lazy. So now I instantiate it by handing it the api url, a user name and password, and the whole thing is ready for duty. A get function gives me the content (needed for the latest changes, where I kill the last line in the box and add a new one on top) and an edit function that allows me to simply overwrite the current version. All the magic is done behind the scenes and no worries for me. Twitter of course also has an API to which Zend already has Zend_Service_Twitter. Again, three lines and the new status is posted. And lastly, WordPress has an XMLPRC interface, which has had me quiet confused for a while. Now that I understand it (or at least I think I do) I am able to post news in there by just supplying a title and description, while the tags and categories are preset (for this particular thing).

My 20 minutes of monkeying around with everything, plus additional time for writing a blog post if I was so inclined to do, has been reduced to copy pasting the story (then write the blog post) and hit a button. So for the last eight chapters I have lost two and a half hours of time, which no longer happens. Not only do I gain time, but the step to actually publish the next chapter has become pretty much nothing, as opposed to the big process it has been before.

Once I clean up the API class a little, I might even release my code for the MediaWiki class.

Translate uses an adapter to actually do the heavy lifting. Translate itself is just a gateway to the adapter classes, so the magic happens there. Now my solution was to just create a new adapter and have it give back a string that’s fixed every single time. So any request to the translate method would return the same string. With this you can just look on your page, see where it doesn’t display your special string and there you have your missing spots. You could set it to be an empty string, but if you use it for links, then you wont see the link anymore.

Well used, you will only see the data strings, like user names, the configured replacement string and of course any string you missed.

I use the App prefix for such very common bits of code that I can share easy between all my applications and of course with others, but there is nothing preventing you from changing this of course.

First create a file called Testing.php in the folder library/App/Translate/Adapter filled with the content below.

<?php

/** Zend_Translate_Adapter */
require_once 'Zend/Translate/Adapter.php';

/**
 * @category   App
 * @package    App_Translate_Adapter_Testing
 * @copyright  Copyright (c) 2009 Christian Riesen <chris.riesen@gmail.com>
 * @license    Public Domain
 */
class App_Translate_Adapter_Testing extends Zend_Translate_Adapter
{
    private $_data = array();
    private $_teststring = 'XtestX';

    /**
     * Generates the adapter
     *
     * @param  array               $data     Translation data
     * @param  string|Zend_Locale  $locale   OPTIONAL Locale/Language to set, identical with locale identifier,
     *                                       see Zend_Locale for more information
     * @param  array               $options  OPTIONAL Options to set
     */
    public function __construct($data, $locale = null, array $options = array())
    {
        $this->_loadTranslationData($data, $locale, $options);
    }

    /**
     * Load translation data
     *
     * @param  string|array  $data
     * @param  string        $locale  Locale/Language to add data for, identical with locale identifier,
     *                                see Zend_Locale for more information
     * @param  array         $options OPTIONAL Options to use
     * @return array
     */
    protected function _loadTranslationData($data, $locale, array $options = array())
    {
        $this->_data = array();
        if (!is_null($data)) {
        	$this->_teststring = $data;
        }
        return $this->_data;
    }

    /**
     * Translates the given string
     * returns the translation
     *
     * @param  string|array       $messageId Translation string, or Array for plural translations
     * @param  string|Zend_Locale $locale    (optional) Locale/Language to use, identical with
     *                                       locale identifier, @see Zend_Locale for more information
     * @return string
     */
    public function translate($messageId, $locale = null)
    {
        return $this->_teststring;
    }

    /**
     * returns the adapters name
     *
     * @return string
     */
    public function toString()
    {
        return "Testing";
    }
}

So now it’s ready for use. To use it you have a line that starts something like this.

$translate = new Zend_Translate('array', ...

Now this specific one would be for the array adapter. Just comment this line and add this line here.

$translate = new Zend_Translate('App_Translate_Adapter_Testing', 'XtestingX');

All the rest can stay the same as you have used it before.

Load your application and all the translated strings will have changed to XtestingX and whatever is left over points you in the right direction. I checked and could not find anything this class would break, in case you use other functions with translate or the adapter directly. Of course some classes will not return something, like a list of all translations, but they should not break, should you attempt to call them.

Of course once you tested, you have to reverse this, or you could use a configuration option or a switch to enable it. Personally, I prefer to comment it and uncomment when really needed and testing is done. Test on a none live server, or your page will become useless in an instant (or at the least very confusing).

Feel free to use it, anyway you wish. I specified it as Public Domain which I’m trying out as a license now for a while and see how feedback and experiences are with it. Yes I know that public domain is not a license per se, as it’s a free for all thing, but maybe that’s a “really good thing” ™.

<?php
// wget substitute
if ($argc != 2) {
?>
This works a bit like wget. Just run it with the url as argumnet
<?php
    exit;
}

$url = $argv[1];
$filename = basename($url);

file_put_contents($filename,file_get_contents($url));

echo PHP_EOL. 'Written file ' . $filename . PHP_EOL;

Usage is rather simple now:
php wget.php URL

Sometimes you need it for programming reasons, SQL is often pretty unhelpful if you got an extra comma there. Sometimes you just want it that way because it looks nice. Usually you have values in an array to begin with.

$array = ('a', 'b', 'c', 'd');

Now you need to do soemthing with each entry, well, that screams foreach.

$array = ('a', 'b', 'c', 'd');
$out = '';
foreach ($array as $value)
{
	$out .= $value . ', ';
}

But that gives you the dangling comma and a space at the end. So you could now trim that off the end (substr) or already do it the right way in the foreach. That could look something like this.

$array = ('a', 'b', 'c', 'd');
$out = '';
$first = true;
foreach ($array as $value)
{
	if (true == $first)
	{
		$first = false;
	}
	else
	{
		$out .= ', ';
	}
	$out .= $value;
}

Now we also need to add a link around it.

$array = ('a', 'b', 'c', 'd');
$out = '';
$first = true;
foreach ($array as $value)
{
	if (true == $first)
	{
		$first = false;
	}
	else
	{
		$out .= ', ';
	}
	$out .= '<a href="/' . $value . '/">' . $value . '</a>';
}

A lot of code and a lot of extra checks for not much gain really. Isn’t there a better way? After all there is only a need for this addition in between the parts of the array. If only there was a function that puts something between the pieces of an array. Ah but yes there is, and it’s called implode. If we had the first simple array and list this could be over rather quick like this.

$array = ('a', 'b', 'c', 'd');
$out = implode(', ', $array);

But we have this extra data we need around the array, so what do we do? Well first generate a new array, then implode of course, like this.

$array = ('a', 'b', 'c', 'd');
$temp = array();
foreach ($array as $value)
{
	$temp[] = '<a href="/' . $value . '/">' . $value . '</a>';
}
$out = implode(', ', $temp);
unset($temp);

Much more comfortable and also much more readable now. First one round preparing, then gluing the pieces into the string. And of course, if you don’t need it anymore, unset the array or you will carry that data around the application for no reason.

So, using a windows workstation I connected to both locations and did a search in the windows explorer. Don’t worry, this insanity has a method. Since I knew the date of when the full copy was made, I could simply search for all files on the windows server that are modified after said date. Only problem now, how to get the list into something usable? I mean you can’t just copy that 6000 files list into notepad or your favorite spreadsheet.

For that there is a free little tool called SysExporter. It will allow you to copy not only the content of any list under windows, but also have it as a tab delimited list.

That saved and a second file for the Linux server in the same format (for easy comparison), now all I had to do was run a PHP script that parses both files as CSV files, using the tabulator character as delimiter and voila, you have two arrays in the same structure that you can compare. There I compared file name and the folder (after normalizing both so the beginning is the same for each). On no match, it’s written in a new file. All those entries in the new file are then my list of files I have to copy and check for needed changes. Of course it’s still the same tab delimited format so you can just import it into a spreadsheet for easier sorting and display.

As a small caveat, windows likes to also search in ZIP files. If you do not really need that, you should disable it. Just go to start, run, then enter this: regsvr32 /u %windir%\system32\zipfldr.dll
A small box appears telling you all went well and now you no longer search for ZIP files. You should maybe get yourself some replacement like 7zip while you are at it.

Often this is paired with an array and keys to find the correct content. Sometimes a function is used as a wrapper (using static variables more often than not) and similar rather ugly things.

Well, what does keep a state in itself very well? A class of course.

On Aidan Lister’s page I cam across a very interesting piece of code that cycles between strings. You instantiate the class with the values to cycle through given to the constructor. Now from here on out all you do is use the variable, holding the object directly like it would contain the value you need directly, say an integer or a string. Through the magic method __toString() it then dispenses the current value and advances the pointer, or wraps it around to the start.

Very simple yet powerful and also some food for thought, as this exact method might work well on other problems.

Instead, I opted for a mini cache in the originating class. The setup is quiet simple, so I now also use this in base classes that get implemented down the line.

private $_localCache = NULL;

public function myFunc()
{
	if (!isset($this->_localCache[__METHOD__]))
	{
		$value = time(); // Create value here
		$this->_localCache[__METHOD__] = $value;
		return $value;
	}
	else
	{
		return $this->_localCache[__METHOD__];
	}
}

The code is as simple as it looks. First you create a private attribute _localCache which is used for caching. Each cache is identified by a key in the _localCache array. That way you can reuse this multiple times very fast and easy. As key, I use the method name. This is especially hand because you don’t have to change any code if you refactor the method or the class.

As with usual caching, it checks if a cache is set, then returns that, otherwise, grabs the real data and caches it. This works of course without any additional cache tools installed, so even though memcached and xcache are wonderful for caching things like this often, they might not be available or simply not really usable.

A prime example is when you know that the value does not change in the execution of this script, but might change between reloads. This way you know for certain you will always use the same data inside the script, while also being sure the next reload fetches the data new.

For static things that get read more often than written to between loads, are better of in a real cache system. If the data changes even less than that, say every few months or so and it’s not a lot, you could also optimize by just hard coding it as an array or class.

Upon testing though there was a weird little snag.

What happened was that the insert scripts failed to find a particular week. Debug revealed that they tried to enter a week 53. But a year has only 52 weeks, right? Wrong. 2009 has actually 53 calendar weeks. So to make it future proof for when such a thing might happen again, here my little code that determines the number of possible weeks in the current year. You need nothing else than the date function to make it work.

$currentYear = date('Y');
$weeksInYear = date('W',mktime(23,59,59,12,31,$currentYear));

This should run fine until someone decides to change the number of days in December, number of months in a year, number of hours in a day or something equally constant.

To make life easy, I’ll use Zend Framework for this. You could use anything else that sends queries, but since I already used it for this project, there was little sense in doing something else.  So, with all useless stuff stripped out, here the code in it’s first, rather infancy way.

$db = Zend_Registry::get('db');
$x = 0;
$y = 0;
$z = 0;

while($z < 100)
{
	while($y < 100)
	{
		while($x < 100)
		{
			$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
			$query .= '('.$x.', '.$y.', '.$z.')';
			$db->query($query);
			$x++;
		}
		$y++;
		$x = 0;
	}
	$z++;
	$y = 0;
}

This sample creates rows for a location in a 3D space in case you wonder. So these lines represent single points in a cube with the side length of 100 whatever units you want to imagine here. It starts at 0,0,0 and ends at 99,99,99. So this gives us our nice 1 million entries.

The $db variable up there is a Zend_Db_Adapter instance, so I can now run queries directly. Yes I know you could write the query differently, but I left it like that to make life easier for the next steps. So instead of firing off 1 Million queries, how about grouping 100 queries together, into one large one? I left the top part out, so I will only post the while loop. The result looks like this.

while($z < 100)
{
	while($y < 100)
	{
		$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
		$first = true;
		while($x < 100)
		{
			if ($first == TRUE)
			{
				$first = false;
			}
			else
			{
				$query .= ', ';
			}
			$query .= '('.$x.', '.$y.', '.$z.')';
			$x++;
		}

		$db->query($query);
		$y++;
		$x = 0;
	}
	$z++;
	$y = 0;
}

It’s faster already, but it still does a lot of trashing around on the DB. While we now have “only” 10’000 queries instead of 1 Million, it still is far from “nice”. I could extended this and create larger queries, so I end up with 100 queries, each of them having 10’000 entries, but that might by too large queries and it does not solve much of my problem if I have even more data. Since I’m using MySQL for this, I have transactions at my disposal as well. So instead of writing it all with every query, I start a transaction, run 100 queries then commit that batch. But wait isn’t that the same thing than just writing 100 queries? In theory, yes, practically though this changes a few things. Since those queries are small, the system can keep them in RAM and write them in one big write, combined together, instead of 100 small changes, it makes one big one, lightning fast. Here how this looks like.

while($z < 100)
{
	$db->beginTransaction();
	while($y < 100)
	{
		$query = 'INSERT INTO location (locx, locy, locz) VALUES ';
		$first = true;
		while($x < 100)
		{
			if ($first == TRUE)
			{
				$first = false;
			}
			else
			{
				$query .= ', ';
			}
			$query .= '('.$x.', '.$y.', '.$z.')';
			$x++;
		}

		$db->query($query);
		$y++;
		$x = 0;
	}
	$db->commit();
	$z++;
	$y = 0;
}

Now PHP will write all 1 Million rows in under 30 seconds even on a relative small system.

You could of course optimize this, maybe even wrap the whole process in a transaction instead of single steps, but that’s up to you and your ingenuity. For my needs this did the job so I had no need to go even further with it. Either way you should be able to take it and run with it fast and far for your problem of inserting large amounts of data.

Again, I used Zend Framework, so in your case the beginTransaction and commit function might not exist or named differently.