I’m accessing a blob in a MS SQL database that is pretty much nothing more than a JPG dumped straight into the field. I read them and echo them with a jpg header in PHP. All is well. The script runs and parses all the images, doing some caching magic with them. The result is that about a third of the images are broken, seemingly random. Mostly they just “stop” at a random spot over the whole set, but in one image it always stops at the same spot.

I suspected data corruption on input and a refresh on what inserts them in the first place worked, sometimes at least.

Digging down, I saw that the result I get back for the broken images is always 64512 bytes, exactly. That’s 2 to the power of 16. Can’t be a coincidence. Turns out, it isn’t.

I’m using freetds to access the database and there is an entry in freetds.conf that reads something like this:
text size = 64512

Upping that number (and restarting the webserver, apache or IIS) did the trick. The images stored in the database were all around that point, so it looked like a random bug, but thankfully it wasn’t random.

If you don’t use freetds but the mssql extension instead, then this line would be of interest:
mssql.textsize = 4096

Yes the default is only 4 kilobytes.

Maybe the desk will work as a third hand for the triple face palm.

Feel free to Flattr this post at flattr.com, if you like it.

Taking my little experiment Toreas, the free fantasy book or novel if you like, as a sample, I was in updating hell. The page itself where the book is readable from is an instance of MediaWiki. I chose it for the ease of adding new things, a good and stable code base, but also because I had a lot of experience in extending it. Then I wanted to add notes to each release and of course that looks like a blog. So my number one choice was simply WordPress (which also powers this site), for a lot of the same reasons that MediaWiki was the page software. Twitter is not a software for me, but a web service as such and I wanted to add tweets as soon as there is a new chapter out (even though there probably is nobody reading those). I also have plans to include more services, like Facebook, but I have not yet come around to those.

So, a new chapter is out. I have to edit 4 MediaWiki pages (3 of those templates), one of them I have to calculate newly with a script I have written. So for calculating the statistics I have to open the template (which requires editing the main article first to get the edit link) run from a website a script that calls up each and every page already created and creates the stats out of that in wikitext. Now I copy paste this over from my script to the template and save it. Next I have to login to twitter, post the comment in the right format. And last but not least post the blog post on the page, again logging in, selecting the right category and tags and so on and so forth. All of this made it a horror to update and I frankly found myself thinking a few times to delay an update just because of this horrible (admittedly self induced) process.

Then I stumbled across an artifact in my memory. MediaWiki has an API. Looking at it closely I found that you could login, open pages and edit them. You could do pretty much anything with it, but those were my only requirements. After looking around a bit, I found a class named Snoopy that is supposed to take care of it, but it was still too much “manual labor” on my side. So A quick check on ZendFramework and in the Http Client library I found all the parts that I needed to make this work. I could define one client and use it like a browser. It would even keep the cookies for me and do all that handling.

So I whipped up my own MediaWiki API class, based on Zend_Http_Client. It actually uses the autoloader as well, but that’s because I’m lazy. So now I instantiate it by handing it the api url, a user name and password, and the whole thing is ready for duty. A get function gives me the content (needed for the latest changes, where I kill the last line in the box and add a new one on top) and an edit function that allows me to simply overwrite the current version. All the magic is done behind the scenes and no worries for me. Twitter of course also has an API to which Zend already has Zend_Service_Twitter. Again, three lines and the new status is posted. And lastly, WordPress has an XMLPRC interface, which has had me quiet confused for a while. Now that I understand it (or at least I think I do) I am able to post news in there by just supplying a title and description, while the tags and categories are preset (for this particular thing).

My 20 minutes of monkeying around with everything, plus additional time for writing a blog post if I was so inclined to do, has been reduced to copy pasting the story (then write the blog post) and hit a button. So for the last eight chapters I have lost two and a half hours of time, which no longer happens. Not only do I gain time, but the step to actually publish the next chapter has become pretty much nothing, as opposed to the big process it has been before.

Once I clean up the API class a little, I might even release my code for the MediaWiki class.

Feel free to Flattr this post at flattr.com, if you like it.

Translate uses an adapter to actually do the heavy lifting. Translate itself is just a gateway to the adapter classes, so the magic happens there. Now my solution was to just create a new adapter and have it give back a string that’s fixed every single time. So any request to the translate method would return the same string. With this you can just look on your page, see where it doesn’t display your special string and there you have your missing spots. You could set it to be an empty string, but if you use it for links, then you wont see the link anymore.

Well used, you will only see the data strings, like user names, the configured replacement string and of course any string you missed.

I use the App prefix for such very common bits of code that I can share easy between all my applications and of course with others, but there is nothing preventing you from changing this of course.

First create a file called Testing.php in the folder library/App/Translate/Adapter filled with the content below.

<?php

/** Zend_Translate_Adapter */
require_once 'Zend/Translate/Adapter.php';

/**
 * @category   App
 * @package    App_Translate_Adapter_Testing
 * @copyright  Copyright (c) 2009 Christian Riesen <chris.riesen@gmail.com>
 * @license    Public Domain
 */
class App_Translate_Adapter_Testing extends Zend_Translate_Adapter
{
    private $_data = array();
    private $_teststring = 'XtestX';

    /**
     * Generates the adapter
     *
     * @param  array               $data     Translation data
     * @param  string|Zend_Locale  $locale   OPTIONAL Locale/Language to set, identical with locale identifier,
     *                                       see Zend_Locale for more information
     * @param  array               $options  OPTIONAL Options to set
     */
    public function __construct($data, $locale = null, array $options = array())
    {
        $this->_loadTranslationData($data, $locale, $options);
    }

    /**
     * Load translation data
     *
     * @param  string|array  $data
     * @param  string        $locale  Locale/Language to add data for, identical with locale identifier,
     *                                see Zend_Locale for more information
     * @param  array         $options OPTIONAL Options to use
     * @return array
     */
    protected function _loadTranslationData($data, $locale, array $options = array())
    {
        $this->_data = array();
        if (!is_null($data)) {
        	$this->_teststring = $data;
        }
        return $this->_data;
    }

    /**
     * Translates the given string
     * returns the translation
     *
     * @param  string|array       $messageId Translation string, or Array for plural translations
     * @param  string|Zend_Locale $locale    (optional) Locale/Language to use, identical with
     *                                       locale identifier, @see Zend_Locale for more information
     * @return string
     */
    public function translate($messageId, $locale = null)
    {
        return $this->_teststring;
    }

    /**
     * returns the adapters name
     *
     * @return string
     */
    public function toString()
    {
        return "Testing";
    }
}

So now it’s ready for use. To use it you have a line that starts something like this.

$translate = new Zend_Translate('array', ...

Now this specific one would be for the array adapter. Just comment this line and add this line here.

$translate = new Zend_Translate('App_Translate_Adapter_Testing', 'XtestingX');

All the rest can stay the same as you have used it before.

Load your application and all the translated strings will have changed to XtestingX and whatever is left over points you in the right direction. I checked and could not find anything this class would break, in case you use other functions with translate or the adapter directly. Of course some classes will not return something, like a list of all translations, but they should not break, should you attempt to call them.

Of course once you tested, you have to reverse this, or you could use a configuration option or a switch to enable it. Personally, I prefer to comment it and uncomment when really needed and testing is done. Test on a none live server, or your page will become useless in an instant (or at the least very confusing).

Feel free to use it, anyway you wish. I specified it as Public Domain which I’m trying out as a license now for a while and see how feedback and experiences are with it. Yes I know that public domain is not a license per se, as it’s a free for all thing, but maybe that’s a “really good thing” ™.

Feel free to Flattr this post at flattr.com, if you like it.

You want one entry form a database, selected by a unique ID number. In most cases you have an index, probably even a primary index that will find your record in no time. That is the ideal case though. Now you make lists with many entries, where you already have a much larger set of data to wrangle and again indexes can be your friend, but are less well performing as when you could select them by ID. And worst of all is full text searching. You want all entries that contain a text in multiple fields. Maybe even partial matches and so on. Doing this with just a normal query involves things as LIKE and probably under MySQL the % sign. If you don’t have many entries this might even work well enough. Queue up large databases and you are completely out of luck.

Sphinx helps you with these scenarios.

In Sphinx you run a indexer program that goes into the database and generates it’s own index for words and even partial words. Depending on the configuration you can teach it to not do partial matches, even match in the middle of the words or treat similar words as one and the same word (both on indexing and on search queries). Switching to Sphinx is very simple. You have to install Sphinx on your server (which is very easy), setup the configuration file to reflect your database and search preferences and then just run the indexer script periodically (depending on your content). Inside your application you change the path that leads to those LIKE queries to ask Sphinx over the supplied API. Now Sphinx does not return the results from the database, but gives you a list of id’s you can then use to query the database. Now you can run just a fast fetch for those id’s and you have your result set.

If you just search for all entries from category X or user Z then you are still better off just using the usual ways. But you need full text searching, then Sphinx is your perfect solution.

And now the latest version is out. There are a lot of bugs squished in it, but most not very important, which goes to show how well it has worked now for a long while already. My oldest installation runs for over a year now without a glitch on the Sphinx end.

Currently Sphinx needs to rebuild the whole index each time you want it rebuilt, or use something called delta indexing, which is more a stopgap measure. A real-time updating index is on the menu for the next version though and will make this extremely powerful tool even better.

Feel free to Flattr this post at flattr.com, if you like it.

<?php
// wget substitute
if ($argc != 2) {
?>
This works a bit like wget. Just run it with the url as argumnet
<?php
    exit;
}

$url = $argv[1];
$filename = basename($url);

file_put_contents($filename,file_get_contents($url));

echo PHP_EOL. 'Written file ' . $filename . PHP_EOL;

Usage is rather simple now:
php wget.php URL

Feel free to Flattr this post at flattr.com, if you like it.

Feel free to Flattr this post at flattr.com, if you like it.

Sometimes you need it for programming reasons, SQL is often pretty unhelpful if you got an extra comma there. Sometimes you just want it that way because it looks nice. Usually you have values in an array to begin with.

$array = ('a', 'b', 'c', 'd');

Now you need to do soemthing with each entry, well, that screams foreach.

$array = ('a', 'b', 'c', 'd');
$out = '';
foreach ($array as $value)
{
	$out .= $value . ', ';
}

But that gives you the dangling comma and a space at the end. So you could now trim that off the end (substr) or already do it the right way in the foreach. That could look something like this.

$array = ('a', 'b', 'c', 'd');
$out = '';
$first = true;
foreach ($array as $value)
{
	if (true == $first)
	{
		$first = false;
	}
	else
	{
		$out .= ', ';
	}
	$out .= $value;
}

Now we also need to add a link around it.

$array = ('a', 'b', 'c', 'd');
$out = '';
$first = true;
foreach ($array as $value)
{
	if (true == $first)
	{
		$first = false;
	}
	else
	{
		$out .= ', ';
	}
	$out .= '<a href="/' . $value . '/">' . $value . '</a>';
}

A lot of code and a lot of extra checks for not much gain really. Isn’t there a better way? After all there is only a need for this addition in between the parts of the array. If only there was a function that puts something between the pieces of an array. Ah but yes there is, and it’s called implode. If we had the first simple array and list this could be over rather quick like this.

$array = ('a', 'b', 'c', 'd');
$out = implode(', ', $array);

But we have this extra data we need around the array, so what do we do? Well first generate a new array, then implode of course, like this.

$array = ('a', 'b', 'c', 'd');
$temp = array();
foreach ($array as $value)
{
	$temp[] = '<a href="/' . $value . '/">' . $value . '</a>';
}
$out = implode(', ', $temp);
unset($temp);

Much more comfortable and also much more readable now. First one round preparing, then gluing the pieces into the string. And of course, if you don’t need it anymore, unset the array or you will carry that data around the application for no reason.

Feel free to Flattr this post at flattr.com, if you like it.

So, using a windows workstation I connected to both locations and did a search in the windows explorer. Don’t worry, this insanity has a method. Since I knew the date of when the full copy was made, I could simply search for all files on the windows server that are modified after said date. Only problem now, how to get the list into something usable? I mean you can’t just copy that 6000 files list into notepad or your favorite spreadsheet.

For that there is a free little tool called SysExporter. It will allow you to copy not only the content of any list under windows, but also have it as a tab delimited list.

That saved and a second file for the Linux server in the same format (for easy comparison), now all I had to do was run a PHP script that parses both files as CSV files, using the tabulator character as delimiter and voila, you have two arrays in the same structure that you can compare. There I compared file name and the folder (after normalizing both so the beginning is the same for each). On no match, it’s written in a new file. All those entries in the new file are then my list of files I have to copy and check for needed changes. Of course it’s still the same tab delimited format so you can just import it into a spreadsheet for easier sorting and display.

As a small caveat, windows likes to also search in ZIP files. If you do not really need that, you should disable it. Just go to start, run, then enter this: regsvr32 /u %windir%\system32\zipfldr.dll
A small box appears telling you all went well and now you no longer search for ZIP files. You should maybe get yourself some replacement like 7zip while you are at it.

Feel free to Flattr this post at flattr.com, if you like it.

But there is no simple –csv switch in mysqldump, your weapon of choice for these tasks. So here the command that will allow you to do what you are after:

mysqldump -p -u USER -T DIRECTORY --fields-enclosed-by=\" --fields-terminated-by=, DATABASE

So this is the short version, and here what it all means:

• -p : Asks for a password, as most users have one. If you don’t specify this on a database with a user/password, it will error
• -u USER : Replace USER with your actual username to connect to the database.
• -T DIRECTORY : This creates a tab delimited file. Not what we wanted but it’s the base we need.
• –fields-enclosed-by=\” : Will add ” characters around the fields. This will allow CSV implementations to find everything that fits together. You will need that backslash or it wont run.
• –fields-terminated-by=, : The so much sought after comma. This replaces the tab and puts a comma in its place, which, you guessed it, creates the CSV file.
• DATABASE : Well you know, the thing this is all about…

To actually be able to do it though, you will need the FILE privilege on this database. Armed with this, you should be able to do your CSV exports easy now.

Feel free to Flattr this post at flattr.com, if you like it.

Often this is paired with an array and keys to find the correct content. Sometimes a function is used as a wrapper (using static variables more often than not) and similar rather ugly things.

Well, what does keep a state in itself very well? A class of course.

On Aidan Lister’s page I cam across a very interesting piece of code that cycles between strings. You instantiate the class with the values to cycle through given to the constructor. Now from here on out all you do is use the variable, holding the object directly like it would contain the value you need directly, say an integer or a string. Through the magic method __toString() it then dispenses the current value and advances the pointer, or wraps it around to the start.

Very simple yet powerful and also some food for thought, as this exact method might work well on other problems.

Feel free to Flattr this post at flattr.com, if you like it.