How to get zendx_jquery_form_element_autocomplete to respond in real time

Were I work for a living now, they use Zend Framework and zendx_jquery_form_element_autocomplete. It's slow, and definitely not responding in real time. A quick test with Firebug shows a median response time of 300 ms. Far from the 200 ms needed to make it appear instantaneous.
Everyone says the database is slow, I didn't take it further at first.

But, yes, the whole thing is slow. Something was missing an index, which I added, but the time gains were very slim. I found that suspicious. It's not a big deal, it's an application with few users, so I concentrated on the auto-complete part, which is worth optimizing.

The index was right, the code was ok, and it took 300 ms. It has to be the database. Or slow machine executing the code. I blindly tried xcache, it saved about 100 ms. Or, well, there was something else going on.

I decided to bypass everything and wrote a very simple script, doing the bare query, printing in json the result array, along the line of this:


// Connection and database selection

$query = "SELECT name FROM table WHERE name LIKE '" . mysql_real_escape ($_REQUEST['term']) . "%'";

$my_res = mysql_query ($query);

$result = array ();
while ($item = mysql_fetch_array ($my_res))
    $result[] = $item['name'];

print json_encode ($result);
 
 
And I ran the thing in Firebug. Tadam... It suddenly took only 6 ms. Right, that's a 50 fold time gain.

Is it clean ? No, not really. Is Zend Framework slowing your applications ? Well, now you know how to deal with that one issue which matters.

Building a corpus for classification: size isn't always the only problem

Building a meaningful corpus for classifying documents is a tedious task. Building a few is even more tedious.

That's an obvious fact.

The trouble is, as the rule stands with anything involving language: ambiguity finally rules and corners you.

Let's say, for instance, that we are trying to classify articles from newspapers, in three categories: "business", "energy" and "environment".
That seems simply enough. The next article about Lady Gaga is out. The next articles on coal prices is in. It's in. Right. It fall in the business section, as it's going to influence energy prices. Wait, energy prices, that should be also in energy then, right ? But won't coal prices influence the environment too, as renewable will be less competitive ?
Wait, once again, wouldn't classifying this in environment be biased ? What about global warming naysayers ?
What began as a simple and trivial classification finally ends up in a mess. The multiple section facet is solved fast: use that article as a reference for all three categories, never mind the confusion it will potentially give to the computer which has to decide which category the evaluated articles will fall into. But what about the implicit point of view concerning this simple example, and its potential bias ?

If that seems too obvious (and in some ways, it is), let's try another one: identifying if a search string refers to a person or anything else. It's easy to define a person, right ?
Are Greek Gods persons for instance ? One can find pictures of them, they were treated as beings and interacted quite intimately with human beings. They could be persons then.
Let's say Gods are people. Then, what about Allah, which isn't quite really a person, but definitely a God to many people ?
There's also the more trivial case of movie and tv characters: should they be classified as beings, or ?...

Once again, the brain with its seemingly endless prowesses for confusion puzzles easily the man trying to teach a thing or two to the computer.

Searching images for portraits

Building my search engine drew me to try to index other things than raw text: images, videos, news, etc. It has been a while now since I worked on the stuff, and I recently said to myself that it's a shame not to do something useful and creative with the bunch of images that I gathered years ago.
I forgot that I had more than 10 millions images from all over the web. It would indeed be a shame not to do something with that.

Following what was done at the time, and not having so much time to look back at all the possibilities, I thought about what anyone was already doing: search by tone color, sizes, etc. Things (more or less) easily technically done, but most of the time with very little value for the user.

And then, a simple thought came to mind: what about allowing search by portraits ? As a first step towards search by shapes and objects.

It's rather simple to do nowadays: you probably have stumbled upon one of those software for webcams which track faces: that's essentially the kind of software  we need to find a portrait in an image.
If there's a face, and it's big enough, it must be a portrait.

So, the first big step is being able to recognize a face on an image. That's simple enough thanks to a great library which can (among a lot of other things) do that: OpenCV. For that task, cvHaarDetectObjects is what we are looking for (it's for C, the exact function/method name which applies will vary according to your programming language of choice).

And to make things even more simple for a quick and dirty evaluation of the thing, there's for example a very basic PECL extention for PHP, PHP-Facedetect, which does exactly what we need.
The configure script being broken as the time of this writing and the git pull request waiting to be accepted, you can check out the patched extension I wrote.

That should be enough ground and ideas to get you started ! Share your ideas ! :)

Design patterns

A while ago in the company I worked for, I was assigned a task.

Me being given that task was an error: a guy with the same first name as me was supposed to get it.
I could go on and on how that situation got managed, but this blog is about serious stuff and not about politics and egos in the workplace.

So, I got a look at the application I was supposed to maintain and to make evolve. A very nice java application, quite well written, hundreds of line of code, readable and so on. Nicely written from a design patterns point of view.

What was this application doing with its hundreds of lines of code ?
$ xslproc sheet.xsl document.xsl > output.xml

Yes, these hundred of lines of code fit into one line. What about datasource are you asking ? Well, everything is a file in unix...

Don't get me wrong, there are a lot of things I hate about the Unix way, but...

Please, let's keep things as simple as they are... And you know what ? They are usually more efficient and as powerful when they are kept simple...

Ubuntu and "large files" (files greater than 2 G)

Recently, for whatever reason, I decided that I should have Linux on a few machines. It might have been to be sure that my tools run on Linux.

It happened that Linux ended up on one of the key machines.
That was pure over-confidence and trust.

If FreeBSD hasn't any problem with "big files", that is, files greater that 2 gigabytes, something one could expect in 2009, one could expect the same from Linux.

Well, not quite so. The spider got stuck around 2 G of data.

The two magics options to add at compile time before banging your heads on the walls (it does hurt, trust my experience):

-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

Why does one need to add these defines ? (which trigger an ugly cascade of other defines, it seems a lot of people decided this needs to be frightening)
I have no idea why.

Anyway, that's two flags one needs to add.

Error

An interesting thing to feed a search engine is the word "error".

It's a very simple, single word.

You say, a good search engine will know how to handle all those "mysql error", "php error", "error page not found", etc etc.

Well, it's pretty interesting to see how well each behaves to that.

Some just give you in their top results "Error! Reason: File 'index.html was not found!" (right, this is the second result on a popular French engines). Some do it as their top result (Error. Reason: File "menu.asp" was not found!)

Others will play it on the safe side, displaying news that mentions error, the wikipedia error entry, then, the algorithm seeing that the remaining thousands of pages are wrong show only a handful of results.

I found one search engine that seemed less prone to this kind of troubles that the major search engine we know.

One the second hand, my search engine seems to be affected a lot later than the other search engines. That's suspicious, considering it's unlikely I'm the only one to run and try to fix that trouble. The real answer is... well, it doesn't show very well the content of the page, and where that "error" comes from.

Is this question relevant in any way ? It's the traditional "the written words don't really mean what the website is about".
Even if we are using the link text from web page to web page, there's going to be guys saying "this pages yields an error" or something like that.
One can say that, well, if enough guys say something different about the page, use what the majority is saying; true, but if you are looking for fast raising page per words...

I don't have an answer to that; it looks like I'm not the only one, and that's not really a comforting idea as to the future of search.

Security concepts and an open source search engine

I was reading tonight a very long list of comments on how an ideal distributed opensource search engine could be.

The interesting things, reading the comments, is how it relates to security. Let me explain.

The main argument on why an opensource (even more so a distributed one) search engine can't work in practice is because when you know how the thing works, you can easily influence the results (ie: spam). And then people begin to praise the "security through obscurity" that the major search engines have: it's, according to them, the best way to preserve security.
No need to say, this is wrong. If it wasn't so, big companies wouldn't be spending money optimizing their ranking, especially if that wasn't working at all. Even if you consider the "moron factor", it's too easy to see if it's effective: run a search and see if you are on the first page.

So, obviously, even for ranking, security through obscurity doesn't work.

As a reminder, the most widely used library for secure communication, openssl, which source code is widely available, which encryption algorithm are know, isn't (officially at least), easily cracked. True, there's a lot of money involved in being on the major search engines first page, and people are desperate to get there. It's true too that brilliant guys do spend their days trying to break that openssl thing.

So, maybe that's one of the accurate goals for the next big search engine: a ranking algorithm that can't be diverted, even if you precisely know what the algorithm is.