Of course a bug is important and should be dealt with, shouldn't it ? Every bug is important and affects the global quality of a product.
Just one second... for whom is that bug important ? (And by the way, is it really a bug, or a user complaining about a product he doesn't understand ? More on this in a later post).
Bug are important, but that importance lays on different axis: the customer's, the manager's, the developer's, the marketer's (add your own).
To put things shortly, different interests will collide on how important that bug is.
And there will be a great confusion and a lot of tension around that issue shortly, repeatedly. Even the tools designed to help with bugs will not help most of the time: very few tools take this conflict of interest into account. To make things worse the default reporting value for a bug is by default "high". Guess what priority all bugs get.
This situation obviously ends up generating a lot of tensions all over. Developers are frustrated being told that a spelling mistake gets a high priority over a major refactoring which will not get done, while marketing is frantic about having corrupt data about the users, said users not understanding why it takes more than a month for a bug (simple, obvious !) to get fixed. And the management tries to fix things by spending longer hours in meetings and making everyone work longer hours "to get people to work right".
The real issue with handling known bugs is about setting the priorities right.
And that means at least to things: evaluating what the different priorities are for each axis, and balancing them. It's not an easy task.
Is that very abusive user being just... abusive ? Is that technical bug creating a crash, but for a very rare use case ? Is it slowly corrupting data in a non-recoverable way ? Is that bug a spelling mistake on welcome screen ? Is the bug reported by a check-signing customer ? And so on...
At the end of the day, a developer will have to fix the bug.
Being able to say for the assigners "hey, we know this is crap for you", and then, but "please fix it right now" really helps to get the developers going.
Avoiding bug reporting tools which do not allow you to use at least two different axis is a very good first step. Being able to say "hey this is futile", but, then, "it's important for our paying users" will make developers laugh at first, that's for sure. it will introduce a very important concept to all though: there are different priorities, and those need to be evaluted.
It will also be a kind reminder to developer that software is meant for final users.
Things will get more easily done. And according to a consistent set of priorities understood and agreed by all.
2014/04/10
2014/01/09
Platform sizing 101
Today at work introducing a new database and application server were on the menu.
This was happening totally against my better judgement, but, somewhere, somehow, I came across an article on management which said "get things done, don't fuss too much how", and it's indeed probably a better way: we are getting things done in the spirit I had, let's just don't mind about the fussy details, things are evolving, and we're avoiding speeding up against a wall.
Enough about politics, to the real stuff now.
Sizing. We have n users. The literature around, I was told, says that the application server can handle n/2 users. Straightforward answer: we need two instances. Right. Wrong ?
Yes, that's the wrong answer. We care about user experience: if one server dies, or, more mundanely, one server is off for an update, the whole load goes to... well, that one server which can actually handle n/2 users. Oops.
Right, there's one way management likes, but users (and workers don't): take your service off and work at odd hours.
Or just setup three servers.
And there's a goodie along the way of that alternative scenario: when your load increases, you get more time to add another one before user experience deteriorates.
This was happening totally against my better judgement, but, somewhere, somehow, I came across an article on management which said "get things done, don't fuss too much how", and it's indeed probably a better way: we are getting things done in the spirit I had, let's just don't mind about the fussy details, things are evolving, and we're avoiding speeding up against a wall.
Enough about politics, to the real stuff now.
Sizing. We have n users. The literature around, I was told, says that the application server can handle n/2 users. Straightforward answer: we need two instances. Right. Wrong ?
Yes, that's the wrong answer. We care about user experience: if one server dies, or, more mundanely, one server is off for an update, the whole load goes to... well, that one server which can actually handle n/2 users. Oops.
Right, there's one way management likes, but users (and workers don't): take your service off and work at odd hours.
Or just setup three servers.
And there's a goodie along the way of that alternative scenario: when your load increases, you get more time to add another one before user experience deteriorates.
2013/10/22
The boss issue with distributing (computational) work
Here in France (your mileage may very then), the boss gives jobs to subordinates. That's what subordinates are for, and that's what a boss is here for.
Now, as some of you might have noticed, there are a few glitches to that scenario. The boss most of the time does not correctly estimate the time needed for a task if he does that only on his own. If he asks, it means hours of meetings. Not very efficient.
From the boss' view, there was no doubt that the task must take the estimated time; if it's taking more time, it's because the worker is lazy; if it's taking less time than estimated, it's good, time to drown the guy with work. End of story.
So far, this all looks like the usual ranting about a pyramidal work organization. But here comes the real stuff: when dealing with distributing computational workload between machines, the same approach is, amazingly, actually used. And that does not make sense. Let me explain where the trouble is.
It's very difficult and unlikely, when doing some computations on a set of computers (let's call that a computing farm for big data to follow the hype) that each and every machine will have the same and exact computational power; or memory or hard storage. In real life, it's highly unlikely to happen.
And if it does, there will be different hop times between machines, network congestions happening, etc.
In a few words: the plateform is bound to be heterogeneous to a noticeable degree.
Yet, most distributed systems fail to take that into account. There is always a grand master (or, a boss, to fall back to the teamwork metaphor) which will decide everything.
Not to mention most will assume an equal RAM, link speed, etc.
That model has at its root a hidden major flaw: the boss needs to know about each and every machine status, do estimates on each one as to where to push jobs. In other word, it has to do a lot of tracking, guessing, ordering, keeping track on dead machines, etc.
What about now reversing the model: the workers ask for work. What happens then ?
The only thing for the boss to handle is to have a job queue, and keep track of dead tasks. Workers keep on asking for jobs.
And that's it. The work is consumed at the full available computing power with little managerial tasks. The fine tuning boils down to finding the right slice size of work.
This model is resilient and simple. Works well with multithreading too.
So, pile up tasks, and have your workers ask for the next task when they're done.
Now, as some of you might have noticed, there are a few glitches to that scenario. The boss most of the time does not correctly estimate the time needed for a task if he does that only on his own. If he asks, it means hours of meetings. Not very efficient.
From the boss' view, there was no doubt that the task must take the estimated time; if it's taking more time, it's because the worker is lazy; if it's taking less time than estimated, it's good, time to drown the guy with work. End of story.
So far, this all looks like the usual ranting about a pyramidal work organization. But here comes the real stuff: when dealing with distributing computational workload between machines, the same approach is, amazingly, actually used. And that does not make sense. Let me explain where the trouble is.
It's very difficult and unlikely, when doing some computations on a set of computers (let's call that a computing farm for big data to follow the hype) that each and every machine will have the same and exact computational power; or memory or hard storage. In real life, it's highly unlikely to happen.
And if it does, there will be different hop times between machines, network congestions happening, etc.
In a few words: the plateform is bound to be heterogeneous to a noticeable degree.
Yet, most distributed systems fail to take that into account. There is always a grand master (or, a boss, to fall back to the teamwork metaphor) which will decide everything.
Not to mention most will assume an equal RAM, link speed, etc.
That model has at its root a hidden major flaw: the boss needs to know about each and every machine status, do estimates on each one as to where to push jobs. In other word, it has to do a lot of tracking, guessing, ordering, keeping track on dead machines, etc.
What about now reversing the model: the workers ask for work. What happens then ?
The only thing for the boss to handle is to have a job queue, and keep track of dead tasks. Workers keep on asking for jobs.
And that's it. The work is consumed at the full available computing power with little managerial tasks. The fine tuning boils down to finding the right slice size of work.
This model is resilient and simple. Works well with multithreading too.
So, pile up tasks, and have your workers ask for the next task when they're done.
A simple libevent http get server example
I recently tried to serve json queries in the "lightest" way, with a simple C program, with as little overhead as possible (no fcgi module, no scripting, standalone).
It happens that libevent has already all one need to do just that: answer http queries with as little overhead as possible, laid on an efficient networking library.
The trouble is, there is some documentation, but not that much, and a few pieces are missing to write a simple GET http server.
So, here is a skeleton for a simple GET http server with libevent.
It's handy for a very simple GET json server for instance.
Just in case that's of any help.
It happens that libevent has already all one need to do just that: answer http queries with as little overhead as possible, laid on an efficient networking library.
The trouble is, there is some documentation, but not that much, and a few pieces are missing to write a simple GET http server.
So, here is a skeleton for a simple GET http server with libevent.
It's handy for a very simple GET json server for instance.
Just in case that's of any help.
2013/08/19
Tips to build a corpus for statistical classification faster
Building a corpus for statical classification is usually a tedious task. I will try to give you here a few ideas to get started to help speed that up.
As is usual with any optimization work: the fastest way to do something is not to do it at all.
Most of the time, the task has already been tackled, and with some chance, someone already built a corpus and made it available. The chances may seem small, but it's usually more common than one could expect.
So, the first step is to use your favorite search engine, try different keywords, and look for that. If you can't find it, look harder, change keywords.
A ready-made corpus is not necessarily where one would look in the first place. One such example is a classic used for training statistical machine translation engines: the United Nations speeches. It has all the speeches translated in many languages (and there's a lot of speeches made there). A variant could be works at the European Parliament, which are also translated to different languages. You get the idea.
A "new" trove of human knowledge is Wikipedia. This may seem obvious, but until recently it was quite unusable, as it was poorly structured, if at all. This has changed, and, though minimalistic in many ways, the "infobox" tag for instance is gathering more and more useful information.
It is for those who are already a bit experienced with coping with size though (or have a lot of patience), as the English Wikipedia dump is, as of this writing, 8.3G bzipped (that's something above 40G). (Mahout has a nice ready-made splitter for the Wikipedia dump, here is a quick introduction to the stuff, it's really useful to have a smaller set for debugging for instance)
Now, if that's not enough to get the bulk human information you need, things are getting tricky.
The usual not subtle path of doing the stuff could be then:
- write a quick viewer which will allow you to easily classify content
- befriend grep and pcre
- building a set of "good" keywords
- being even more creative
There's also the possibility to seed your classification engine with a little data, then run it on some more data, check what is a positive and what is not in your corpus yet, and then add it.
That is _not_ a good idea, as it will give a bias to your corpus. However, this can be mitigated by ignoring the best matching data, and targeting the "more or less" matching data" according to your classifier. You better check the maths though !
Have more tips ? Share them in the comments :)
As is usual with any optimization work: the fastest way to do something is not to do it at all.
Most of the time, the task has already been tackled, and with some chance, someone already built a corpus and made it available. The chances may seem small, but it's usually more common than one could expect.
So, the first step is to use your favorite search engine, try different keywords, and look for that. If you can't find it, look harder, change keywords.
A ready-made corpus is not necessarily where one would look in the first place. One such example is a classic used for training statistical machine translation engines: the United Nations speeches. It has all the speeches translated in many languages (and there's a lot of speeches made there). A variant could be works at the European Parliament, which are also translated to different languages. You get the idea.
A "new" trove of human knowledge is Wikipedia. This may seem obvious, but until recently it was quite unusable, as it was poorly structured, if at all. This has changed, and, though minimalistic in many ways, the "infobox" tag for instance is gathering more and more useful information.
It is for those who are already a bit experienced with coping with size though (or have a lot of patience), as the English Wikipedia dump is, as of this writing, 8.3G bzipped (that's something above 40G). (Mahout has a nice ready-made splitter for the Wikipedia dump, here is a quick introduction to the stuff, it's really useful to have a smaller set for debugging for instance)
Now, if that's not enough to get the bulk human information you need, things are getting tricky.
The usual not subtle path of doing the stuff could be then:
- write a quick viewer which will allow you to easily classify content
- befriend grep and pcre
- building a set of "good" keywords
- being even more creative
There's also the possibility to seed your classification engine with a little data, then run it on some more data, check what is a positive and what is not in your corpus yet, and then add it.
That is _not_ a good idea, as it will give a bias to your corpus. However, this can be mitigated by ignoring the best matching data, and targeting the "more or less" matching data" according to your classifier. You better check the maths though !
Have more tips ? Share them in the comments :)
2013/07/24
How to get zendx_jquery_form_element_autocomplete to respond in real time
Were I work for a living now, they use Zend Framework and zendx_jquery_form_element_autocomplete. It's slow, and definitely not responding in real time. A quick test with Firebug shows a median response time of 300 ms. Far from the 200 ms needed to make it appear instantaneous.
Everyone says the database is slow, I didn't take it further at first.
But, yes, the whole thing is slow. Something was missing an index, which I added, but the time gains were very slim. I found that suspicious. It's not a big deal, it's an application with few users, so I concentrated on the auto-complete part, which is worth optimizing.
The index was right, the code was ok, and it took 300 ms. It has to be the database. Or slow machine executing the code. I blindly tried xcache, it saved about 100 ms. Or, well, there was something else going on.
I decided to bypass everything and wrote a very simple script, doing the bare query, printing in json the result array, along the line of this:
And I ran the thing in Firebug. Tadam... It suddenly took only 6 ms. Right, that's a 50 fold time gain.
Is it clean ? No, not really. Is Zend Framework slowing your applications ? Well, now you know how to deal with that one issue which matters.
Everyone says the database is slow, I didn't take it further at first.
But, yes, the whole thing is slow. Something was missing an index, which I added, but the time gains were very slim. I found that suspicious. It's not a big deal, it's an application with few users, so I concentrated on the auto-complete part, which is worth optimizing.
The index was right, the code was ok, and it took 300 ms. It has to be the database. Or slow machine executing the code. I blindly tried xcache, it saved about 100 ms. Or, well, there was something else going on.
I decided to bypass everything and wrote a very simple script, doing the bare query, printing in json the result array, along the line of this:
// Connection and database selection
$query = "SELECT name FROM table WHERE name LIKE '" . mysql_real_escape ($_REQUEST['term']) . "%'";
$my_res = mysql_query ($query);
$result = array ();
while ($item = mysql_fetch_array ($my_res))
$result[] = $item['name'];
print json_encode ($result);
And I ran the thing in Firebug. Tadam... It suddenly took only 6 ms. Right, that's a 50 fold time gain.
Is it clean ? No, not really. Is Zend Framework slowing your applications ? Well, now you know how to deal with that one issue which matters.
2013/07/09
Building a corpus for classification: size isn't always the only problem
Building a meaningful corpus for classifying documents is a tedious task. Building a few is even more tedious.
That's an obvious fact.
The trouble is, as the rule stands with anything involving language: ambiguity finally rules and corners you.
Let's say, for instance, that we are trying to classify articles from newspapers, in three categories: "business", "energy" and "environment".
That seems simply enough. The next article about Lady Gaga is out. The next articles on coal prices is in. It's in. Right. It fall in the business section, as it's going to influence energy prices. Wait, energy prices, that should be also in energy then, right ? But won't coal prices influence the environment too, as renewable will be less competitive ?
Wait, once again, wouldn't classifying this in environment be biased ? What about global warming naysayers ?
What began as a simple and trivial classification finally ends up in a mess. The multiple section facet is solved fast: use that article as a reference for all three categories, never mind the confusion it will potentially give to the computer which has to decide which category the evaluated articles will fall into. But what about the implicit point of view concerning this simple example, and its potential bias ?
If that seems too obvious (and in some ways, it is), let's try another one: identifying if a search string refers to a person or anything else. It's easy to define a person, right ?
Are Greek Gods persons for instance ? One can find pictures of them, they were treated as beings and interacted quite intimately with human beings. They could be persons then.
Let's say Gods are people. Then, what about Allah, which isn't quite really a person, but definitely a God to many people ?
There's also the more trivial case of movie and tv characters: should they be classified as beings, or ?...
Once again, the brain with its seemingly endless prowesses for confusion puzzles easily the man trying to teach a thing or two to the computer.
That's an obvious fact.
The trouble is, as the rule stands with anything involving language: ambiguity finally rules and corners you.
Let's say, for instance, that we are trying to classify articles from newspapers, in three categories: "business", "energy" and "environment".
That seems simply enough. The next article about Lady Gaga is out. The next articles on coal prices is in. It's in. Right. It fall in the business section, as it's going to influence energy prices. Wait, energy prices, that should be also in energy then, right ? But won't coal prices influence the environment too, as renewable will be less competitive ?
Wait, once again, wouldn't classifying this in environment be biased ? What about global warming naysayers ?
What began as a simple and trivial classification finally ends up in a mess. The multiple section facet is solved fast: use that article as a reference for all three categories, never mind the confusion it will potentially give to the computer which has to decide which category the evaluated articles will fall into. But what about the implicit point of view concerning this simple example, and its potential bias ?
If that seems too obvious (and in some ways, it is), let's try another one: identifying if a search string refers to a person or anything else. It's easy to define a person, right ?
Are Greek Gods persons for instance ? One can find pictures of them, they were treated as beings and interacted quite intimately with human beings. They could be persons then.
Let's say Gods are people. Then, what about Allah, which isn't quite really a person, but definitely a God to many people ?
There's also the more trivial case of movie and tv characters: should they be classified as beings, or ?...
Once again, the brain with its seemingly endless prowesses for confusion puzzles easily the man trying to teach a thing or two to the computer.
2013/06/25
Searching images for portraits
Building my search engine drew me to try to index other things than raw text: images, videos, news, etc. It has been a while now since I worked on the stuff, and I recently said to myself that it's a shame not to do something useful and creative with the bunch of images that I gathered years ago.
I forgot that I had more than 10 millions images from all over the web. It would indeed be a shame not to do something with that.
Following what was done at the time, and not having so much time to look back at all the possibilities, I thought about what anyone was already doing: search by tone color, sizes, etc. Things (more or less) easily technically done, but most of the time with very little value for the user.
And then, a simple thought came to mind: what about allowing search by portraits ? As a first step towards search by shapes and objects.
It's rather simple to do nowadays: you probably have stumbled upon one of those software for webcams which track faces: that's essentially the kind of software we need to find a portrait in an image.
If there's a face, and it's big enough, it must be a portrait.
So, the first big step is being able to recognize a face on an image. That's simple enough thanks to a great library which can (among a lot of other things) do that: OpenCV. For that task, cvHaarDetectObjects is what we are looking for (it's for C, the exact function/method name which applies will vary according to your programming language of choice).
And to make things even more simple for a quick and dirty evaluation of the thing, there's for example a very basic PECL extention for PHP, PHP-Facedetect, which does exactly what we need.
The configure script being broken as the time of this writing and the git pull request waiting to be accepted, you can check out the patched extension I wrote.
That should be enough ground and ideas to get you started ! Share your ideas ! :)
I forgot that I had more than 10 millions images from all over the web. It would indeed be a shame not to do something with that.
Following what was done at the time, and not having so much time to look back at all the possibilities, I thought about what anyone was already doing: search by tone color, sizes, etc. Things (more or less) easily technically done, but most of the time with very little value for the user.
And then, a simple thought came to mind: what about allowing search by portraits ? As a first step towards search by shapes and objects.
It's rather simple to do nowadays: you probably have stumbled upon one of those software for webcams which track faces: that's essentially the kind of software we need to find a portrait in an image.
If there's a face, and it's big enough, it must be a portrait.
So, the first big step is being able to recognize a face on an image. That's simple enough thanks to a great library which can (among a lot of other things) do that: OpenCV. For that task, cvHaarDetectObjects is what we are looking for (it's for C, the exact function/method name which applies will vary according to your programming language of choice).
And to make things even more simple for a quick and dirty evaluation of the thing, there's for example a very basic PECL extention for PHP, PHP-Facedetect, which does exactly what we need.
The configure script being broken as the time of this writing and the git pull request waiting to be accepted, you can check out the patched extension I wrote.
That should be enough ground and ideas to get you started ! Share your ideas ! :)
2009/10/04
Design patterns
A while ago in the company I worked for, I was assigned a task.
Me being given that task was an error: a guy with the same first name as me was supposed to get it.
I could go on and on how that situation got managed, but this blog is about serious stuff and not about politics and egos in the workplace.
So, I got a look at the application I was supposed to maintain and to make evolve. A very nice java application, quite well written, hundreds of line of code, readable and so on. Nicely written from a design patterns point of view.
What was this application doing with its hundreds of lines of code ?
$ xslproc sheet.xsl document.xsl > output.xml
Yes, these hundred of lines of code fit into one line. What about datasource are you asking ? Well, everything is a file in unix...
Don't get me wrong, there are a lot of things I hate about the Unix way, but...
Please, let's keep things as simple as they are... And you know what ? They are usually more efficient and as powerful when they are kept simple...
Me being given that task was an error: a guy with the same first name as me was supposed to get it.
I could go on and on how that situation got managed, but this blog is about serious stuff and not about politics and egos in the workplace.
So, I got a look at the application I was supposed to maintain and to make evolve. A very nice java application, quite well written, hundreds of line of code, readable and so on. Nicely written from a design patterns point of view.
What was this application doing with its hundreds of lines of code ?
$ xslproc sheet.xsl document.xsl > output.xml
Yes, these hundred of lines of code fit into one line. What about datasource are you asking ? Well, everything is a file in unix...
Don't get me wrong, there are a lot of things I hate about the Unix way, but...
Please, let's keep things as simple as they are... And you know what ? They are usually more efficient and as powerful when they are kept simple...
2009/08/13
Ubuntu and "large files" (files greater than 2 G)
Recently, for whatever reason, I decided that I should have Linux on a few machines. It might have been to be sure that my tools run on Linux.
It happened that Linux ended up on one of the key machines.
That was pure over-confidence and trust.
If FreeBSD hasn't any problem with "big files", that is, files greater that 2 gigabytes, something one could expect in 2009, one could expect the same from Linux.
Well, not quite so. The spider got stuck around 2 G of data.
The two magics options to add at compile time before banging your heads on the walls (it does hurt, trust my experience):
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
Why does one need to add these defines ? (which trigger an ugly cascade of other defines, it seems a lot of people decided this needs to be frightening)
I have no idea why.
Anyway, that's two flags one needs to add.
It happened that Linux ended up on one of the key machines.
That was pure over-confidence and trust.
If FreeBSD hasn't any problem with "big files", that is, files greater that 2 gigabytes, something one could expect in 2009, one could expect the same from Linux.
Well, not quite so. The spider got stuck around 2 G of data.
The two magics options to add at compile time before banging your heads on the walls (it does hurt, trust my experience):
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
Why does one need to add these defines ? (which trigger an ugly cascade of other defines, it seems a lot of people decided this needs to be frightening)
I have no idea why.
Anyway, that's two flags one needs to add.
2009/01/18
Error
An interesting thing to feed a search engine is the word "error".
It's a very simple, single word.
You say, a good search engine will know how to handle all those "mysql error", "php error", "error page not found", etc etc.
Well, it's pretty interesting to see how well each behaves to that.
Some just give you in their top results "Error! Reason: File 'index.html was not found!" (right, this is the second result on a popular French engines). Some do it as their top result (Error. Reason: File "menu.asp" was not found!)
Others will play it on the safe side, displaying news that mentions error, the wikipedia error entry, then, the algorithm seeing that the remaining thousands of pages are wrong show only a handful of results.
I found one search engine that seemed less prone to this kind of troubles that the major search engine we know.
One the second hand, my search engine seems to be affected a lot later than the other search engines. That's suspicious, considering it's unlikely I'm the only one to run and try to fix that trouble. The real answer is... well, it doesn't show very well the content of the page, and where that "error" comes from.
Is this question relevant in any way ? It's the traditional "the written words don't really mean what the website is about".
Even if we are using the link text from web page to web page, there's going to be guys saying "this pages yields an error" or something like that.
One can say that, well, if enough guys say something different about the page, use what the majority is saying; true, but if you are looking for fast raising page per words...
I don't have an answer to that; it looks like I'm not the only one, and that's not really a comforting idea as to the future of search.
It's a very simple, single word.
You say, a good search engine will know how to handle all those "mysql error", "php error", "error page not found", etc etc.
Well, it's pretty interesting to see how well each behaves to that.
Some just give you in their top results "Error! Reason: File 'index.html was not found!" (right, this is the second result on a popular French engines). Some do it as their top result (Error. Reason: File "menu.asp" was not found!)
Others will play it on the safe side, displaying news that mentions error, the wikipedia error entry, then, the algorithm seeing that the remaining thousands of pages are wrong show only a handful of results.
I found one search engine that seemed less prone to this kind of troubles that the major search engine we know.
One the second hand, my search engine seems to be affected a lot later than the other search engines. That's suspicious, considering it's unlikely I'm the only one to run and try to fix that trouble. The real answer is... well, it doesn't show very well the content of the page, and where that "error" comes from.
Is this question relevant in any way ? It's the traditional "the written words don't really mean what the website is about".
Even if we are using the link text from web page to web page, there's going to be guys saying "this pages yields an error" or something like that.
One can say that, well, if enough guys say something different about the page, use what the majority is saying; true, but if you are looking for fast raising page per words...
I don't have an answer to that; it looks like I'm not the only one, and that's not really a comforting idea as to the future of search.
2009/01/06
Security concepts and an open source search engine
I was reading tonight a very long list of comments on how an ideal distributed opensource search engine could be.
The interesting things, reading the comments, is how it relates to security. Let me explain.
The main argument on why an opensource (even more so a distributed one) search engine can't work in practice is because when you know how the thing works, you can easily influence the results (ie: spam). And then people begin to praise the "security through obscurity" that the major search engines have: it's, according to them, the best way to preserve security.
No need to say, this is wrong. If it wasn't so, big companies wouldn't be spending money optimizing their ranking, especially if that wasn't working at all. Even if you consider the "moron factor", it's too easy to see if it's effective: run a search and see if you are on the first page.
So, obviously, even for ranking, security through obscurity doesn't work.
As a reminder, the most widely used library for secure communication, openssl, which source code is widely available, which encryption algorithm are know, isn't (officially at least), easily cracked. True, there's a lot of money involved in being on the major search engines first page, and people are desperate to get there. It's true too that brilliant guys do spend their days trying to break that openssl thing.
So, maybe that's one of the accurate goals for the next big search engine: a ranking algorithm that can't be diverted, even if you precisely know what the algorithm is.
The interesting things, reading the comments, is how it relates to security. Let me explain.
The main argument on why an opensource (even more so a distributed one) search engine can't work in practice is because when you know how the thing works, you can easily influence the results (ie: spam). And then people begin to praise the "security through obscurity" that the major search engines have: it's, according to them, the best way to preserve security.
No need to say, this is wrong. If it wasn't so, big companies wouldn't be spending money optimizing their ranking, especially if that wasn't working at all. Even if you consider the "moron factor", it's too easy to see if it's effective: run a search and see if you are on the first page.
So, obviously, even for ranking, security through obscurity doesn't work.
As a reminder, the most widely used library for secure communication, openssl, which source code is widely available, which encryption algorithm are know, isn't (officially at least), easily cracked. True, there's a lot of money involved in being on the major search engines first page, and people are desperate to get there. It's true too that brilliant guys do spend their days trying to break that openssl thing.
So, maybe that's one of the accurate goals for the next big search engine: a ranking algorithm that can't be diverted, even if you precisely know what the algorithm is.
2008/11/29
Encoding
A friend of mine reminded me tonight about something obvious I had to deal with: encodings. Let's recap'.
First, the Computer made ASCII and ANSI. And the Computer saw it was good.
And the Computer said "Let there be KOI", and there was KOI.
And the Computer saw KOI was good. And the Computer separated KOI from the others. (...) And the Computer said "Let there be BIG-5 in the midst of the sea of encodings and let it separate the encoding from the dark encodings". And the Computer made the firmaments and separated the encodings which where under firmament from the encodings which where above the firmaments.(...)
And the Computer saw everything he had made, and behold, it was very good. And there was evening, and there was morning, a sixth day. (...)
To make a long story short, at one point, thanks to the Computer's in-depth look and long term view, we get to the tale of the tower of Babel.
And then, well, we get to the search engine. It's bad enough already that websites all around the earth will use different encodings, but, to make matters worse, everyone seem to pretend their encoding is obviously the right one.
And that's where the "obvious" seriously gets in the way. How is one supposed to know that a Russian website hosted in the US is using the latin1 encoding ? Or that a Korean website hosted in Japan is using iso-8859-1 encoding ?
In case you think that it's easy, consider that the page is advertising another impossible encoding.
Do you think I'm over-doing it ?
I have 7 million pages for you.
Anyone who has the generic good answer for that one gets a free beer on me. (International shipping is ok). And, no, dropping the pages which are that crazy is not the satisfactory answer.
First, the Computer made ASCII and ANSI. And the Computer saw it was good.
And the Computer said "Let there be KOI", and there was KOI.
And the Computer saw KOI was good. And the Computer separated KOI from the others. (...) And the Computer said "Let there be BIG-5 in the midst of the sea of encodings and let it separate the encoding from the dark encodings". And the Computer made the firmaments and separated the encodings which where under firmament from the encodings which where above the firmaments.(...)
And the Computer saw everything he had made, and behold, it was very good. And there was evening, and there was morning, a sixth day. (...)
To make a long story short, at one point, thanks to the Computer's in-depth look and long term view, we get to the tale of the tower of Babel.
And then, well, we get to the search engine. It's bad enough already that websites all around the earth will use different encodings, but, to make matters worse, everyone seem to pretend their encoding is obviously the right one.
And that's where the "obvious" seriously gets in the way. How is one supposed to know that a Russian website hosted in the US is using the latin1 encoding ? Or that a Korean website hosted in Japan is using iso-8859-1 encoding ?
In case you think that it's easy, consider that the page is advertising another impossible encoding.
Do you think I'm over-doing it ?
I have 7 million pages for you.
Anyone who has the generic good answer for that one gets a free beer on me. (International shipping is ok). And, no, dropping the pages which are that crazy is not the satisfactory answer.
2008/11/15
Thank you !
I was very surprise of the very warm welcome our presentation received and how many question it sparked, as well as encouragements.
Thank you all for this :)
The other presentations were really interesting, and we were nervous to be the ones speaking after a great presentation about The New York Times, and how they so embrace the web.
You will see me at the next Ignite session for sure :)
Thank you all for this :)
The other presentations were really interesting, and we were nervous to be the ones speaking after a great presentation about The New York Times, and how they so embrace the web.
You will see me at the next Ignite session for sure :)
2008/11/13
OsO @ Ignite Paris #3
Nicolas Toper and I will be giving a short presentation on OsO in Paris for the third Ignite event there (more information here).
The rules are to give a presentation with 20 slides during exactly 5 minutes. The presentation can be about anything (geeky), so we chose to do ours on my search engine and some of the hard lessons learned.
You can find the slides here to get a feeling of what we are going to talk about.
The rules are to give a presentation with 20 slides during exactly 5 minutes. The presentation can be about anything (geeky), so we chose to do ours on my search engine and some of the hard lessons learned.
You can find the slides here to get a feeling of what we are going to talk about.
2008/07/23
By the Book (of law)
A few weeks ago, I was quite excited about a "good idea" I had.
I spent some time on alexa, and looked at what the most popular sites are. It seemed that anything that had to do with news got to the top audience.
So I said to myself, this looks like a good idea: something of interest, a "small" corpus and a lot to do with natural language processing.
My tools being quite modular, I had rapidly a news search engine at my fingers with about 40 different sources and even toyed with graphics comparing terms frequency in articles, to be used with anything from brands to politicians.
I thought to myself at first, well, I've got a nice idea (something a bit more elaborate than the usual news search engine) and it's going to be some sort of win/win strategy: the newspapers will attract readers to articles they might have missed and are interested in, the newspapers will be generating more revenue, and my idea will bring me traffic.
Remembering a few articles I read some while ago, I did a quick search on trials going on that were about this kind of tool. There are a few, for large sums of money. The Belgian press syndicate seems to refuse any link to their newspapers. It seemed quite ridiculous.
Then I took a look at the Berne Convention. A news search engine could fall in the category of "fair use", the result being "quotations from newspaper articles and periodicals in the form of press summaries" (Article 10). But it might not.
The French law for instance can be even more restrictive: displaying the number of words of an article can be considered a "transformation" (Article L122-4 of the Code de la Propriété Intellectuelle) and thus forbidden. Or showing the size of a document, as about any search engine does. And it goes on and on. Any f(document) could be illegal.
True, I could just contact every newspaper and wait for their answer. For whatever reason, I don't expect any answer.
If someone is interested in developing that project, that's great, the code is ready. I'm off to other territories for the time being, at least until a few trials come to an end.
Next time, things will be technical again.
I spent some time on alexa, and looked at what the most popular sites are. It seemed that anything that had to do with news got to the top audience.
So I said to myself, this looks like a good idea: something of interest, a "small" corpus and a lot to do with natural language processing.
My tools being quite modular, I had rapidly a news search engine at my fingers with about 40 different sources and even toyed with graphics comparing terms frequency in articles, to be used with anything from brands to politicians.
I thought to myself at first, well, I've got a nice idea (something a bit more elaborate than the usual news search engine) and it's going to be some sort of win/win strategy: the newspapers will attract readers to articles they might have missed and are interested in, the newspapers will be generating more revenue, and my idea will bring me traffic.
Remembering a few articles I read some while ago, I did a quick search on trials going on that were about this kind of tool. There are a few, for large sums of money. The Belgian press syndicate seems to refuse any link to their newspapers. It seemed quite ridiculous.
Then I took a look at the Berne Convention. A news search engine could fall in the category of "fair use", the result being "quotations from newspaper articles and periodicals in the form of press summaries" (Article 10). But it might not.
The French law for instance can be even more restrictive: displaying the number of words of an article can be considered a "transformation" (Article L122-4 of the Code de la Propriété Intellectuelle) and thus forbidden. Or showing the size of a document, as about any search engine does. And it goes on and on. Any f(document) could be illegal.
True, I could just contact every newspaper and wait for their answer. For whatever reason, I don't expect any answer.
If someone is interested in developing that project, that's great, the code is ready. I'm off to other territories for the time being, at least until a few trials come to an end.
Next time, things will be technical again.
2008/06/02
Over five million pages
The search engine now handles more than five million pages. The global performance is ok.
Stay tuned, I wish to give you exciting news in a few weeks.
Stay tuned, I wish to give you exciting news in a few weeks.
2008/02/18
Close to three million
Despite being quite busy with other projects lately, the last index update now has about 3 million pages, and response time is ok when doing a query (about half a second), but far from great. Multiple words search is now available.
A lot of work is still ahead.
A lot of work is still ahead.
2007/10/09
Update
It's been a while since my last post.
Things are going more slowly these day, although rather smoothly.
So far I have implemented the three main user functionalities I wished to have: text search, image search and video search.
Right now I'm starting to fill up my disks. About 5 million URL should do to start with and see a few things not adequately coping with that number of pages. I worked hard to be safe on all grounds, but working for years with computers taught me to expect (at least) the unexpected.
Let me give you a few numbers about the raw data. These are figures from a innacurate (emphasis intended) sample of 200 000 pages from the web.
Mean page size: 14.5 kb
Mean number of words per page: 449
Mean unique words per page: 216
Mean number of links per page: 35
Mean number of JPEG per page: 19
What do these numbers mean ? I don't think they mean a lot. I expect them to vary greatly with time.
The number of links per page for instance is suspiciously high. Refining the data in three categories of unique links: number of links in a page, number of links pointing inside the website, number of links pointing outside the website. Here two links pointing to the same page make for two links instead of one. The same applies to the JPEG.
I will try to update the stats from time to time to let you know how crude or accurate this first sample was, as well as other interesting stats.
Things are going more slowly these day, although rather smoothly.
So far I have implemented the three main user functionalities I wished to have: text search, image search and video search.
Right now I'm starting to fill up my disks. About 5 million URL should do to start with and see a few things not adequately coping with that number of pages. I worked hard to be safe on all grounds, but working for years with computers taught me to expect (at least) the unexpected.
Let me give you a few numbers about the raw data. These are figures from a innacurate (emphasis intended) sample of 200 000 pages from the web.
Mean page size: 14.5 kb
Mean number of words per page: 449
Mean unique words per page: 216
Mean number of links per page: 35
Mean number of JPEG per page: 19
What do these numbers mean ? I don't think they mean a lot. I expect them to vary greatly with time.
The number of links per page for instance is suspiciously high. Refining the data in three categories of unique links: number of links in a page, number of links pointing inside the website, number of links pointing outside the website. Here two links pointing to the same page make for two links instead of one. The same applies to the JPEG.
I will try to update the stats from time to time to let you know how crude or accurate this first sample was, as well as other interesting stats.
2007/09/19
Optimizing: things that matter
In my previous post, I explained how to optimize (broadly speaking) the initialization of a block of data.
I also mentioned that it was useless. There is at least two reasons for that:
- Initializing data can't (and shouldn't) be the costliest part of your program:
If you are doing some string processing, some computations on an array, initializing the array is unlikely the part that will require the most computer instructions. You should instead focus on optimizing your algorithms.
- do your best to never have to initialize an array to a uniform value more than once in a program's lifetime:
This may sound surprising, even unfeasible. However, if initializing happens very often in your program, then truth #1 doesn't hold anymore, and optimizing your algorithms now include optimizing the initialization: choosing an algorithm were there is no, or a very minimal need to initialize data.
This will lead us to our next article on optimizing scripting languages execution time.
I also mentioned that it was useless. There is at least two reasons for that:
- Initializing data can't (and shouldn't) be the costliest part of your program:
If you are doing some string processing, some computations on an array, initializing the array is unlikely the part that will require the most computer instructions. You should instead focus on optimizing your algorithms.
- do your best to never have to initialize an array to a uniform value more than once in a program's lifetime:
This may sound surprising, even unfeasible. However, if initializing happens very often in your program, then truth #1 doesn't hold anymore, and optimizing your algorithms now include optimizing the initialization: choosing an algorithm were there is no, or a very minimal need to initialize data.
This will lead us to our next article on optimizing scripting languages execution time.
Subscribe to:
Posts (Atom)