Overcoming the lag
June 14th, 2007
One of the most frustrating aspects of federating out a search process is the lag time in receiving the results back from the various query targets (especially with, but not limited to, Z39.50 queries). Ideally, at some point in time, there will be an infrastructure service made available which harvests from all of our content providers, and provides us the ability to work with locally (or near-locally) indexed data instead of having to bootstrap via federated querying.
Until that time, however, we need to be pragmatic in order to provide the best user experience possible. Up through LF 0.8 (yes, 0.8 is not quite released yet, but will be very shortly), our approach with the client / server interaction has been to hold open an http request until we can return the full set of search results to the user. Since we are post-processing the results to improve the relevancy, we need to wait until all of the query targets have returned their results before we can provide the user their results. This, in turn, creates open http processes that last way too long - sometimes up to 30 seconds.
So, how to improve on this? In truth, we have no control over the response time of the remote query targets - we cannot make them quicker. Our only approach with these targets is a longer-term effort to work with the providers to allow us better access to their servers or their data (sometimes through an XML gateway instead of Z39.50; optimally through harvesting via OAI-PMH). Since we cannot improve the efficiency of the queries we farm out to remote targets, we need to focus on how to minimize this effect on our users. One method we currently employ to help with this is search result caching - we use a tiered caching system to enable users who happen to enter a search that some other user had previously performed to take advantage of that fact of get their results quite quickly - by caching the results of the first instance of the search, we do not need to go out an query the remote targets again.
Caching works best within user sessions, when a user is more likely to replicate a search precisely. It is less effective across user sessions, because users need to utilize the same precise search terms for it to work. So, caching is no panacea when it comes to reducing the lag, though it does improve the situation.
One of our big changes in the 0.9 branch will be moving away from the persistent http request and instead implementing a ‘constant ping’ approach. Not only will this improve efficiencies from the LF server end (holding open http requests is extremely inefficient, especially with Rails), but it will give us greater flexibility in using the user interface to minimize the impact of slow target responses. We will be able to better inform the user of the status of their search, and we can even provide the user the ability to stop a search mid-stream and look at the results that are currently present. It also opens the door to different UI approaches - for instance, we could more easily devise an A9-style UI, if desired.
None of the above is a silver bullet solution for dealing with the lag - having data available for local or near-local indexing and access is still going to be more effective and efficient than federating a search out across various provider targets, but there currently isn’t the infrastructure in place to support such a scenario. Until then, a combination of bootstrap approaches will continue to improve on the experience LF provides users.
Posted by jaf