tag:blogger.com,1999:blog-8016696494330504473.post6947152143741044369..comments2008-09-16T17:25:47.335-04:00Comments on Redirecting to http://thenoisychannel.com...: Thinking about IR EvaluationDaniel Tunkelanghttp://www.blogger.com/profile/10240432137428080022noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-8016696494330504473.post-59758309072530587572008-05-14T22:39:00.000-04:002008-05-14T22:39:00.000-04:00Indeed, I think diversity of results is important ...Indeed, I think diversity of results is important and underemphasized by today's search engines, at least based on what I infer from personal experience. Interfaces that promote query refinement (e.g., faceted search, clustering) may offer a more diverse or risk-mitigating experience. And formal models like MMR or Zhai/Lafferty are certainly aiming for the benefits of diversity.<BR/><BR/>But Chen and Karger, who restrict themselves to returning a list of results rather than suggesting query refinements, aren't just talking about diversity. The measure they propose, k-call at n is binary: the measure returns 1 if at least k of the top n results are relevant, 0 otherwise. Hence, at k=1, an algorithm does well to return a diverse set of results, in the hopes that at least one will be relevant. But at k=n, the algorithm does better to return a homogeneous set of results, at least if Van Rijsbergen's cluster hypothesis holds.Daniel Tunkelanghttps://www.blogger.com/profile/10240432137428080022noreply@blogger.comtag:blogger.com,1999:blog-8016696494330504473.post-61724359119624382702008-05-14T09:49:00.000-04:002008-05-14T09:49:00.000-04:00those of you interested in formal models justifyin...those of you interested in formal models justifying diversity should look at Hal Varian's <A HREF="http://www.sims.berkeley.edu/~hal/Papers/sigir/sigir.pdf" REL="nofollow">"Economics of Search"</A> talk from SIGIR or Zhai/Lafferty's <A HREF="http://authors.elsevier.com/sd/article/S0306457304001530" REL="nofollow">risk minimization framework for IR</A>.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8016696494330504473.post-83527078147980759162008-05-14T04:30:00.000-04:002008-05-14T04:30:00.000-04:00sorry - correction - gary's student had a poster a...sorry - correction - gary's student had a poster at sigir07.Max L. Wilsonhttps://www.blogger.com/profile/06255892256341371135noreply@blogger.comtag:blogger.com,1999:blog-8016696494330504473.post-64833735284771254392008-05-14T04:29:00.000-04:002008-05-14T04:29:00.000-04:00This concept of diversity in search results, i bel...This concept of diversity in search results, i believe, is especially important in faceted search, as every result item it <I>usually</I> equally weighted to a facet value. so when you make a selecetion in a facet, every result is equally relevant. so - how do we organise the results?<BR/><BR/>There was a paper at SIGIR07 by gary marchionini's student that was showing how they had created a matrix of how similar/different each result was to every other result in the list. Sadly it didnt show WHY it was different.<BR/><BR/>but lets take that WHY for a second. if the results list showed what made each result novel according to the rest, then could we more accurately choose what was important to show in the per-result summaries? 'if you choose this one - you find this information that is not in the rest of the results'. it could not only affect which text snippit to show, but which other facet-values (in other facets) it belongs to that makes it unique etc.<BR/><BR/>id love to see a decent representation of novelty in a results set - id be doing it if i had the time/resources. there are some people working on novelty - david losada <A HREF="http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VC8-4P77GDP-2&_user=126770&_coverDate=03%2F31%2F2008&_rdoc=15&_fmt=high&_orig=browse&_srch=doc-info(%23toc%235948%232008%23999559997%23681288%23FLA%23display%23Volume)&_cdi=5948&_sort=d&_docanchor=&_ct=42&_acct=C000010399&_version=1&_urlVersion=0&_userid=126770&md5=f6b29fb6fc2ca3360dcc87a6a4ff6704" REL="nofollow">in this IPandM article</A> appear to be talking about this sort of topic for example. i havent read the paper yet though.Max L. Wilsonhttps://www.blogger.com/profile/06255892256341371135noreply@blogger.comtag:blogger.com,1999:blog-8016696494330504473.post-64626629854470675102008-05-13T23:46:00.000-04:002008-05-13T23:46:00.000-04:00Diversity seems to be an increasingly important me...Diversity seems to be an increasingly important metric for many techniques that return a ranked list of results: given that we cannot have perfect personalization and that we cannot figure out exactly what the user wants to see, let's offer a variety of different results, and let the user pick. We may not get all the results right, but we will have something for everyone. <BR/><BR/>It is very close to the idea of faceted search (or guided summarization that you mentioned in an earlier post). But instead of exposing the diversity of the results using multiple orthogonal browsing components, you try to "embed" diversity in the ranked list. (Or, even better, you expose both the facets and generate diverse results.)<BR/><BR/>I also like to connect this idea (conceptually) to the ideas of minimizing risk in finance: In Finance we know that stocks have high expected performance. However, <BR/>they tend to go all up or all down together (high correlation). Therefore you want to mix these with some uncorrelated or anti-correlated investments (commodities, bonds), so that you can have slightly lower expected performance, but much lower risk of complete failure.<BR/><BR/>Perhaps we should start having risk-sensitive evaluations in IR, in the same way that people in Finance measure the "value at risk" in investment portfolios?Panos Ipeirotishttps://www.blogger.com/profile/15283752183704062501noreply@blogger.com