I've mused a fair amount about to apply the concept of the Phetch human computation game to evaluate browsing-based information retrieval interfaces. I'd love to be able to better evaluate faceted navigation and clustering approaches, relative to conventional search as well as relative to one another.
Here is the sort of co-operative game I have in mind. It uses shopping as a scenario, and has two roles: the Shopper and the Shopping Assistant.
As a Shopper, you are presented with an shopping list and a browsing interface (i.e., you can click on links but you cannot type free text into a search box). Your goal is to find as many of the items on your shopping list as possible within a fixed time limit. In a variation of this game, not all of the items on the list are findable.
As a Shopping Assistant, you know the complete inventory, but not what the Shopper is looking for. Your goal is to help the Shopper find as many of the items on his or her shopping list as possible within a fixed time limit. On each round of interaction, you present the Shopper with information and links within the constraints of a fixed-size page. The links may include options to select items (the Shopper's ultimate goal), as well as options that show more items or modify the query.
Either role could be played by a human or a machine, and, like Phetch, the game could be made competitive by having multiple players in the same role. I'd think the interesting way to implement such a game would be with human Shoppers and algorithmic Shopping Assistants.
Is anyone aware of research along these lines? I'm hardly wed to the shopping list metaphor--it could be some other task that seems suitable for browsing-oriented interfaces.
Subscribe to:
Post Comments (Atom)
Monday, June 16, 2008
A Game to Evaluate Browsing Interfaces?
I've mused a fair amount about to apply the concept of the Phetch human computation game to evaluate browsing-based information retrieval interfaces. I'd love to be able to better evaluate faceted navigation and clustering approaches, relative to conventional search as well as relative to one another.
Here is the sort of co-operative game I have in mind. It uses shopping as a scenario, and has two roles: the Shopper and the Shopping Assistant.
As a Shopper, you are presented with an shopping list and a browsing interface (i.e., you can click on links but you cannot type free text into a search box). Your goal is to find as many of the items on your shopping list as possible within a fixed time limit. In a variation of this game, not all of the items on the list are findable.
As a Shopping Assistant, you know the complete inventory, but not what the Shopper is looking for. Your goal is to help the Shopper find as many of the items on his or her shopping list as possible within a fixed time limit. On each round of interaction, you present the Shopper with information and links within the constraints of a fixed-size page. The links may include options to select items (the Shopper's ultimate goal), as well as options that show more items or modify the query.
Either role could be played by a human or a machine, and, like Phetch, the game could be made competitive by having multiple players in the same role. I'd think the interesting way to implement such a game would be with human Shoppers and algorithmic Shopping Assistants.
Is anyone aware of research along these lines? I'm hardly wed to the shopping list metaphor--it could be some other task that seems suitable for browsing-oriented interfaces.
Here is the sort of co-operative game I have in mind. It uses shopping as a scenario, and has two roles: the Shopper and the Shopping Assistant.
As a Shopper, you are presented with an shopping list and a browsing interface (i.e., you can click on links but you cannot type free text into a search box). Your goal is to find as many of the items on your shopping list as possible within a fixed time limit. In a variation of this game, not all of the items on the list are findable.
As a Shopping Assistant, you know the complete inventory, but not what the Shopper is looking for. Your goal is to help the Shopper find as many of the items on his or her shopping list as possible within a fixed time limit. On each round of interaction, you present the Shopper with information and links within the constraints of a fixed-size page. The links may include options to select items (the Shopper's ultimate goal), as well as options that show more items or modify the query.
Either role could be played by a human or a machine, and, like Phetch, the game could be made competitive by having multiple players in the same role. I'd think the interesting way to implement such a game would be with human Shoppers and algorithmic Shopping Assistants.
Is anyone aware of research along these lines? I'm hardly wed to the shopping list metaphor--it could be some other task that seems suitable for browsing-oriented interfaces.
2 comments:
-
Panos Ipeirotis said...
-
Wisam Dakka implemented such games to evaluate different faceted interfaces as part of his PhD thesis. The goal of the user was to find a picture that was matching a description (e.g., children playing with dogs while running on a green field), or find where a picture is stored in a hierarchy. The users were getting bonuses for accomplishing these goals as fast as possible.
Of course, I do not know how much *fun* these games were. (Luis von Anh said that he rejects many games because users do not consider them fun.) However, for the purpose of making sure that the users are doing their best, it was a good incentive. (If I remember correctly, users were getting extra monetary bonus proportional to the "time left", so they tried to complete each task as soon as possible.) -
June 16, 2008 at 3:35 PM
-
Daniel Tunkelang said...
-
Now I'm really sad that I missed Wisam's defense. Panos, could you post a pointer to his dissertation?
As for making it a fun game, I suspect that a "shopping mania" game could be fun, particularly if it has a game show feeling. But your and Luis's point is well taken. -
June 16, 2008 at 4:02 PM
Subscribe to:
Post Comments (Atom)
Monday, June 16, 2008
A Game to Evaluate Browsing Interfaces?
I've mused a fair amount about to apply the concept of the Phetch human computation game to evaluate browsing-based information retrieval interfaces. I'd love to be able to better evaluate faceted navigation and clustering approaches, relative to conventional search as well as relative to one another.
Here is the sort of co-operative game I have in mind. It uses shopping as a scenario, and has two roles: the Shopper and the Shopping Assistant.
As a Shopper, you are presented with an shopping list and a browsing interface (i.e., you can click on links but you cannot type free text into a search box). Your goal is to find as many of the items on your shopping list as possible within a fixed time limit. In a variation of this game, not all of the items on the list are findable.
As a Shopping Assistant, you know the complete inventory, but not what the Shopper is looking for. Your goal is to help the Shopper find as many of the items on his or her shopping list as possible within a fixed time limit. On each round of interaction, you present the Shopper with information and links within the constraints of a fixed-size page. The links may include options to select items (the Shopper's ultimate goal), as well as options that show more items or modify the query.
Either role could be played by a human or a machine, and, like Phetch, the game could be made competitive by having multiple players in the same role. I'd think the interesting way to implement such a game would be with human Shoppers and algorithmic Shopping Assistants.
Is anyone aware of research along these lines? I'm hardly wed to the shopping list metaphor--it could be some other task that seems suitable for browsing-oriented interfaces.
Here is the sort of co-operative game I have in mind. It uses shopping as a scenario, and has two roles: the Shopper and the Shopping Assistant.
As a Shopper, you are presented with an shopping list and a browsing interface (i.e., you can click on links but you cannot type free text into a search box). Your goal is to find as many of the items on your shopping list as possible within a fixed time limit. In a variation of this game, not all of the items on the list are findable.
As a Shopping Assistant, you know the complete inventory, but not what the Shopper is looking for. Your goal is to help the Shopper find as many of the items on his or her shopping list as possible within a fixed time limit. On each round of interaction, you present the Shopper with information and links within the constraints of a fixed-size page. The links may include options to select items (the Shopper's ultimate goal), as well as options that show more items or modify the query.
Either role could be played by a human or a machine, and, like Phetch, the game could be made competitive by having multiple players in the same role. I'd think the interesting way to implement such a game would be with human Shoppers and algorithmic Shopping Assistants.
Is anyone aware of research along these lines? I'm hardly wed to the shopping list metaphor--it could be some other task that seems suitable for browsing-oriented interfaces.
2 comments:
-
Panos Ipeirotis said...
-
Wisam Dakka implemented such games to evaluate different faceted interfaces as part of his PhD thesis. The goal of the user was to find a picture that was matching a description (e.g., children playing with dogs while running on a green field), or find where a picture is stored in a hierarchy. The users were getting bonuses for accomplishing these goals as fast as possible.
Of course, I do not know how much *fun* these games were. (Luis von Anh said that he rejects many games because users do not consider them fun.) However, for the purpose of making sure that the users are doing their best, it was a good incentive. (If I remember correctly, users were getting extra monetary bonus proportional to the "time left", so they tried to complete each task as soon as possible.) -
June 16, 2008 at 3:35 PM
-
Daniel Tunkelang said...
-
Now I'm really sad that I missed Wisam's defense. Panos, could you post a pointer to his dissertation?
As for making it a fun game, I suspect that a "shopping mania" game could be fun, particularly if it has a game show feeling. But your and Luis's point is well taken. -
June 16, 2008 at 4:02 PM
Subscribe to:
Post Comments (Atom)
2 comments:
Wisam Dakka implemented such games to evaluate different faceted interfaces as part of his PhD thesis. The goal of the user was to find a picture that was matching a description (e.g., children playing with dogs while running on a green field), or find where a picture is stored in a hierarchy. The users were getting bonuses for accomplishing these goals as fast as possible.
Of course, I do not know how much *fun* these games were. (Luis von Anh said that he rejects many games because users do not consider them fun.) However, for the purpose of making sure that the users are doing their best, it was a good incentive. (If I remember correctly, users were getting extra monetary bonus proportional to the "time left", so they tried to complete each task as soon as possible.)
Now I'm really sad that I missed Wisam's defense. Panos, could you post a pointer to his dissertation?
As for making it a fun game, I suspect that a "shopping mania" game could be fun, particularly if it has a game show feeling. But your and Luis's point is well taken.
Post a Comment