Web-collaborative filtering: recommending music by crawling the Web

We show that it is possible to collect data that are useful for collaborative filtering (CF) using an autonomous Web spider. In CE entities are recommended to a new user based on the stated preferences of other, similar users. We describe a CF spider that collects from the Web lists of semantically related entities. These lists can then be used by existing CF algorithms by encoding them as 'pseudo-users'. Importantly the spider can collect useful data without pre-programmed knowledge about the format of particular pages or particular sites. Instead, the CF spider uses commercial Web-search engines to find pages likely to contain lists in the domain of interest, and then applies previously proposed heuristics to extract lists from these pages. We show that data collected by this spider are nearly as effective for CF as data collected from real users, and more effective than data collected by two plausible hand-programmed spiders. In some cases, autonomously spidered data can also be combined with actual user data to improve performance. (C) 2000 Published by Elsevier Science B.V. All rights reserved.