Collaborative filtering has been the main steam engine of the recommender systems community since the early 1990s. Collaborative filtering (and other) algorithms, however, have been predominantly evaluated by aggregating results across users or user groups. These performance averages hide large disparities: an algorithm may perform very well for some users (or groups) and very poorly for others. We show that performance variation is large and systematic. In experiments on three large scale datasets and using an array of collaborative filtering algorithms, we demonstrate the large performance disparities for different users across algorithms and datasets. We then show that performance variation is systematic and that two key features that characterize users, their mean taste similarity with other users and the dispersion in taste similarity, can explain performance variation better than previously identified features. We use these two features to visualize algorithm performance for different users, and point out that this mapping can be used to capture different categories of users that have been proposed before. Our results demonstrate an extensive mainstream-taste bias in all collaborative filtering algorithms, and they imply a fundamental fairness limitation that needs to be mitigated.