Recommendation systems in large-scale online marketplaces are essential to aiding users in discovering new content. However, state-of-the-art systems for item-to-item recommendation tasks are often based on a shallow level of contextual relevance, which can make the system insufficient for tasks where item relationships are more nuanced. Contextually relevant item pairs can sometimes have controversial or problematic relationships, and they could degrade user experiences and brand perception when recommended to users. For example, a recommendation of a divorce and co-parenting book can create a disturbing experience for someone who is downloading or viewing a marriage therapy book. In this paper, we propose a classifier to identify and prevent such problematic item-to-item recommendations and to enhance overall user experiences. The proposed approach utilizes active learning to sample hard examples effectively across sensitive item categories and uses human raters for data labeling. We also perform offline experiments to demonstrate the efficacy of this system for identifying and filtering controversial recommendations while maintaining recommendation quality.