An Evaluation and Annotation Methodology for Product Category Matching in E-Commerce

Product category matching is an important task in digital marketplaces and e-commerce, helping to power better search and recommendations in an online context. While variants of the problem have received some attention in academia, there is no documented guidance on how to efficiently acquire annotations for evaluating multiple (current and future) models, many of which rely on modern machine learning techniques such as neural representation learning. In this paper, we motivate and formalize the problem of product category matching in e-commerce, and present a rigorously designed set of guidelines and methodology for acquiring annotations in a cost-effective and reliable manner. We also present a methodology for using the annotations to compare solutions of two or more product category matching methods, including comparing models both before and after annotation. Three widely used e-commerce product category taxonomies, and multiple metrics, are used to demonstrate the utility of our proposals.