Advancing UN Comtrade for Physical Trade Flow Analysis: Addressing the Issue of Outliers

The UN Comtrade is one of the most widely used data sources for physical trade analysis. However, the issue of outliers would result in misleading interpretations and biased results, limiting its applications. Assuming that no deals would be made at unreasonable prices, we define an outlier as the data record whose unit price (trade value divided by net weight) is unusually high or low. To address the outlier issue, we develop a framework of first applying the kernel density estimation method to detect outliers and then using different statistical models to handle them based on their potential causes, then develop a deviation index to assess the impacts of outliers, and present the data quality improvement and the significance of our framework; and finally evaluate its perfor-mance by comparing with previous methods to show its outperformance on adaptability to different commod-ities' data. Our results reveal that outliers exist for almost all reporters (207 in 209, 99%), all commodities, and all years, and most outliers (92%) are with wrong net weight values. With a higher deviation index, reporters are Canada, China, France, etc., while commodities are high-price electronic products, clocks, etc. The data quality would be greatly improved by addressing the outlier issue, thus benefiting UN-based physical trade analysis.

Automated summary

UN Comtrade, a widely used data source for physical trade analysis, faces the issue of outliers that limit its application. To address this issue, researchers developed a framework using kernel density estimation and statistical models to detect and handle outliers, as well as a deviation index to assess their impacts, resulting in improved data quality.