Comparing weighted difference and earth mover's distance via Young diagrams

We consider two natural statistics on pairs of histograms, in which the n bins have weights 0, ... , n - 1. The signed difference (D) between the weighted totals of the histograms is, in a sense, refined by the earth mover's distance (EMD), which measures the amount of work required to equalize the histograms. We were recently surprised, however, by how little the EMD actually does refine D in certain real-world applications, which led to the main problem in this paper: what is the probability that EMD = vertical bar D vertical bar? We derive a formula for this probability, as well as the expected value of vertical bar D vertical bar, via the combinatorics of Young diagrams and plane partitions. We then generalize our results to an arbitrary number of histograms, where we realize the higher-dimensional analogue |D| as distance on the Type-A root lattice. (c) 2023 Elsevier B.V. All rights reserved.

Automated summary

The paper considers two natural statistics on pairs of histograms and derives formulas for the probability of EMD=|D| and the expected value of |D| using the combinatorics of Young diagrams and plane partitions.