Divergence

Basic Concepts

Divergence is a measure of the difference between two probability distributions. Often, this is used to determine the difference between a sample and a known probability distribution.

Divergence takes a non-negative value. The value is zero when the two probability distributions are equal.

We consider two measurements of divergence here.

Kullback-Leibler Divergence (KL)

Given two finite distributions p and q with corresponding elements x₁, …, x_n and y₁, …, y_n

We assume that when y_i = 0, then x_i = 0, and in this case, we set ln(x_i/y_i) = 0. Note too that it is assumed that the elements x₁, …, x_n and y₁, …, y_n are the values of a discrete probability distribution, and so each must sum to one. Thus, you may need to replace each x_i by x_i/, and similarly for the y_i.

We can also use the terminology KL(p||q) where p is the pdf whose values are x₁, …, x_n and q is the pdf whose values are y₁, …, y_n. We can view KL(p||q)  as representing the information gained by using p instead of q. From a Bayesian perspective, it measures the information gained when revising your prior beliefs from p to the posterior distribution q.

Note that KL(x||y) and KL(y||x) are not necessarily equal. The natural log can be replaced by a log of base 2.

Jensen-Shannon Divergence (JS)

Given two finite distributions p and q with corresponding elements x₁, …, x_n and y₁, …, y_n

where z_i = (x_i+y_i)/2. For this measure of divergence JS(x,y) = JS(y,x).

Examples

Example 1: Determine how well the discrete distribution displayed in column C of Figure 1 approximates the binomial distribution with n = 3 and p = .4.

Figure 1 – Kulback-Leibler divergence

The binomial distribution elements are shown in column B. E.g. cell B4 contains the formula =BINOM.DIST(A4,3,0.4,FALSE). We next insert the formula =B4*LN(B4/C4) in cell E4, highlight range E4:E7, and then press Ctrl-D. We calculate the Kulback-Leibler measure of divergence by placing the formula =SUM(E4:E7) in cell E8.

Example 2: Calculate the Jensen-Shannon divergence for the distributions from Example 1.

We first calculate the mean distribution as shown in column D of Figure 2. We then calculate KL(bin||mean) and KL(dist||mean) as shown in cells F8 and H8. Finally, JS(bin, dist) is the mean of these two values, namely .039847, as shown in cell J8.

Figure 2 – Jensen-Shannon divergence

Worksheet Functions

Real Statistics Functions: The Real Statistics Resource Pack provides the following worksheet functions where R1 and R2 are column arrays with the same number of rows.

KL_DIVERGE(R1, R2) = Kullback-Leibler divergence for R1 || R2

JS_DIVERGE(R1, R2) = Jenson-Shannon divergence for R1 and R2

E.g. the KL divergence measure for Example 1 can be calculated via the formula =KL_DIVERGE(B4:B7,C4:C7). Similarly, we can calculate the JD divergence measure for Example 2 via the formula =JS_DIVERGE(B4:B7,C4:C7).

Credit scoring divergence

There is also another measure of divergence which is used for credit scoring. Click here for more information about this type of divergence.

Links

↑ Descriptive statistics

Examples Workbook

Click here to download the Excel workbook with the examples described on this webpage.

References

Wikipedia (2023) Divergence (statistics)
https://en.wikipedia.org/wiki/Divergence_(statistics)

Brownlee, J. (2019) How to calculate the KL divergence for machine learning. Machine Learning Mastery
https://machinelearningmastery.com/divergence-between-probability-distributions/

Wikipedia (2023) Kulback-Leibler divergence
https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence