For now, relative entropy can be thought of as a measure of discrepancy between two probability distributions. We will soon see that it is central to information theory.

That’s because we used the fact that the relative entropy D1p k q o is separable in the scalar setting; it is a sum of two-variate functions. Thus we could establish convexity in two …

We will soon show that relative entropy is always non-negative and is zero if and only if p = q. However, it is not a true distance between distributions since it is not symmetric and does not …

In this section we introduce two related concepts: relative entropy and mutual information. The relative entropy is a measure of the distance between two distribu-tions.

In this lecture, we will show some properties of the entropy and we will define some related quantities that will be useful to analyze information-processing systems.

Section 28.2 describes relative entropy, or Kullback-Leibler di-vergence, which measures the discrepancy between two probability distributions, and from which Shannon entropy can be …

entropic quantities derived from the quantum relative entropy. In order to achieve our goal of re ning these optimal rates, we therefore need to consider relative entropies