What does entropy H(X) mean?

Shannon entropy measures the average uncertainty of a random variable. H=0 means deterministic, maximum H = log2(n) for uniform distribution.

How is KL divergence asymmetric?

KL(P||Q) measures information lost when Q approximates P, while KL(Q||P) is generally different. Both are non-negative and zero only if P=Q.

Perplexity is 2^H (bits) or e^H (nats). It represents the weighted average number of choices a model is uncertain about.

Shannon · Cross‑entropy · KL · Mutual info · Perplexity

Probability distribution (comma‑separated, up to 20)

—

bits · nats · efficiency

H = – Σ p·log₂(p)

Interpretation

Max entropy (uniform): —

True distribution P (comma‑separated) Predicted distribution Q

—

H(P) · KL divergence

H(P,Q) = – Σ p·log₂(q)

KL(P||Q) =

P distribution Q distribution

—

asymmetry

KL(P||Q) = Σ p·log(p/q)

KL(P||Q) vs KL(Q||P)

Joint probability matrix (rows semicolon, cols comma) 2x2–4x4

—

bits

I = Σ p(x,y)·log(p(x,y)/(p(x)p(y)))

P(X) and P(Y) displayed

Enter cross‑entropy H (bits) OR token probabilities (comma)

—

2^H / e^H

PP = 2^H (bits) or e^H (nats)

If probabilities given, entropy is computed first. Perplexity = 2^H.

Recommended by our team

The #1 AI writing tool for freelancers — perfect grammar in any language, instantly.