Pointwise Mutual Information & Entropy Calculator

A rare pair can have striking pointwise mutual information while contributing little to the information shared by two variables. Enter overlapping event counts or a labeled categorical count table to explore that distinction. Compare observed counts with independence expectations, select heatmap cells, inspect signed contributions and see how shared information relates to remaining uncertainty. The full cell ledger and downloadable CSV retain zero-count and undefined states explicitly. Results describe the empirical distribution you supply; they do not establish statistical significance, predict unseen observations or demonstrate causation.

Categorical association & information

See the difference between a striking pair and shared information

Explore pointwise association, the cells driving it, and how much uncertainty two categorical variables share.

Counts must describe one common observation universe. PMI measures an individual pair; MI averages signed PMI across the full distribution. These empirical scores do not establish significance or causation. No smoothing is applied.

Describe your observations

Must be at least max(0, X + Y − N) and at most min(X, Y).

The complete 2 × 2 table includes both, X only, Y only and neither. Use a consistent definition of one observation—for example, one document or one eligible word-pair window.

Updated: 100 observations; shared information 0.09127745 bits.

Shared information (MI)

0.09127745

bits

X entropy

0.9709506

bits

Y entropy

0.8812909

bits

Observations

100

common universe

Where the association lives

Select a cell to inspect its counts and probabilities. Positive association is blue (+), negative is amber (−), zero is neutral, and undefined cells have a dashed border. Values and signs carry the meaning independently of color.

PMI in bits. Row and column categories share the same 100 observations.
X / YY presentY absent
X present
X absent

X present × Y present

Observed 20; independence expectation 12. Joint probability 0.2; row probability 0.4; column probability 0.3.

PMI
0.7369656
Positive PMI
0.7369656
Normalized PMI
0.3173938
Weighted contribution
0.1473931 bits

Lift (observed / expected): 1.666667. A large PMI for a rare pair can contribute very little to the table’s total MI. A zero observed count is not evidence that the pair can never occur.

Shared and remaining uncertainty

All bars use the same scale in bits. Shared information is the common part; conditional entropy is the uncertainty left after knowing the other category. Very small components may be too narrow to see, so their values remain below.

X entropy0.9709506 bits
  • X given Y: 0.8796731
  • Shared information: 0.09127745
Y entropy0.8812909 bits
  • Y given X: 0.7900135
  • Shared information: 0.09127745
Joint entropy1.760964 bits
  • X given Y: 0.8796731
  • Shared information: 0.09127745
  • Y given X: 0.7900135
Arithmetic-normalized MI
0.0985589 · 2 MI / (H(X) + H(Y))
Joint-entropy-normalized MI
0.05183379 · MI / H(X,Y)

Undefined means the normalization denominator is zero. These scores are not adjusted for chance and are not significance tests.

Inspect the cell ledger

Filtering changes this display only. Every cell still contributes to the same probabilities and summary, and every cell remains in the export.

4 of 4 cells displayed; information in bits.
X / YObservedExpectedPMINPMIWeighted contribution
X present / Y present20120.73696560.31739380.1473931
X absent / Y absent50420.25153880.25153880.1257694
X absent / Y present1018-0.8479969-0.2552725-0.08479969
X present / Y absent2028-0.4854268-0.209062-0.09708537

Keep the full calculation

Export all counts, marginals, cell metrics, undefined states and summary values. The display filter never removes data from the CSV. Both mode drafts are retained and identified as active or inactive.

Method and interpretation

PMI = log[p(X,Y) / (p(X)p(Y))]. MI sums joint probability × PMI over every cell. Positive PMI means more co-occurrence than the independence expectation; negative PMI means less. Positive PMI (PPMI) clips negative values to zero and is a separate cell metric, not the contribution used in MI.

NPMI divides PMI by −log p(X,Y). It is −1 by the zero-joint limit when marginals are positive, and undefined when a marginal is zero or the joint probability is one. Zero-count cells contribute zero to MI. Arithmetic-normalized MI uses the mean marginal entropy; joint-normalized MI uses joint entropy. Neither is adjusted for chance.

Whole counts are validated before calculation. Exact integer products and stable logarithms preserve near-independent differences; small nonzero information remains visible in scientific notation. The observed distribution is not a population estimate with a confidence interval. Rare pairs can rank highly in PMI without providing much overall information.

Sources reviewed November 18, 2026: Bouma’s PMI and normalization paper, count-based MI, and arithmetic normalization. Zero-entropy normalizations are shown as undefined here.

How to use this calculator

  1. Choose an editable example, then select overlapping events or a categorical table and Information units. Bits, nats and dits change logarithmic units; normalized scores do not change.
  2. In event mode enter total observations, the count containing X, the count containing Y and the count containing both. All counts must refer to the same observations and consistent event definitions.
  3. In table mode edit row and column labels and enter whole counts. Add or remove categories as needed, or preview and apply a rectangular numeric-only pasted table without headers. Empty cells are invalid, not zeros.
  4. Select PMI, normalized PMI or weighted contribution in the heatmap, then choose a cell to inspect observed and expected counts, probabilities, lift and explicit zero states.
  5. Compare total shared information with the entropy bars and remaining uncertainty. Use Minimum observed count to display and Sort cells by to inspect the ledger; these controls do not discard observations from the calculation.
  6. Download information CSV or open the copyable fallback to retain inputs, complete cell results and assumptions. Treat rare-pair scores cautiously and use a separate inference method when you need a significance test.

The Formula

The mathematical logic powering this calculation is:

Let Oij be a cell count, Ri and Cj its row and column totals, and N the grand total. The empirical joint probability is pij = Oij/N; marginals are pi = Ri/N and pj = Cj/N. The expected count under independence is Eij = Ri Cj/N. Lift = pij/(pi pj), and PMIij = log(pij/(pi pj)). Choose base 2 for bits, e for nats or 10 for dits. PPMI = max(0,PMI). Normalized PMI = PMI/[-log(pij)] when 0 < pij < 1. These logarithms use the same base.
Mutual information I(X;Y) = sum pij PMIij over positive cells. A zero-count cell contributes zero. Individual contributions may be negative; total MI is nonnegative. Entropy H(X) = -sum pi log(pi), H(Y) analogously, and joint entropy H(X,Y) = -sum pij log(pij). H(X|Y) = H(X,Y)-H(Y), H(Y|X) = H(X,Y)-H(X), and I = H(X)+H(Y)-H(X,Y). Conditional entropies are evaluated directly from positive cell counts to avoid subtraction cancellation. MI uses a numerically stable equivalent relative-entropy expression with exact integer count products and a near-independence series, preserving tiny positive values.
Arithmetic normalized MI = 2I/[H(X)+H(Y)]. Joint normalized MI = I/H(X,Y). A zero denominator is reported as undefined, rather than using a clustering-agreement convention. Neither measure adjusts for chance. For a zero joint count with positive marginals, PMI is negative infinity, PPMI is zero and normalized PMI is shown at its -1 limit. With a zero marginal, the pointwise ratio is undefined. If a single joint outcome has probability one, PMI is zero but normalized PMI is undefined (0/0).
Event mode constructs a two-by-two table: both, X-only, Y-only and neither. The counts must satisfy max(0,X+Y-N) <= both <= min(X,Y), with X and Y no greater than N. Table mode accepts 1–12 rows and 1–12 columns, unique nonblank labels on each axis and a positive total. Each count is an unsigned whole number no greater than 1000000000000. Empty marginal categories remain visible. There is no smoothing, significance test, confidence interval, continuous-variable estimation or automatic text tokenization. Display filtering changes only the ledger, not the underlying distribution or results.

Worked Example

If you input:

  • Mode / units: Overlapping events / bits
  • Total observations: 100
  • X / Y / both: 40 / 30 / 20

Result:

The X-present/Y-present cell has 20 observations versus 12 expected under independence: lift 1.666667, PMI 0.7369656 bits and normalized PMI 0.3173938. Its weighted contribution is 0.1473931 bits. Across all four cells, total mutual information is 0.09127745 bits; H(X) is 0.9709506 bits, H(Y) is 0.8812909 bits and joint entropy is 1.760964 bits. Arithmetic normalized MI is 0.09855890. One positive cell contributes more than the total because other cells have negative contributions.

Frequently Asked Questions

How is pointwise mutual information different from mutual information?

PMI describes one pair of outcomes relative to independence. Mutual information is the probability-weighted sum across the entire joint distribution. A rare pair may have high PMI but a small weighted contribution; negative cell contributions also matter.

Why can a cell have negative PMI?

Its observed joint probability is below the product of its marginal probabilities. Negative PMI is meaningful association information, not an input error. PPMI clips it to zero for display as a separate metric; the total MI calculation still uses signed contributions.

What happens when a count is zero?

An unobserved pair with positive marginals has negative-infinite PMI, zero weighted contribution and normalized PMI at its -1 limit. A zero marginal makes the pointwise ratio undefined. A certain joint outcome has zero PMI but undefined normalized PMI. No pseudocount is silently added.

Does normalized PMI mean the same thing as normalized MI?

No. Normalized PMI describes an individual cell. The arithmetic and joint normalized MI values summarize the full distribution using different entropy denominators. Zero denominators are undefined here, and neither normalization corrects for chance.

Can I use this for words or search queries?

Yes, if you first define a consistent observation unit and construct compatible counts. For event mode that could be documents containing each word and both words. Do not mix token frequencies with document co-occurrence counts. This tool does not tokenize a corpus or choose a context window.

Does a large score prove a relationship is reliable or causal?

No. These are descriptive empirical scores. Sparse samples, many categories and rare pairs can distort interpretation. There is no p-value, bias correction, confidence interval or causal conclusion. The related chi-square tool offers a different inference workflow with its own assumptions.

Does the minimum count filter recalculate entropy?

No. It only filters the displayed ledger. All supplied observations remain in the heatmap, totals, entropy calculations and full CSV. To analyze a different distribution, explicitly edit the input counts.

Similar Calculators