Cohen’s Kappa Calculator
Cohen's Kappa Calculator
Agreement between two observers, corrected for the agreement chance alone would have produced. Enter a 2×2 table and read κ beside the raw agreement it came from — because with skewed marginals those two numbers tell opposite stories.
Cohen's Kappa
2×2 table → κTwo histopathologists independently classify 100 biopsies as positive or negative. Both positive in 40, both negative in 49, observer 1 positive alone in 5, observer 2 positive alone in 6
Formula
Pe = [(a+b)(a+c) + (c+d)(b+d)] ⁄ n²
κ = (Po − Pe) ⁄ (1 − Pe)
- Po
- observed agreement — the proportion of cases the two observers classified the same way. The number everyone can interpret, and the number that must be reported beside kappa
- Pe
- expected agreement by chance, computed from each observer's own marginal totals on the assumption that they classify independently at their own observed rates. It is not 0.5: it depends on how often each observer says positive, and with skewed marginals it can exceed 0.9
- 1 − Pe
- the room left for agreement above chance. This denominator is what makes kappa behave strangely. When chance agreement is 0.90, every point of observed agreement above chance is divided by 0.10, so kappa swings violently for small changes in the table
- κ
- the proportion of the available above-chance agreement that was actually achieved. 1 is perfect agreement, 0 is exactly chance, and negative means worse than chance
- the Landis and Koch bands
- <0 poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect. Landis and Koch introduced them with the warning that "these divisions are clearly arbitrary" and offered them only as "useful benchmarks for the discussion". They are a convention from one 1977 paper, not a standard, and no threshold in them corresponds to any clinical consequence
- band edges here
- Landis and Koch print their bands to two decimal places with a gap — 0.20 then 0.21 — which cannot be implemented. This page's bands are half-open, so a κ of exactly 0.600 reads as substantial where their printed table would call it moderate. Nothing turns on it; it is stated so that an edge case is not a mystery
- no confidence interval here
- the simple formula in wide circulation, √[Po(1 − Po) ⁄ n(1 − Pe)²], treats Pe as a fixed constant and therefore understates the variance. The correct asymptotic variance, from Fleiss, Cohen and Everitt, needs the whole table of marginal proportions. Rather than print an interval derived from the wrong formula, this page prints none — and for an interval on the observed agreement, which is a plain proportion, the Wilson interval is the right tool
- what kappa is not
- a measure of correctness. It compares two observers with each other, not with the truth. Two observers who make the same mistake consistently will have an excellent kappa, and a high kappa is therefore evidence of reproducibility and no evidence at all of validity
Worked example
Two histopathologists independently classify 100 biopsies as positive or negative. Both positive in 40, both negative in 49, observer 1 positive alone in 5, observer 2 positive alone in 6
n = 40 + 5 + 6 + 49 = 100
Observed agreement Po = (40 + 49) ⁄ 100 = 0.89 — the two agreed on 89 of the 100 biopsies
Observer 1 called 45 positive, observer 2 called 46 positive; both called 55 and 54 negative
Expected agreement Pe = (45 × 46 + 55 × 54) ⁄ 100² = (2070 + 2970) ⁄ 10000 = 0.504
κ = (0.89 − 0.504) ⁄ (1 − 0.504) = 0.386 ⁄ 0.496 = 0.778 — 'substantial' on the Landis and Koch scale
Read that as: of the 49.6 percentage points of agreement available above chance, the observers captured 38.6 of them, or 78%
Now the paradox. Take 100 biopsies where both observers call 90 positive, both call 1 negative, and they disagree on the other 9. Observed agreement rises to 0.91, higher than before — but each observer now calls about 94% of cases positive, so chance agreement is 0.896 and κ = (0.91 − 0.896) ⁄ 0.104 = 0.135, 'slight'
And the reverse. A table of 45 / 15 / 15 / 25 has observed agreement of only 0.70, well below both of the above, with κ = (0.70 − 0.52) ⁄ 0.48 = 0.375 — nearly three times the kappa of the 91%-agreement table. Kappa is not a relabelling of observed agreement, and neither number can be quoted alone
The kappa paradox, in three tables
| a / b / c / d | Observed Po | Chance Pe | κ | Landis and Koch | How skewed? |
|---|---|---|---|---|---|
| 90 / 4 / 5 / 1 | 0.910 | 0.896 | 0.135 | Slight | 94% and 95% positive — extreme |
| 97 / 1 / 1 / 1 | 0.980 | 0.961 | 0.490 | Moderate | 98% positive — extreme |
| 85 / 5 / 5 / 5 | 0.900 | 0.820 | 0.444 | Moderate | 90% positive — marked |
| 40 / 5 / 6 / 49 | 0.890 | 0.504 | 0.778 | Substantial | 45% and 46% — balanced |
| 20 / 10 / 10 / 60 | 0.800 | 0.580 | 0.524 | Moderate | 30% positive — moderate |
| 45 / 15 / 15 / 25 | 0.700 | 0.520 | 0.375 | Fair | 60% positive — near balanced |
What kappa answers and what it does not
| Question | Is kappa the right tool? |
|---|---|
| Do these two observers classify cases the same way? | Yes — this is what it was built for |
| Is either observer correct? | No. Kappa never looks at the truth, only at the two observers. Use sensitivity and specificity against a reference standard |
| How often do they actually agree? | No — that is Po, which kappa is derived from and then hides |
| Is a new qualitative method acceptable against the current one? | Partly. Kappa summarises the agreement; the acceptance criterion is usually a lower confidence bound on the agreement proportion, which is a Wilson interval on Po |
| Do two quantitative methods agree? | No. Dichotomising a measurement to compute a kappa throws away most of the information; use limits of agreement or Deming regression |
| Do the observers differ in threshold? | No, and kappa actively hides it. Compare b with c: a large imbalance means one observer calls positive more readily, which is a finding of its own |
Chance-corrected agreement, and the paradox that comes with it
Two observers classify the same hundred cases as positive or negative and agree on eighty-nine of them. Eighty-nine per cent sounds like a good result, and on its own it is not interpretable, because some of that agreement would have happened if both observers had been guessing at their own observed rates. Cohen's kappa removes that component. Compute the agreement chance alone would produce from each observer's marginal totals, subtract it from the observed agreement, and divide by the agreement that was available above chance. In the example, chance agreement is 0.504, so kappa is (0.89 − 0.504) divided by 0.496, or 0.778: the observers captured 78% of the agreement that was there to be had.
Landis and Koch's 1977 paper supplied the bands almost everyone uses — below zero poor, up to 0.20 slight, 0.40 fair, 0.60 moderate, 0.80 substantial, above that almost perfect — and it is worth knowing what they said about them. They wrote that "these divisions are clearly arbitrary", and offered them only as "useful benchmarks for the discussion". That is the honest framing: these are a convention from one paper, not a standard, and no boundary in them corresponds to a clinical consequence. A kappa of 0.58 is not meaningfully worse than one of 0.62, and calling the first moderate and the second substantial invents a distinction the arithmetic does not contain.
The serious problem with kappa is the paradox Feinstein and Cicchetti set out in 1990, and it is not a curiosity — it is the commonest reason a laboratory gets a surprising answer. When the marginal totals are skewed, so that nearly every case falls into one category, chance agreement becomes very high and the denominator 1 − Pe becomes very small. Take a hundred cases where both observers call ninety positive and one negative, disagreeing on the other nine. Observed agreement is 91%, better than the example above. But each observer is calling about 94% of cases positive, chance agreement is 0.896, and kappa comes out at 0.135 — 'slight'. Meanwhile a table with only 70% observed agreement and balanced marginals gives a kappa of 0.375, nearly three times as high. Both statistics are computed correctly. They are answering different questions, and the only defence is to report observed agreement and the marginal totals every single time kappa is quoted.
This page does not print a confidence interval for kappa, and the reason is worth stating rather than leaving as an omission. The formula that circulates on calculator sites — the square root of Po(1 − Po) over n(1 − Pe)² — treats expected agreement as a fixed constant when it is estimated from the same table as everything else, and therefore reports an interval narrower than the data support. The correct asymptotic variance, given by Fleiss, Cohen and Everitt, is a multi-term expression over the full set of marginal proportions; statistical packages compute it, and a bootstrap over the cases is the practical alternative. Printing a number from the wrong formula would be worse than printing none. For the quantity a validation report is usually judged on — a lower confidence bound on the agreement proportion itself — the Wilson interval on Po is the right tool and is straightforward.
Two last cautions. Kappa compares observers with each other and never with the truth, so two people trained together who share a misconception will produce an excellent kappa while both being wrong; reproducibility is not validity. And a large imbalance between the two disagreement cells is a finding in its own right that kappa conceals: if one observer calls positive far more readily than the other, they differ in threshold rather than in care, and the remedy is a clearer definition of the category rather than more training.
Frequently asked questions
How do you calculate Cohen's kappa from a 2×2 table?
Observed agreement is (a + d)/n. Expected agreement is [(a+b)(a+c) + (c+d)(b+d)]/n², computed from the two observers' marginal totals. Kappa is the observed agreement minus the expected, divided by one minus the expected. For 40/5/6/49 that is Po = 0.89, Pe = 0.504 and κ = 0.778.
What is a good Cohen's kappa?
The bands almost everyone quotes are Landis and Koch's: slight up to 0.20, fair to 0.40, moderate to 0.60, substantial to 0.80, almost perfect above. Landis and Koch themselves called these divisions "clearly arbitrary" and offered them only as benchmarks for discussion, so no boundary in them means anything clinically. What counts as good depends on the decision the classification supports.
Why is my kappa low when the observers agreed on almost every case?
This is the kappa paradox. When the marginal totals are skewed — nearly every case in one category — chance agreement becomes very high and kappa divides the small excess above chance by the small amount of room left. A table with 91% observed agreement and 94% of cases called positive gives a kappa of 0.135. The value is arithmetically correct and misleading on its own, which is why observed agreement and the marginals must always be reported alongside kappa.
What is the confidence interval for kappa?
Not the formula you will find on most calculator sites. That one, √[Po(1 − Po)/n(1 − Pe)²], treats expected agreement as a fixed constant and gives an interval that is too narrow. The correct asymptotic variance is Fleiss, Cohen and Everitt's and needs the full table of marginal proportions; statistical packages compute it, and bootstrapping over the cases is a practical alternative. This page prints no interval rather than print one from the wrong formula.
Does a high kappa mean the observers are right?
No. Kappa compares two observers with each other and never with a reference standard. Two observers who were trained together and share the same misconception will agree almost perfectly and both be wrong. Kappa measures reproducibility; for correctness you need sensitivity and specificity against a reference standard.
Should I use kappa to compare two quantitative methods?
No. Splitting a measurement at a cut-off to build a 2×2 table discards nearly all the information the measurement carried, and the kappa you get then depends on where the cut-off happened to fall relative to the sample distribution. Compare quantitative methods with limits of agreement and with a regression that allows for error in both, such as Deming.
Related calculators
References
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46.
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174.
- Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543–549.
- Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43(6):551–558.
- Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72(5):323–327.
Medical Disclaimer: The tools and content provided here are for educational and reference purposes only. They are not intended to substitute for professional medical advice, diagnosis, or treatment. Clinical decisions should always be based on the comprehensive assessment of a qualified healthcare professional.