Gwet’s AC2 Confidence Interval

Approach

The calculations of the standard error for subjects and the corresponding confidence intervals are similar to those used for Krippendorff’s alpha, based on the following formulas. The calculations for the standard error for ratings and the total standard error and confidence intervals are also as for Krippendorff’s alpha.

where γ is the AC2 measurement.

If the number of subjects in the population of subjects is assumed to be large, then f = 0. If the actual number of subjects is a relatively small finite number, then the calculation of the standard error can be corrected by using the correction factor f = n/N, where N = the number of subjects in the population.

Example

Example 1: Calculate the standard errors and 95% confidence intervals for Gwet’s AC2 of Example 1 of Gwet’s AC2 Basic Concepts.

The calculations are shown in Figure 1.

Get's AC2 confidence interval

Figure 1 – Standard errors and confidence intervals

Key formulas from Figure 1 are shown in Figure 2. References are made to cells in Figure 2 of Gwet’s AC2 Basic Concepts.

The formula for the standard error in cell AB6 is an array formula.

Cell Entity Formula
W4 γ  =IF(T4>=2,$U$15*(U4-$U$20)/($U$16*(1-$U$20)),0)
X4  e1 =$U$18*SUMPRODUCT($O$21:$R$21,1-I4:L4/T4)/($U$17*($U$17-1))
Y4  γi* =W4-2*(1-$U$21)*(X4-$U$20)/(1-$U$20)
AB6  s.e. =SQRT(SUMSQ(Y4:Y12-U21)/(U15*(U15-1)))
AB7 lower =U21-AB6*TINV(AB5,U15-1)
AB8 upper =MIN(U21+AB6*TINV(AB5,U15-1),1)

Figure 2 – Key formulas

The standard error for raters (cell AB11) is calculated via jackknifing as for Krippendorff’s alpha (see Standard Error for Krippendorff’s Alpha), as are the total standard error (cell AC6) and 95% confidence interval (AC7:AC8).

Examples Workbook

Click here to download the Excel workbook with the examples described on this webpage.

References

Gwet, K. L. (2001) Handbook of Interrater reliability. STATAXIS Publishing

Gwet, K. L. (2015) On Krippendorff’s alpha coefficient
https://agreestat.com/papers/onkrippendorffalpha_rev10052015.pdf

Gwet, K. L. (2015) Standard error of Krippendorff’s alpha coefficient
http://inter-rater-reliability.blogspot.it/2015/08/standard-error-of-krippendorffs-alpha.html

12 thoughts on “Gwet’s AC2 Confidence Interval”

  1. Hi,

    I am noticing that the AC2 calculation is slightly different in the analytical tool compared with the manual calculation by the hundredth’s decimal place. I am assuming this will also affect the confidence interval as well. There must be something that I am overlooking here. Could I send you my calculations for both?

    Amy

    Reply

  2. Hi Charles

    Thank you so much for these pages!

    I am currently investigating the inter-rater reliability of a swimming test and the different items of that test. The test contains 17 items, that are rated 1, 2 or 3 (ordinal). The 5 raters rated a total of 4 tests (4x17items). For this study each item, even within one “test”, was performed by a different subject. (Therefore, no sum score was calculated for the test, the different items are looked at separately.)

    I want to calculate Gwet’s AC2 and confidence interval
    1) for each test (each test containing 17 skills, each skill performed by a different child)
    2) for each test item across tests. For example: the Gwet’s AC2 for the ratings of item 4 on test 1, 2, 3 and 4

    I am not sure which value I have to use for f in both cases.
    And what is the difference between the scorrection and rcorrection?
    Can you help me?

    I hope this explanation was somewhat clear, please tell me if something wasn’t.

    Kind regards
    Lisa

    Reply

    • One additional question: does the f-value only have an influence on the standard error or also on the confidence interval?

      Kind regards

      Lisa

      Reply

      • Hello Lisa,
        Since the confidence interval uses the standard error and the standard error uses the f-correction, both the standard error and the confidence interval are influenced by the f-correction value.
        Charles

        Reply

Leave a Comment