The objective of the package of agreements is to calculate estimates of conformity and reliability between evaluators using generalised formulas that take into account different models (e.B.g. cross-over or unsused), missing data and orderly or disordered categories. The package contains generalised functions for all major conformity indices (α, γ, Ir2, κ, π and S) as well as all major intra-class correlation coefficients (one- and two-sided models, types of coherence and consistency, dimensions and average units). Estimates include bootstrap-resampling distributions, confidence intervals, and custom cleaning and plot features. The “Cohen” bit comes from its inventor, Jacob Cohen. Kappa (κ) is the Greek letter he used to designate his measure (others used Roman letters, z.B. the “t” in “t-test”, but the measurement of conformity conventionally use Greek letters). The R command is kappa2 and not kappa, because the kappa command also exists and does something completely different that, by chance, uses the same letter to represent it. It probably would have been better to call the order something like cohen.kappa, but they didn`t. For this reason, many texts recommend an 80% agreement as an acceptable minimum agreement between evaluators. Each Kappa of less than 0.60 indicates a lack of compliance between evaluators and little confidence in the results of the study.
Traditionally, the reliability of the InterRater has been measured as a simple total percentage convergence, calculated as the number of cases where the two assessors are identical, divided by the total number of cases considered. We can now use the command to establish a match percentage. The agree command is part of the error (abbreviated for Inter-Rater Reliability), which is why we must first load this package: Cohen`s Kappa (Jacob Cohen 1960, J Cohen (1968) is used to measure the compliance of two evaluators (i.e. “judges”, “observers”) or evaluating methods on categorical scales. This process of measuring the extent to which two reviewers assign the same categories or scores to the same topic is called inter-counselor reliability. Our approval percentage is around 79%, but as soon as you take random consent into account, our Cohen`s Kappa is much lower: 0.52. You see why a discussion and a better codebook here was the right approach. On the other hand, we showed a much better concordance and cohens Kappa for a variable that evaluates the instructions received from the control group: 0 = nothing, 1 = A message article not about any type of crime, 2 = An information article about crime in general, but not about the specific case, 3 = An information article about the specific case that contained only neutral information. The most important result here is %-agree, which is your percentage of match.
The edition also tells you how many topics you`ve reviewed and how many people have given reviews. The bit, which says tolerance = 0, refers to an aspect of the percentage concordance that is not covered in this course. . . .