While percent agreement is a straightforward and intuitive measure, it has several limitations:
1. Chance Agreement: Percent agreement does not account for the agreement that might occur by chance. This can overestimate the true level of agreement. 2. Imbalance in Marginal Totals: If one category is much more prevalent than another, high percent agreement can be misleading. 3. Lack of Sensitivity: It does not provide information about the nature of disagreements, such as whether they are systematic or random.
