MSA Nominal Level Output
The output for a Measurement Systems Analysis for Nominal level data is shown below.
Several unique diagnostic assessments are performed. Pair-wise comparisons are made for each set of measures for each appraiser. Pair-wise comparisons are also made between each appraiser.
Output Statistics
| n | Sample Size |
| Agree | The number of items in which agreement occurred |
| Disagree | The number of items in which disagreement occurred |
| P(Agree) | The proportion of items in which agreement occurred |
| P(Disagree) | The proportion of items in which disagreement occurred |
| P(Chance) | The proportion of agreement expected due to chance |
| Kmax | kappa Max |
| kappa | The kappa Statistic |
| CI(low) | The lower bound of the confidence interval for kappa |
| CI(high) | The upper bound of the confidence interval for kappa |
| p(kappa) | The p-value or statistical significance of kappa |
| p(Sym) | The p-value of the test for symmetry |
example — Three Appraisers, One Measure Each
Measurement Systems Analysis - Nominal Data
File: C:\DATA\LWI\MSA-Nominal\Christie, Lisa, and Patsy.mvp Confidence Level = 95.00%
Variables
Christie [Christie] Lisa [Lisa] Patsy [Patsy]Concordance Analysis (Between Appraisers)
Group n Agree Disagree P(Agree) P(Disagree) P(Chance) Kmax kappa CI(low) CI(high) p(kappa) p(Sym) Christie vs Lisa 175 169 6 0.966 0.034 0.510 0.953 0.930 0.875 0.985 0.000* 0.219 Christie vs Patsy 175 130 45 0.743 0.257 0.497 0.830 0.489 0.359 0.618 0.000* 0.036* Lisa vs Patsy 175 126 49 0.720 0.280 0.496 0.785 0.445 0.311 0.579 0.000* 0.009* Average 175 142 33 0.810 0.190 0.501 0.856 0.618 -- -- -- -- Overall 175 125 50 0.714 0.286 0.251 0.855 0.618 0.532 0.705 0.000* --
example — Two Appraisers, Three Measures Each
Measurement Systems Analysis - Nominal Data
File: C:\DATA\LWI\MSA-Nominal\Ray and Victoria.mvp Confidence Level = 95.00%
Variables
Ray [Ray1,Ray2,Ray3] Victoria [Victoria1,Victoria2,Victoria3]
Specifications
Acceptable Code = 1
Internal Consistency (Within Appraiser)
Ray - Internal Consistency Analysis
Group n Agree Disagree P(Agree) P(Disagree) P(Chance) Kmax kappa CI(low) CI(high) p(kappa) p(Sym) m1 vs m2 250 239 11 0.956 0.044 0.337 0.964 0.934 0.895 0.972 0.000* 0.446 m1 vs m3 250 233 17 0.932 0.068 0.338 0.976 0.897 0.851 0.944 0.000* 0.333 m2 vs m3 250 230 20 0.920 0.080 0.333 0.982 0.880 0.830 0.930 0.000* 0.255 Average 250 234 16 0.936 0.064 0.336 0.974 0.904 -- -- -- -- Overall 250 227 23 0.908 0.092 0.139 0.972 0.893 0.836 0.950 0.000* -- Category Analysis (m1 vs m2) 1 vs Not 250 244 6 0.976 0.024 0.500 0.968 0.952 0.914 0.990 0.000* 0.219 2 vs Not 250 243 7 0.972 0.028 0.673 0.963 0.914 0.852 0.977 0.000* 0.453 3 vs Not 250 244 6 0.976 0.024 0.710 0.972 0.917 0.852 0.983 0.000* 0.687 4 vs Not 250 247 3 0.988 0.012 0.792 0.942 0.942 0.878 1.000 0.000* 0.250 Overall 250 239 11 0.956 0.044 0.337 0.964 0.934 0.895 0.972 0.000* 0.446 Category Analysis (m1 vs m3) 1 vs Not 250 242 8 0.968 0.032 0.500 0.968 0.936 0.892 0.980 0.000* 0.289 2 vs Not 250 242 8 0.968 0.032 0.670 0.951 0.903 0.837 0.969 0.000* 0.289 3 vs Not 250 240 10 0.960 0.040 0.705 1.000 0.864 0.783 0.946 0.000* 1.000 4 vs Not 250 242 8 0.968 0.032 0.801 1.000 0.839 0.730 0.948 0.000* 1.000 Overall 250 233 17 0.932 0.068 0.338 0.976 0.897 0.851 0.944 0.000* 0.333 Category Analysis (m2 vs m3) 1 vs Not 250 240 10 0.960 0.040 0.500 1.000 0.920 0.871 0.969 0.000* 1.000 2 vs Not 250 237 13 0.948 0.052 0.664 0.988 0.845 0.764 0.927 0.000* 1.000 3 vs Not 250 240 10 0.960 0.040 0.710 0.972 0.862 0.779 0.945 0.000* 0.754 4 vs Not 250 243 7 0.972 0.028 0.792 0.942 0.866 0.768 0.963 0.000* 0.453 Overall 250 230 20 0.920 0.080 0.333 0.982 0.880 0.830 0.930 0.000* 0.255 Category Analysis (All Measures) 1 vs Not 250 238 12 0.952 0.048 0.250 0.979 0.936 0.864 1.000 0.000* -- 2 vs Not 250 236 14 0.944 0.056 0.503 0.968 0.887 0.762 1.000 0.000* -- 3 vs Not 250 237 13 0.948 0.052 0.562 0.982 0.881 0.741 1.000 0.000* -- 4 vs Not 250 241 9 0.964 0.036 0.692 0.961 0.883 0.697 1.000 0.000* -- Overall 250 227 23 0.908 0.092 0.139 0.972 0.893 0.836 0.950 0.000* -- Category Analysis (m1 vs m2) - Nonconforming Only 2 vs Not 122 120 2 0.984 0.016 0.519 1.000 0.966 0.919 1.000 0.000* 1.000 3 vs Not 122 117 5 0.959 0.041 0.541 0.946 0.911 0.834 0.987 0.000* 0.375 4 vs Not 122 119 3 0.975 0.025 0.633 0.933 0.933 0.858 1.000 0.000* 0.250 Overall 122 117 5 0.959 0.041 0.347 0.962 0.937 0.808 1.000 0.000* -- Category Analysis (m1 vs m3) - Nonconforming Only 2 vs Not 121 118 3 0.975 0.025 0.516 0.949 0.949 0.892 1.000 0.000* 0.250 3 vs Not 121 112 9 0.926 0.074 0.535 0.982 0.840 0.740 0.940 0.000* 1.000 4 vs Not 121 115 6 0.950 0.050 0.653 0.952 0.857 0.746 0.968 0.000* 0.687 Overall 121 112 9 0.926 0.074 0.352 0.962 0.885 0.754 1.000 0.000* -- Category Analysis (m2 vs m3) - Nonconforming Only 2 vs Not 122 118 4 0.967 0.033 0.519 0.966 0.932 0.866 0.997 0.000* 0.625 3 vs Not 122 113 9 0.926 0.074 0.541 0.982 0.839 0.738 0.940 0.000* 1.000 4 vs Not 122 115 7 0.943 0.057 0.633 0.933 0.844 0.732 0.956 0.000* 0.453 Overall 122 112 10 0.918 0.082 0.347 0.962 0.875 0.745 1.000 0.000* -- Category Analysis (All Measures) - Nonconforming Only 2 vs Not 120 116 4 0.967 0.033 0.280 0.965 0.954 0.842 1.000 0.000* -- 3 vs Not 120 109 11 0.908 0.092 0.305 0.964 0.868 0.749 0.987 0.000* -- 4 vs Not 120 113 7 0.942 0.058 0.458 0.923 0.892 0.728 1.000 0.000* -- Overall 120 109 11 0.908 0.092 0.125 0.952 0.895 0.818 0.972 0.000* --
Victoria - Internal Consistency Analysis
Group n Agree Disagree P(Agree) P(Disagree) P(Chance) Kmax kappa CI(low) CI(high) p(kappa) p(Sym) m1 vs m2 250 239 11 0.956 0.044 0.341 0.982 0.933 0.895 0.971 0.000* 0.423 m1 vs m3 250 235 15 0.940 0.060 0.339 0.970 0.909 0.865 0.954 0.000* 0.477 m2 vs m3 250 230 20 0.920 0.080 0.336 0.976 0.880 0.829 0.930 0.000* 0.490 Average 250 235 15 0.939 0.061 0.339 0.976 0.907 -- -- -- -- Overall 250 228 22 0.912 0.088 0.141 0.977 0.898 0.841 0.954 0.000* -- Category Analysis (m1 vs m2) 1 vs Not 250 245 5 0.980 0.020 0.500 0.976 0.960 0.925 0.995 0.000* 0.375 2 vs Not 250 246 4 0.984 0.016 0.675 0.975 0.951 0.903 0.999 0.000* 0.625 3 vs Not 250 243 7 0.972 0.028 0.687 0.987 0.910 0.845 0.976 0.000* 1.000 4 vs Not 250 244 6 0.976 0.024 0.820 1.000 0.867 0.762 0.971 0.000* 1.000 Overall 250 239 11 0.956 0.044 0.341 0.982 0.933 0.895 0.971 0.000* 0.423 Category Analysis (m1 vs m3) 1 vs Not 250 240 10 0.960 0.040 0.500 0.968 0.920 0.871 0.969 0.000* 0.344 2 vs Not 250 243 7 0.972 0.028 0.682 0.987 0.912 0.848 0.976 0.000* 1.000 3 vs Not 250 240 10 0.960 0.040 0.680 0.950 0.875 0.799 0.951 0.000* 0.344 4 vs Not 250 247 3 0.988 0.012 0.817 0.978 0.934 0.861 1.000 0.000* 1.000 Overall 250 235 15 0.940 0.060 0.339 0.970 0.909 0.865 0.954 0.000* 0.477 Category Analysis (m2 vs m3) 1 vs Not 250 239 11 0.956 0.044 0.500 0.992 0.912 0.861 0.963 0.000* 1.000 2 vs Not 250 241 9 0.964 0.036 0.678 0.963 0.888 0.817 0.960 0.000* 0.508 3 vs Not 250 237 13 0.948 0.052 0.678 0.963 0.839 0.754 0.924 0.000* 0.581 4 vs Not 250 243 7 0.972 0.028 0.817 0.978 0.847 0.736 0.958 0.000* 1.000 Overall 250 230 20 0.920 0.080 0.336 0.976 0.880 0.829 0.930 0.000* 0.490 Category Analysis (All Measures) 1 vs Not 250 237 13 0.948 0.052 0.250 0.979 0.931 0.859 1.000 0.000* -- 2 vs Not 250 240 10 0.960 0.040 0.518 0.975 0.917 0.789 1.000 0.000* -- 3 vs Not 250 235 15 0.940 0.060 0.522 0.967 0.874 0.745 1.000 0.000* -- 4 vs Not 250 242 8 0.968 0.032 0.727 0.985 0.883 0.681 1.000 0.000* -- Overall 250 228 22 0.912 0.088 0.141 0.977 0.898 0.841 0.954 0.000* -- Category Analysis (m1 vs m2) - Nonconforming Only 2 vs Not 122 120 2 0.984 0.016 0.516 1.000 0.966 0.920 1.000 0.000* 1.000 3 vs Not 122 116 6 0.951 0.049 0.523 0.966 0.897 0.817 0.977 0.000* 0.687 4 vs Not 122 118 4 0.967 0.033 0.684 0.948 0.896 0.797 0.996 0.000* 0.625 Overall 122 116 6 0.951 0.049 0.361 0.974 0.923 0.789 1.000 0.000* -- Category Analysis (m1 vs m3) - Nonconforming Only 2 vs Not 120 118 2 0.983 0.017 0.523 1.000 0.965 0.917 1.000 0.000* 1.000 3 vs Not 120 115 5 0.958 0.042 0.522 0.983 0.913 0.838 0.988 0.000* 1.000 4 vs Not 120 117 3 0.975 0.025 0.665 0.975 0.925 0.842 1.000 0.000* 1.000 Overall 120 115 5 0.958 0.042 0.355 0.987 0.935 0.803 1.000 0.000* -- Category Analysis (m2 vs m3) - Nonconforming Only 2 vs Not 121 119 2 0.983 0.017 0.525 1.000 0.965 0.917 1.000 0.000* 1.000 3 vs Not 121 112 9 0.926 0.074 0.520 0.983 0.845 0.748 0.942 0.000* 1.000 4 vs Not 121 114 7 0.942 0.058 0.667 0.975 0.826 0.702 0.950 0.000* 1.000 Overall 121 112 9 0.926 0.074 0.356 0.987 0.885 0.752 1.000 0.000* -- Category Analysis (All Measures) - Nonconforming Only 2 vs Not 119 116 3 0.975 0.025 0.283 1.000 0.965 0.852 1.000 0.000* -- 3 vs Not 119 110 9 0.924 0.076 0.281 0.965 0.895 0.782 1.000 0.000* -- 4 vs Not 119 113 6 0.950 0.050 0.507 0.949 0.898 0.716 1.000 0.000* -- Overall 119 110 9 0.924 0.076 0.133 0.971 0.913 0.833 0.992 0.000* --
Concordance Analysis (Between Appraisers)
Group n Agree Disagree P(Agree) P(Disagree) P(Chance) Kmax kappa CI(low) CI(high) p(kappa) p(Sym) Ray vs Victoria 250 224 26 0.896 0.104 0.343 0.963 0.842 0.786 0.897 0.000* 0.102 Category Analysis (Ray vs Victoria) 1 vs Not 250 242 8 0.968 0.032 0.500 1.000 0.936 0.892 0.980 0.000* 1.000 2 vs Not 250 240 10 0.960 0.040 0.675 1.000 0.877 0.802 0.951 0.000* 1.000 3 vs Not 250 230 20 0.920 0.080 0.705 0.919 0.729 0.617 0.841 0.000* 0.263 4 vs Not 250 236 14 0.944 0.056 0.807 0.876 0.710 0.566 0.853 0.000* 0.180 Overall 250 224 26 0.896 0.104 0.343 0.963 0.842 0.786 0.897 0.000* 0.102 Category Analysis (Ray vs Victoria) - Nonconforming Only 2 vs Not 119 112 7 0.941 0.059 0.514 0.983 0.879 0.792 0.966 0.000* 1.000 3 vs Not 119 103 16 0.866 0.134 0.538 0.927 0.709 0.577 0.841 0.000* 0.454 4 vs Not 119 106 13 0.891 0.109 0.653 0.879 0.685 0.527 0.844 0.000* 0.267 Overall 119 101 18 0.849 0.151 0.352 0.935 0.766 0.634 0.899 0.000* --
