Dissertation Statistics Helper

A Data Transformation Should Not Be Used to Make Outliers Appear to Be More Valid Than They Are

When a distribution of scores, particularly scores on a dependent variable, are non-normal and contain outliers, it is sometimes the case that a data transformation that normalizes the distribution also makes the outliers less extreme, even to the point that they are no longer recognized as outliers. Does this mean that the transformed outliers can be “saved” and that they needn’t be deleted? I don’t think so. Outliers are not just mathematically inconvenient values that exert a disproportionate effect on statistical outcomes and contribute to non-normal distributions. They are scores of questionable validity. Changing their numerical values through transformation and making them less extreme in the process doesn’t make them any more valid. Raw scores that are identified as invalid due to their outlier status as just as invalid once transformed, even though they may no longer be as extreme as transformed scores. Identify raw score outliers, delete them, and then use a normalizing transform if the distribution is still strongly non-normal. A data transformation should not be used to make bad data appear to be well behaved.

I provide research design and statistical consulting services to dissertation, thesis, and DNP capstone students whose universities allow the use of such services. For more information, check my website: https://DissertationStatsHelper.com. I do not check FaceBook messages. Please use the Contact page on my website.

website title website description

Measuring Interrater Reliability When the Measure is Categorical: Cohen's Kappa and a Handy Kappa Calculator

When a construct is measured subjectively by a scorer, it is always wise to check the reliability of the scores. If the scores assigned are numbers along a continuum, inter-scorer and intra-scorer reliability are both typically measured by computing a Pearson correlation between the two sets of scores. Well trained and motivated scorers can be expected to produce inter- or intra-scorer reliability coefficients of .80 or higher. If the reliability correlations run lower than this, the quality of the data (or the scorers) is doubtful.

But what about the situation in which the “scores” aren’t numbers, but rather, categories? For instance, suppose a scorer is asked to read the autobiographical data of 20 individuals and then categorize those individuals according to whether they are oriented predominantly toward achievement, power, or affiliation. Regardless of whether the evaluations were completed once by two different scorers (inter-rater reliability) or twice by the same scorer (intra-rater reliability), agreement between the two sets of categorical scores could not be evaluated using the Pearson correlation. However, Cohen’s kappa statistic is perfect for use in evaluating the reliability of categorical scores of this sort.

Calculating kappa by hand is a tedious task, but there on online calculators that can help. One in particular is unusually easy to use and complete. Written by Richard Lowry at Vassar, the kappa calculator can be found at http://vassarstats/kappa.html. The input to the calculator is a cross tabulation showing how many times each scorer assigned each case (e.q., autobiography in the example in the preceding paragraph) to each category (e.g., achievement, power, and affiliation in the example in the preceding paragraph).

But where is one to get these cross tabulations? SPSS provides a simple solution: Analyze > Descriptive Statistics > Crosstabs. In the dialog box, move the scores from Scorer 1 to “Rows” and the scores from Scorer 2 to “columns.” Click the Statistics button, check the kappa selection, and click Continue. Finally, click OK. The output will include the cross tabulations table which is easily typed into the kappa calculator.

The SPSS calculator will also include an “unweighted” kappa value which is fine if the categories your scorers are using are truly nominal scale with absolutely no ordinal properties. However, if the categories have some ordinal qualities (like letter grades A, B, C, D, F or the categories mild, moderate, severe) then you probably need to use the linearly weighted kappa value which is available from the Vassar statistics calculator, but not from SPSS.

Issues of scoring reliability are part of the larger specialty within statistics called psychometrics. I can help you with the psychometric evaluations associated with your dissertation or other research. Please visit my website, www.DissertationStatsHelper.com.

Have you ever needed a measure of variability for a nominal scale variable? I did recently, as a way of measuring the diversity of corporate boards of directors. (I’ll give examples later.) Statistics texts focus on measuring variability of continuous variables (e.g., range, inter-quartile range, variance, standard deviation), but are remarkably silent on the matter of measuring variability in categorical variables. Kader, G. D. and Perry, M. (2007) Variability for categorical variables. Journal of Statistics Education, 15(2), 1-16, www.amstat.org/publications/jse/v15n2/kader.html provide one measure, however, with the unlikely name “coefficient of unalikability,” abbreviated with the symbol, u2. The logic behind the unalikeability statistic takes them 16 pages to explain, but we can cut to the computational formula and see how it works with some examples.

Here are some data to illustrate the unalikeability statistic. The variable is Religious Preference, with four categories: Christian, Jewish, Muslim, Hindu. Shown next are frequency distributions for two samples, one with no variability on the variable, and the second with maximum possible variability.
Sample 1: shows very little variability or diversity in religious preference
freq proportion
Christian 8 1.0
Jewish 0 0.0
Muslim 0 0.0
Hindu 0 0.0
N = 8

Sample 2: shows maximum possible variability or diversity as cases are evenly distributed across categories of the variable

freq proportion
Christian 2 0.25
Jewish 2 0.25
Muslim 2 0.25
Hindu 2 0.25
N = 8

Next, here’s the formula for the unalikeability statistic:

u2 = 1 - Sum of the squared proportions

In words: (1) square the proportions associated with each category of the variable; (2) add these squared proportions; (3) subtract that sum from 1.

For the first distribution: u2 = 1 – (1^2 + 0^2 + 0^2 + 0^2) = 0

For the second distribution: u2 = 1 – (.25^2 + .25^2 + .25^2 + .25^2) = .75

You can see that where there is no variability or diversity, u2 takes on a value of 0. With a more variable or diverse distribution, the value of u2 increases appropriately to reflect that greater variability.

The only thing that’s annoying about the unalikeability statistic is that the maximum value of u2 is different depending on the number of categories. For a two-category variable (like s*x, for instance), the highest possible value is u2 = 0.50. For a three-category variable, maximum u2 = 0.67. For a four-category variable, maximum u2 = .75. What this means is that one can’t use u2 to compare levels of variability in categorical variables that contain different numbers of categories. However, u2 does at least allow us to measure differences from one group to the next on the same categorical variable.

I suppose that this limitation of the unalikeability statistic really isn’t that limiting. After all, we can’t compare the variances or standard deviations of two different variables either because those measures of variability are influenced not only by actual data variability but also by score magnitude.

If you need statistical or research design assistance with a dissertation, thesis, or other project, I invite you to visit my website, www.DissertationStatsHelper.com to learn more about the services I offer.

www.amstat.org