Looking for Inter-Group Differences

David J. Weiss

When middle-aged psychologists like me were in school, the “subject” was a simple organism. We randomly assigned people to experimental groups and didn’t worry about what they brought with them to the study. The notion was named by Shanteau (1999) as the “GNAHM” – the Generalized Normal Adult Human Mind.” Any normal person could be assigned to any experimental condition. We were studying general principles of behavior, seeking regularities that describe the species. Even social psychologists preferred to construct groups within a study rather than use pre-existing classifications.

Sometime during the 80’s, things changed. The very term “subject” came to be considered offensive; APA has banned it. Currently we call them “participants”, emphasizing the voluntary, collaborative spirit that is supposed to characterize the folks who grace our studies. Many psychologists began to elicit personal information from their participants, and to include summarized age and race statistics in journal articles.

In my view, the modern tendency to provide information has its upsides and downsides. Knowing the specifics of samples has the potential to help us understand why results didn’t replicate from one laboratory to another. On the other hand, the same kind of specific information may lead readers to discount a result if the sample seems somehow unusual. Generally, researchers are going to use whatever volunteers they can easily get their hands on. It may be all too easy to ignore subtle but crucial discrepancies in procedure when an easy explanation for inconsistent results in terms of differences between samples is available. In the bad old days, when intro psych students were just plain “subjects”, readers presumed that samples were homogeneous across the country, largely because we didn’t know any better.

Regardless of my old-fashioned perspective, the tide of history is moving toward examining personal characteristics of participants. In my graduate seminars, students propose experiments. The most commonly proposed hypotheses involve race or ethnicity. “Do people of my group have more orgasms than those of some other groups?” is the modal question among students in Psy 542. I try to dissuade the proposers, hinting that variation within groups is likely to be sizable, but my efforts are usually in vain.

If we gracefully accept the trend, then we are obliged to seek out effective ways to explore how group membership predicts behavior. The haphazard approach, using regression methods to tease out links from data collected with convenience samples, suffers from the disadvantage that such samples seldom generate enough power to overcome the differences among people whose labels are the same. An alternative strategy is to design studies that incorporate demographic variables as experimental factors.

The most direct approach is to have group membership be one of several variables in a standard (i.e., fully crossed) factorial design. For example, for her thesis research, C. Linda Egu enlisted Latino and White teachers in a study of judgments of child abuse. A sizable number of Latino and White teachers was recruited from graduate Education classes. Egu wanted to include Black teachers as well, but they proved too hard to find. Each teacher was randomly assigned to a design cell formed by combining the manipulated factors, which in this study were degree of abuse, race of the child, and whether the response was recognizing, as opposed to reporting, the abuse. Each teacher made only one judgment, as we were concerned both about contextual effects and the participant’s guessing that race was an element in the study.

Some other useful designs are not as well known (but – commercial message - they are discussed in my Psy 515 course). The nested group design is called for when each participant within each of several groups is scheduled to make more than one response. Christine Rundall used this design in a study of medication compliance among patients with various diseases (Rundall & Weiss, 1998). We anticipated that patients with more threatening diseases and flagrant symptoms were more likely to be compliant regardless of medication side effects, whereas patients with less traumatic diseases or with mild symptoms would allow the same irritating side effects to disrupt their pill-taking.

The compliance study illustrates one of the interesting limitations of classifying people according to specific criteria. Diseases tend to be age-linked and to some extent, gender-linked. In Rundall’s diligently recruited sample, patients with Coronary Obstructive Pulmonary Disease, a severe condition with dramatic symptoms, were all elderly. Those with Inactive Tuberculosis, an asymptomatic disease, were generally young. Consequently, age was confounded with disease severity, not because of carelessness by the researcher, but because the confounding exists in the population. Confounding means that effects we ascribe to disease characteristics may in fact be attributable to age. No statistical magic can undo confounding when it is inherent in the population.

Another design of potential utility is the fractional factorial, so named because the number of treatment combinations used in the study is smaller, usually much smaller, than the number called for by normal factorial crossing. This design is used when the researcher wishes to explore the impact of a large number of controllable factors simultaneously. For example, one might be using focus groups to examine the attractiveness of a (perhaps hypothetical) political candidate, with the idea that characteristics of voters as well as those of candidates affect voting preference. Possible contributing elements might be gender, race, and political affiliation of the voter and of the candidate, as well as the candidate’s stances on various current issues. Aspects of the candidate can be manipulated by the researcher, but characteristics of the voter cannot. This in itself is not necessarily a problem, as we can try to recruit participants having the desired characteristics. The problem is that a normal, fully crossed design with many factors generates an impractically large number of design cells. The solution usually adopted is to omit factors that the researcher hopes are not crucial. A more desirable solution is to adopt a design that maintains the ability to see whether a factor contributes (a main effect), albeit at the cost of sacrificing information about some of the interactions among the factors. That solution is the fractional factorial design (also discussed in Psy 515 – what a coincidence!)

The key for researchers who want to study inter-group differences is whether people who have the desired characteristics can be recruited. Especially as studies progress beyond simple comparisons, e.g., one ethnicity vs. another, to include more factors, finding people who have the requisite combined traits can be a challenge. Even after we identify the people we want, we must convince them to offer their services to our project. Cash is a possible incentive, but (a) we seldom have enough, and (b) we may want participants who are intrinsically motivated to share their views rather than to do as little as they have to in order to get paid.

The art of recruiting is seldom discussed in texts. In my Psy 504 course (another shameless commercial!), we spend a fair amount of time thinking about this issue. A useful, and true, selling argument is that unless people of the desired types participate in research, the picture that scientists paint from the data they collect will be incomplete. The people we need for our studies will not always be conveniently available on campus. Knowledge of the community can be very important. It is often helpful if the recruiter matches the potential recruit in the most prominent demographic characteristics.

References

Rundall, C. S., & Weiss, D. J. (1998). Patients' anticipated compliance: A functional measurement analysis. Psychology, Health, & Medicine, 3, 261-274.

Shanteau, J. (1999). Decision making by experts: The GNAHM effect. In J. Shanteau, B. Mellers, & D. Schum (Eds.), Decision research from Bayesian approaches to normative systems: Reflections on the contributions of Ward Edwards. Norwell, MA: Kluwer Academic Publishers.