Ray and Cardus: Yesterday’s Homeschoolers All Grown Up

I have been fascinated recently by comparing the results of the Cardus Education Survey with a study put out by Brian Ray. Both look at young adults who were homeschooled as children, asking them questions that range form their college attendance to their civic involvement. I wrote my master’s thesis on homeschooling, but did not deal with either study at the time, partly because I was looking at homeschoolers in the present (rather than those who had grown up) and partly because the Cardus study had not yet come out anyway. What is so interesting about these two studies is that they offer diametrically opposed results in essentially every area. Let me start by comparing the purpose and methodology of the two studies.

Brian Ray’s study was commissioned by HSLDA in 2003 in an effort to show that homeschoolers grow up to be well-rounded, well-educated, and civic-minded adults while the Cardus Education Survey was designed in partnership with Notre Dame University in 2011 in an effort to examine the outcomes of Christian education.

Ray’s study involved over 5,000 young adults who had been homeschooled, but relied on volunteers and drew its participants from HSLDA membership and local and state homeschool groups. The Cardus study included 81 young adults who were homeschooled, limiting itself to those who had church-going mothers (because it sought to study Christian education); these participants were randomly selected. This difference is important—because Ray’s participants were not randomly selected, they cannot be presumed to be representative of homeschoolers writ large, while Cardus’ random sample is much more likely to be representative. In other words, even though the Cardus sample size is smaller than Ray’s sample size, the fact that it is randomly selected nevertheless makes it more reliable.

Ray only included young adults who were homeschooled for at least seven years. Cardus does not report how many years exactly its participants were homeschooled, but the two sources it drew from required that those listed as “homeschooled” either have been homeschooled for high school or select homeschooling as their “primary” method of education. Many homeschoolers send their children to public or private high school after homeschooling for the early years, suggesting that those who homeschool for high school are perhaps most likely to be long-term homeschoolers (though there are also high school students who are pulled from school to be homeschooled, so we can’t be completely sure here).

Ray did not correct for things like race, class, or religion. Instead, like most studies conducted by homeschool advocacy groups, Ray simply compares his sample to the general American average. It’s hard to emphasize enough what a no-no this is in statistical research. By not correcting for background factors, Ray is comparing apples to oranges. The Cardus survey, in contrast, does correct for background factors, meaning that, unlike Ray, it is actually comparing things that it makes sense to compare. As with the issue of how samples are selected, once again the Cardus survey can be considered more reliable and more accurate.

With this background, it seems clear that the Cardus survey should be considered to be more accurate. By not using a random sample or correcting for background factors, the Ray study is extremely sloppy. The only thing it shows is how some homeschoolers felt about their experience. The only thing the Ray study shows is how some adults who were homeschooled are faring in life and feel about their experiences. Of course, Ray did gather an impressively large group of young adults with homeschool backgrounds, making his study interesting in its own right, just not necessarily either representative (given the lack of random sampling) or particularly instructive (given the lack of correction for background factors).

One final thing before I move on: Given that these studies are studies of adults who were homeschooled as children and teens, they speak more to homeschooling in the past than in the present. Sure, there will be some continuities, but homeschooling in 2013 is in many ways very different from homeschooling in 1993. The Ray study included adults who were homeschooled from age 18 and up, meaning that it includes those who were homeschooled through 2003 at the latest. The Cardus survey included those age 24 and up, meaning that it includes those who were homeschooled through 2005 at the latest. These surveys thus say absolutely nothing about homeschooling in the past decade, speaking only to the decades before that.

Over the next few months, I hope to compare and contrast some of the results of both Ray’s study and the Cardus study and discuss what they do and do not tell us.

Advertisements

3 thoughts on “Ray and Cardus: Yesterday’s Homeschoolers All Grown Up

  1. On the one hand, the HSLDA study will have a better representation of HSLDA. On the other hand it will be a bad representation of homeschooling in general. If it were 5,000 data points out of 500,000 is one data point for 100 people.

    The Cardus survey will be absolutely pitiful as a representation due to its woefully inadequate survey size, BUT it will potentially give some more *interesting* data due. Seriously, one data point per 1000 or more people is nothing to base a trend on. Using the same estimation as above, 81 data points out of 500,000 is one data point per 6172. That is a LOT of potential for variance we know nothing about.

    Therefore neither are accurate as a picture of the whole of homeschooling, but they will both provide some interesting data.

    • Actually, that’s not entirely how it works. The Ray study won’t necessarily be representative of HSLDA as it is volunteer based and had an agenda—meaning that those doing badly wouldn’t have heard or necessarily filled out the survey. Second, the Cardus study sample size is actually not that bad. The key isn’t size, the key is random selection. And again, when it comes to this sort of thing, their sample size is actually pretty amazing. I’ve seen studies done with only 37 participants and all of them selected by volunteer, and then passed off as somehow representative. I actually think that we can probably put some stock in the Cardus study, though it’s just a first step.

      • Good point about HSLDA. My only critique remaining on the Cardus study is that even in a random selection there is still a mathematical probability that no matter how random, it might not still match up. Even a random number generator will eventually display 81 zeroes in a row when generating enough numbers (in a population example such as 500,000). Larger sample sizes significantly decrease the probability of outside data being different than sampled data. You’re also right that there are some bad studies with lower participants that were selected instead of randomly sampled, but we also don’t want to make that negative to be a justification for our own small sample size.

        The whole reason I bring this up is out of a desire for greater integrity so that in the future when people bring out other studies this isn’t called into question by someone who just happened to have a different enough sample that provided a different picture of the data. Because right now at 81 people, that is just less than two homeschoolers representing each state.

        However, it will definitely still provide some interesting data, and it’s a *great* first step. 🙂

Comments are closed.