I have been fascinated recently by comparing the results of the Cardus Education Survey with a study put out by Brian Ray. Both look at young adults who were homeschooled as children, asking them questions that range form their college attendance to their civic involvement. I wrote my master’s thesis on homeschooling, but did not deal with either study at the time, partly because I was looking at homeschoolers in the present (rather than those who had grown up) and partly because the Cardus study had not yet come out anyway. What is so interesting about these two studies is that they offer diametrically opposed results in essentially every area. Let me start by comparing the purpose and methodology of the two studies.
Brian Ray’s study was commissioned by HSLDA in 2003 in an effort to show that homeschoolers grow up to be well-rounded, well-educated, and civic-minded adults while the Cardus Education Survey was designed in partnership with Notre Dame University in 2011 in an effort to examine the outcomes of Christian education.
Ray’s study involved over 5,000 young adults who had been homeschooled, but relied on volunteers and drew its participants from HSLDA membership and local and state homeschool groups. The Cardus study included 81 young adults who were homeschooled, limiting itself to those who had church-going mothers (because it sought to study Christian education); these participants were randomly selected. This difference is important—because Ray’s participants were not randomly selected, they cannot be presumed to be representative of homeschoolers writ large, while Cardus’ random sample is much more likely to be representative. In other words, even though the Cardus sample size is smaller than Ray’s sample size, the fact that it is randomly selected nevertheless makes it more reliable.
Ray only included young adults who were homeschooled for at least seven years. Cardus does not report how many years exactly its participants were homeschooled, but the two sources it drew from required that those listed as “homeschooled” either have been homeschooled for high school or select homeschooling as their “primary” method of education. Many homeschoolers send their children to public or private high school after homeschooling for the early years, suggesting that those who homeschool for high school are perhaps most likely to be long-term homeschoolers (though there are also high school students who are pulled from school to be homeschooled, so we can’t be completely sure here).
Ray did not correct for things like race, class, or religion. Instead, like most studies conducted by homeschool advocacy groups, Ray simply compares his sample to the general American average. It’s hard to emphasize enough what a no-no this is in statistical research. By not correcting for background factors, Ray is comparing apples to oranges. The Cardus survey, in contrast, does correct for background factors, meaning that, unlike Ray, it is actually comparing things that it makes sense to compare. As with the issue of how samples are selected, once again the Cardus survey can be considered more reliable and more accurate.
With this background, it seems clear that the Cardus survey should be considered to be more accurate. By not using a random sample or correcting for background factors, the Ray study is extremely sloppy. The only thing it shows is how some homeschoolers felt about their experience. The only thing the Ray study shows is how some adults who were homeschooled are faring in life and feel about their experiences. Of course, Ray did gather an impressively large group of young adults with homeschool backgrounds, making his study interesting in its own right, just not necessarily either representative (given the lack of random sampling) or particularly instructive (given the lack of correction for background factors).
One final thing before I move on: Given that these studies are studies of adults who were homeschooled as children and teens, they speak more to homeschooling in the past than in the present. Sure, there will be some continuities, but homeschooling in 2013 is in many ways very different from homeschooling in 1993. The Ray study included adults who were homeschooled from age 18 and up, meaning that it includes those who were homeschooled through 2003 at the latest. The Cardus survey included those age 24 and up, meaning that it includes those who were homeschooled through 2005 at the latest. These surveys thus say absolutely nothing about homeschooling in the past decade, speaking only to the decades before that.
Over the next few months, I hope to compare and contrast some of the results of both Ray’s study and the Cardus study and discuss what they do and do not tell us.