When I wrote my master’s thesis on homeschooling, I was struck by just how much we do not know about homeschooling. When faced with studies purported to show that homeschooled students score in the 87th percentile, I have often pointed out why the findings don’t actually mean what they’re claimed to mean and have frequently responded by explaining that we do not actually know how well homeschooled students as a whole do academically, much less how well they do when compared to their peers. The data just isn’t there. But as I delve into this research more and more, I’m realizing that this isn’t entirely true. While imperfect, there is some pretty fascinating data out there, albeit rather buried and neglected. In this post I will examine some of the problems with conventional research on homeschooled students’ academic success and some of the important research considerations we should keep in mind when approaching this sort of data.
The most prominent studies of homeschooled students’ academic achievement have been conducted by homeschool advocacy groups and are presented to prospective participants as a way to prove that homeschooling is superior to other educational methods. Given this, if you are a homeschooling parent and your child is failing academically and you know it, you’re likely to sit that one out. This may skew the results. In addition, most states do not require standardized testing, meaning that in most states only dedicated and involved parents will have their children tested. Given that these studies use standardized test results, this definitely skews the range of possible participants. Similarly, in many cases these tests are administered at home, by the children’s parents, which may offer them an advantage even if the proper procedure is followed. But these concerns are actually small compared to the two largest considerations.
First, studies of homeschooled students’ academic performance never use a random sample of homeschoolers. And I do mean never. When you conduct a study, if you want to be able to generalize from it to a greater whole, you have to have a sample that is actually representative of that whole. If you recruit volunteers through your organization’s mailing list, say, what you get is going to be representative of the most motivated of your organizations’ followers, but not representative of the population as a whole. This is why, if you want to generalize from your study to the whole population, you have to get a random sample.
To my knowledge, only two studies of homeschooling employ a random sample. These are the National Center for Education Statistics’ National Household Education Surveys, and the Cardus Education Survey. Neither of these directly addresses homeschooled students’ academic performance.
Second, studies of homeschooled students’ academic performance almost never correct for background factors. For example, we know that public school students whose parents have college degrees perform better than those whose parents do not have college degrees. Wealthy students outperform poor students, students with two parents living in the home outperform those with a single parent, etc. The studies homeschool advocates hold up as showing that homeschooled students are academically superior to public school students overwhelmingly involve nonpoor participants who have two college educated parents (interestingly, data from the NCES suggests that homeschooled students’ backgrounds may be fairly close to the average in terms of parental education and income, which indicates that these studies are conclusively not representative of the homeschool population as a whole). These studies do not control for these background factors and simply compare their results—the results of children from wealthier, more educated, more cohesive and stable families—to the public school average rather than to their public school peers (i.e., public schooled children from wealthier, more educated, more cohesive and stable families). These studies, in other words, are not actually studying the effect of being homeschooled. Instead, they are studying the effect of being nonpoor and having two college educated parents.
I only know of two studies that have corrected for background factors. The Cardus Education Survey used a random sample and corrected for background factors, and Michael Cogan’s study of homeschool graduates’ college performance studied homeschool graduates at a particular research university and corrected for background factors. The Cardus found that homeschooling impeded students’ college attendance while Cogan found that the small number of homeschool graduates who attended the single midwestern research university in his study outperformed their peers. Neither of these studies, however, looked at the academic performance of students while being homeschooled.
There actually was a study in 2011 that tried to correct some of these problems. In it, researchers compared 37 homeschooled students and 37 demographically matched public school students eliminated some of these problems. First, background factors were controlled for, and second, but samples were recruited rather than randomly selected, which helps mitigate the problem of comparing homeschool volunteers to non-volunteer public school scores. And what did this study find? It found that some of the homeschool students outperformed their public school peers while some of them underperformed them. In other words, it was a wash. (For more, see this review of the research.)
Another interesting thing is that the press releases regarding the studies most cited by homeschool advocacy groups often depart widely from what the studies themselves actually say. Lawrence Rudner had this to say of his much-touted 1997 Rudner study: “The biggest annoyance was a large number of reporters that had read previous articles and never went to the original source and read the caveats.” And indeed, the very first paragraph of Rudner’s study stated that “This was not a controlled experiment” and that it “does not demonstrate that home schooling is superior to public or private schools” and that “the results must be interpreted with caution.” As Rudner later told a reporter: “I made the case in the paper that if you took the same kids and the same parents and put them in the public schools, these kids would probably do exceptionally well.” In other words, Rudner never claimed his results were representative of all homeschoolers, and he never claimed that his subjects performed better than the public school average as a result of homeschooling rather than as a result of their very non-average family background in terms of parental education, income, and cohesiveness and stability. And yet, newspaper headlines and homeschool advocacy groups alike claimed the study showed that homeschooling was the cause of these students students’ success and even that this study showed that homeschooling was academically superior to other methods of education.
But there is some other data out there.
The majority homeschooled students in Alaska (over 11,000 total) are enrolled in minimalist correspondence programs that provide parents with money for tutors and ask in return only a yearly educational plan, quarterly progress reports, and standardized testing at the end of each year. These programs are officially public or charter schools, and as such must release their data each year. What is most fascinating about this data is that they actually break the results down into “economically disadvantaged” and “not economically disadvantaged.” This allows some measure of correcting for background factors. Now sure, parents who choose these programs may not be representative of the homeschool populations as a whole, as Alaska homeschool law does give parents the option to homeschool with any oversight at all. However, given that the majority of Alaska’s homeschooled students enroll in these programs and that the data allows us to correct for economic factors, I think we may still find some interesting results.
Similarly, Arkansas tests homeschooled students each year, and releases the results. For several years, their releases actually compared the homeschooled children’s scores with the average public school scores. Unfortunately, we cannot correct for background factors as that data is not available (or recorded). This means that comparing the homeschool average with the public school average may be unfair, as it is possible that the homeschooled students are from, say, better educated families than average. Of course, the opposite may also be true. The simple reality is that we do not know. We also do not know how many Arkansas students are homeschooled underground, without following the state requirements. However, Arkansas does not require any minimum score on these exams, which may help cut down on the number of parents who might try to avoid the requirement. While we cannot correct for background factors, and while some homeschoolers may avoid the testing requirement, this data comes as close as we have to a random sample and deserves a look.
In some sense, these two data points allow us to do things no other data on homeschoolers’ academic achievement allow us to do. The Alaska data allows us to correct for background factors, and the Arkansas data gives us what amounts to a random sample. Of course, it should be noted that there may be some particularities in the Alaskan homeschool correspondence school students that make them different from those who choose the minimalist homeschool option, and it should be noted that the Arkansas data is only representative of homeschoolers in that state and that we cannot claim that Arkansas homeschoolers are representative of homeschoolers in the country as a whole. Regardless, I’m going to spend some time going over this data, and I will create charts to illustrate my findings.