Homeschoolers and Academics: The Alaska Data

Homeschool advocates often champion studies they claim show that homeschooled students score in the 87th percentile, far above average, as proof of the success of homeschooling. Unfortunately, these studies generally do not use random samples or correct for background factors. This means that the results cannot be generalized from to all homeschoolers, and that we cannot know whether the data shows the consequence of homeschooling or of some other factor, such as socio-economic status.

In this article, we will be turning to some homeschooling data from Alaska. This data is closer to a random sample than just about any other data out there, and it also allows us to correct for students’ economic background. This data allows us to pose some interesting questions and leads to some interesting findings—findings that may be surprising, and that may challenge some of the current narratives about homeschooling. Continue reading

Advertisements

Homeschoolers and Academics: Sifting through the Data

When I wrote my master’s thesis on homeschooling, I was struck by just how much we do not know about homeschooling. When faced with studies purported to show that homeschooled students score in the 87th percentile, I have often pointed out why the findings don’t actually mean what they’re claimed to mean and have frequently responded by explaining that we do not actually know how well homeschooled students as a whole do academically, much less how well they do when compared to their peers. The data just isn’t there. But as I delve into this research more and more, I’m realizing that this isn’t entirely true. While imperfect, there is some pretty fascinating data out there, albeit rather buried and neglected. In this post I will examine some of the problems with conventional research on homeschooled students’ academic success and some of the important research considerations we should keep in mind when approaching this sort of data.

The most prominent studies of homeschooled students’ academic achievement have been conducted by homeschool advocacy groups and are presented to prospective participants as a way to prove that homeschooling is superior to other educational methods. Given this, if you are a homeschooling parent and your child is failing academically and you know it, you’re likely to sit that one out. This may skew the results. In addition, most states do not require standardized testing, meaning that in most states only dedicated and involved parents will have their children tested. Given that these studies use standardized test results, this definitely skews the range of possible participants. Similarly, in many cases these tests are administered at home, by the children’s parents, which may offer them an advantage even if the proper procedure is followed. But these concerns are actually small compared to the two largest considerations.

First, studies of homeschooled students’ academic performance never use a random sample of homeschoolers. And I do mean never. When you conduct a study, if you want to be able to generalize from it to a greater whole, you have to have a sample that is actually representative of that whole. If you recruit volunteers through your organization’s mailing list, say, what you get is going to be representative of the most motivated of your organizations’ followers, but not representative of the population as a whole. This is why, if you want to generalize from your study to the whole population, you have to get a random sample.

To my knowledge, only two studies of homeschooling employ a random sample. These are the National Center for Education Statistics’ National Household Education Surveys, and the Cardus Education Survey. Neither of these directly addresses homeschooled students’ academic performance.

Second, studies of homeschooled students’ academic performance almost never correct for background factors. For example, we know that public school students whose parents have college degrees perform better than those whose parents do not have college degrees. Wealthy students outperform poor students, students with two parents living in the home outperform those with a single parent, etc. The studies homeschool advocates hold up as showing that homeschooled students are academically superior to public school students overwhelmingly involve nonpoor participants who have two college educated parents (interestingly, data from the NCES suggests that homeschooled students’ backgrounds may be fairly close to the average in terms of parental education and income, which indicates that these studies are conclusively not representative of the homeschool population as a whole). These studies do not control for these background factors and simply compare their results—the results of children from wealthier, more educated, more cohesive and stable families—to the public school average rather than to their public school peers (i.e., public schooled children from wealthier, more educated, more cohesive and stable families). These studies, in other words, are not actually studying the effect of being homeschooled. Instead, they are studying the effect of being nonpoor and having two college educated parents.

I only know of two studies that have corrected for background factors. The Cardus Education Survey used a random sample and corrected for background factors, and Michael Cogan’s study of homeschool graduates’ college performance studied homeschool graduates at a particular research university and corrected for background factors. The Cardus found that homeschooling impeded students’ college attendance while Cogan found that the small number of homeschool graduates who attended the single midwestern research university in his study outperformed their peers. Neither of these studies, however, looked at the academic performance of students while being homeschooled.

There actually was a study in 2011 that tried to correct some of these problems. In it, researchers compared 37 homeschooled students and 37 demographically matched public school students eliminated some of these problems. First, background factors were controlled for, and second, but samples were recruited rather than randomly selected, which helps mitigate the problem of comparing homeschool volunteers to non-volunteer public school scores. And what did this study find? It found that some of the homeschool students outperformed their public school peers while some of them underperformed them. In other words, it was a wash. (For more, see this review of the research.)

Another interesting thing is that the press releases regarding the studies most cited by homeschool advocacy groups often depart widely from what the studies themselves actually say. Lawrence Rudner had this to say of his much-touted 1997 Rudner study: “The biggest annoyance was a large number of reporters that had read previous articles and never went to the original source and read the caveats.” And indeed, the very first paragraph of Rudner’s study stated that “This was not a controlled experiment” and that it “does not demonstrate that home schooling is superior to public or private schools” and that “the results must be interpreted with caution.” As Rudner later told a reporter: “I made the case in the paper that if you took the same kids and the same parents and put them in the public schools, these kids would probably do exceptionally well.” In other words, Rudner never claimed his results were representative of all homeschoolers, and he never claimed that his subjects performed better than the public school average as a result of homeschooling rather than as a result of their very non-average family background in terms of parental education, income, and cohesiveness and stability. And yet, newspaper headlines and homeschool advocacy groups alike claimed the study showed that homeschooling was the cause of these students students’ success and even that this study showed that homeschooling was academically superior to other methods of education.

But there is some other data out there.

The majority homeschooled students in Alaska (over 11,000 total) are enrolled in minimalist correspondence programs that provide parents with money for tutors and ask in return only a yearly educational plan, quarterly progress reports, and standardized testing at the end of each year. These programs are officially public or charter schools, and as such must release their data each year. What is most fascinating about this data is that they actually break the results down into “economically disadvantaged” and “not economically disadvantaged.” This allows some measure of correcting for background factors. Now sure, parents who choose these programs may not be representative of the homeschool populations as a whole, as Alaska homeschool law does give parents the option to homeschool with any oversight at all. However, given that the majority of Alaska’s homeschooled students enroll in these programs and that the data allows us to correct for economic factors, I think we may still find some interesting results.

Similarly, Arkansas tests homeschooled students each year, and releases the results. For several years, their releases actually compared the homeschooled children’s scores with the average public school scores. Unfortunately, we cannot correct for background factors as that data is not available (or recorded). This means that comparing the homeschool average with the public school average may be unfair, as it is possible that the homeschooled students are from, say, better educated families than average. Of course, the opposite may also be true. The simple reality is that we do not know. We also do not know how many Arkansas students are homeschooled underground, without following the state requirements. However, Arkansas does not require any minimum score on these exams, which may help cut down on the number of parents who might try to avoid the requirement. While we cannot correct for background factors, and while some homeschoolers may avoid the testing requirement, this data comes as close as we have to a random sample and deserves a look.

In some sense, these two data points allow us to do things no other data on homeschoolers’ academic achievement allow us to do. The Alaska data allows us to correct for background factors, and the Arkansas data gives us what amounts to a random sample. Of course, it should be noted that there may be some particularities in the Alaskan homeschool correspondence school students that make them different from those who choose the minimalist homeschool option, and it should be noted that the Arkansas data is only representative of homeschoolers in that state and that we cannot claim that Arkansas homeschoolers are representative of homeschoolers in the country as a whole. Regardless, I’m going to spend some time going over this data, and I will create charts to illustrate my findings.

About That Homeschool Infographic You Keep Seeing

If you’re at all connected to the world of homeschooling, you may have seen this snazzy new infographic on homeschooling floating around recently.

HSI Intro

Unfortunately, this infographic is highly misleading in its content and assertions. Using data in this way, though, is sadly common in the homeschooling world, and that’s something that is worth addressing. In the name of good information and accurate data, I am going to run down this infographic and offer some pointers on where the information is incorrect or misleading. My commentary here may also be of interest in pointing to what we do and do not know about homeschooling, and to the limitations of much of the data we currently have.

HSI History

The first date on the list states that in 1840 some 55% of children went to school while the other 45% were educated in the home or by tutors. This is misleading. For one thing, school then was not like school today—most children who went to school did so for only a few years, and school terms were often only a few months long. For another thing, very few parents in 1840 could hope to afford tutors. In reality, most children in 1840 started working at an early age, some in brutal conditions in factories and mines, and most children never learned much in the way of academics beyond reading, writing, and basic arithmetic. The homeschool lobby often paints a picture of modern homeschooling as natural and public education as some sort of historical aberration. This is both incorrect and irrelevant.

Next, it’s factually untrue that in 1980 homeschooling was only legal in 20 states. Homeschooling was on the books either implicitly or explicitly in 36 states in 1980, and had been for some time. In the remaining fourteen states, parents wishing to homeschool often found that they could do so by operating their homeschools as private schools or by getting permission from the local superintendent. Yes, some states had requirements that went too far, and yes, some local superintendents were difficult to work with. Some states required parents who wished to homeschool to have teaching licenses, and some superintendents ignored the law and unfairly targeted homeschoolers. However, the statement that homeschooling was only “legal” in 20 states is false. Milton Gaither goes over all of this in his chapter “Making it Legal” in his book Homeschool: An American History, which is conspicuously absent from the list of sources at the end of the infographic.

The last thing I’ll mention about the timeline is the listing for 1983. What were the “changes in tax laws” that which supposedly forced many Christian schools to close and caused homeschooling rates to soar? The answer is Bob Jones University v. United States, in which the Supreme Court held that if a religious university practiced racial discrimination, the federal government could revoke its tax exempt status. This was about whether Christian schools could be racist and retain their tax exempt status. Regardless, it is untrue that this decision “forced many Christian schools to close.” Many Christian schools have closed since the 1980s, which was the heyday of the Christian school movement, but the reasons have to so with the challenges involved in running a school and the fact that the rise of homeschooling decreased enrollment in Christian schools, not changes in tax law. It is odd that this innacurate entry would be included on this infographic, especially when something hugely important to the history of homeschooling did happen in 1983—that was the year James Dobson had Raymond Moore on his radio show, which provided many Christian parents with the endorsement they needed to make the jump to homeschooling and directly contributed to the growth of Christian homeschooling.

HSI Today

This section is fairly accurate, with a few small caveats. First, Iowa’s law changed several months before this infographic came out, and it no longer requires either assessments or notification but is still listed here as such. Second, the criteria used to highlight six states and list them as states with additional requirements such as curriculum approval, parental notification, or home visits” is a bit unclear. For one thing, it ignores the fact that a full 11 states have some form of parent qualification requirements. For another thing, in spite of its suggestion no states uniformly require home visits, and even those that require home visits in case of a problem stipulate that those home visits must not be unannounced. The left side also has a some inaccuracies. Indiana has a tax credit for homeschoolers, but this isn’t listed.

There’s one more thing to add regarding this section. It is true that 11 states require no notification or assessments, 14 states require only notification, and 25 states require notification and assessment. However, it must be borne in mind that most of the states that require assessments have additional options—such as operating as a private school—that allow many homeschoolers to bypass these requirements. Further, many of these states do not do much in the way of ensuring that their assessment requirements are actually followed. Finally, several of the states that require standardized tests do not require students to hit any particular threshold. Unfortunately, it is really hard to put this sort of nuance into a quick chart for easy reference.

HSI growth

This data is accurate but is slightly outdated given the release of the NCES data for 2011 (for an up to date listing, see here). Still, it’s a fairly good picture of the current number of homeschooled students, and of the growth of homeschooling in the past decade and a half.

HSI Parents

This section relies on data from a study Brian Ray conducted for HSLDA, a homeschool advocacy group, in 2009, and here’s where we come to a problem. Ray’s study does not actually tell us anything about the average homeschool parent or the average homeschooled child. Ray recruited participants through major testing services, meaning that his study only included homeschooling parents who choose to have their students tested, which self-selects for a certain type of parents. Further, Ray had an absolutely terrible response rate; no social scientist would ever dare to suggest that a study with such a low response rate was representative of the population it was targeting. This means that only the most dedicated and involved homeschool parents volunteered to participate in Ray’s study. In other words, Ray was studying what one might call the cream of the crop of homeschooling families. It is incredibly deceptive to suggest that Ray’s results could be generalized to the homeschooling population as a whole, and yet that is what this infographic does. This would be like passing out surveys at suburban PTA meetings and then passing off the results as though they are representative of the average public school family.

So here’s a question. Do we actually know the education levels of homeschool parents? Yes, actually. We have data on this point from the National Center for Education Statistics, which conducts studies on random samples of the population. Unfortunately, while the NCES sample size is large, some of the numbers bounce around enough to suggest that it may not be quite large enough. In addition, a significant population of homeschooling parents are anti-establishment, and would not participate in the NCES should they receive one of its surveys in the mail. So the NCES numbers are not perfect. However, they still offer a better idea of homeschool parents’ educational backgrounds than does any other piece of data we have. And what do those numbers say, exactly?

Parents' Education fancy chart colors matched

That looks a bit different, doesn’t it? According to the infographic, only 1.4% of homeschool fathers and 0.5% of homeschool mothers do not have high school diplomas. But according to the NCES data, a full 11% of all homeschooling takes place in families where neither parent has a high school diploma. Further, according to the NCES data, only 39% of homeschooling takes place in families where where at least one parent has a bachelor’s degree. This contrasts with the infographic’s claims that a 66.3% of homeschool fathers and 62.5% of homeschool mothers have bachelor’s degrees. While the NCES data isn’t perfect, it is the best data we have and the only data on this subject that uses a randomly selected sample and therefore the only data that can be argued to be representative of homeschoolers as a whole. Interestingly, the NCES data suggests that homeschooling parents have, on average, essentially the same level of education as other parents (compare the two here and here).

Rather than relying at the NCES data, which is the best data we have, this infographic relies on the data from Ray’s descriptive study. Remember that Ray was studying the cream of the crop of the homeschooling population. It shouldn’t be surprising, then, that his sample was much more highly educated than the average parent. What should be surprising is that this infographic would be so misleading as to claim that those numbers represent the homeschool average. It’s especially odd given that the creators of the infographic were obviously aware of the existence of the NCES data, because they used it as their source for the information in the very next section.

HSI Reasons

This section uses the NCES data from 2007. While this data has some limitations, as I’ve already mentioned, it is nevertheless the best data we have. Like the section on the number of homeschooled children, this section is a breath of fresh air. I would however offer two small caveats, however: First, the questions the NCES uses to ask about parental motivation for homeschooling are not well designed, which can be challenging for researchers. Second, the infographic only lists what each family put as their number one reason for homeschooling, and leaves out how many parents chose each option when they were allowed to mark more than one reason. You can read that information here, and you can see how parents reasons for homeschooling have changed over time here.

HSI Academics

In contrast, this section is a disaster. Once again, the infographic relies on data from Ray’s 2009 study, and once again that is a problem. Let’s talk just a little more about this study. Ray released his findings in an easy-to-read summary in a highly deceptive manner. In other words, Ray’s findings do not actually say what he says in that summary that they say. Ray’s full study, which he published in 2010, can be read here. In his published study, Ray explains that “this is a nationwide, cross-sectional, descriptive study . . . . It is not an experiment and readers should be careful about assigning causation to anything.” He also states that “it was not possible within the constraints of this study to confirm whether this sample is representative of the population of home-educated students.” In other words, Ray was not studying a random sample and his results cannot be assumed to be representative of homeschooled students as a whole. Yet, both the summary of his findings and this infographic state that Ray’s findings are representative. I am at a loss as to whether this is the result of ignorance or malicious intent.

Remember that Ray found that the participants in his study had parents who were significantly more educated than the average parent. Remember that the NCES data suggests that homeschool parents are not actually on average more educated than the average parent. This alone makes it crystal clear that Ray is not studying the average homeschooler, and that his results cannot be generalized to the homeschooling population as a whole. More than this, Ray does not correct for background factors. His participants are from families that are more educated, more financially well off, and more married than the average family. We know that public schooled children from families that are more educated, more finically well off, and more married outperform other public schooled children. Because Ray does not correct for background factors, which would involve comparing his participants to their demographically matched peers rather than to the national average, what he is actually studying is not the effects of homeschooling but rather the effects of growing up in families that are more educated, more financially well off, and more married. And yet, this infographic suggests not only that Ray’s results reflect the average homeschooler (they don’t) but also that their higher level of performance can be credited to homeschooling (without correcting for background factors, which Ray does not do, it cannot).

So, what do we actually know about homeschooled students academic achievement? A number of things, actually. The state of Arkansas tests homeschooled students annually, and releases those findings. It finds that homeschooled students score in the low 60th percentiles, and score far better in reading than in math (in math, they are only slightly above the public school average, and are sometimes below it). Unfortunately, we do not have the data to correct for background factors, and therefore cannot say whether or not these students outperform their public school peers. By this I mean that, for example, students whose parents are better educated do better in than other students regardless of whether they are educated in public, private, or homeschools. If these Arkansas homeschool students have parents who are, on average, better educated than their peers, that may explain their slightly higher test scores. The Rudner study, conducted in 1997, found a full 20 percentile point difference between the children of homeschool parents with college degrees and the children of homeschool parents without high school diplomas, indicating that parental education does affect homeschool performance. We also know that homeschooled students are more likely to have two parents living in their home than are public schooled students, and studies suggest that having two parents living in the home improves students test scores. In other words, we do not know whether Arkansas homeschoolers’ slightly higher test scores are a result of homeschooling, or of factors such as parental education and family status. Regardless, the Arkansas numbers show that this infographic’s claims that homeschooled students score on average in the 87th percentile are flat out false, and demonstrably so.

We also have some interesting numbers from Alaska, which I’ve played around with a bit. Most homeschool students in Alaska enroll in public school extension programs designed to serve homeschoolers. These programs offer students monthly contact with certified teachers and require parents to create an educational plan for each child each year and to file quarterly progress reports. The students are tested at the end of each year, and the parents receive around $2000 per student per year for use on textbooks, computers, and tutoring. Each of these extension programs must make its test scores public (here is an example), and I’ve compiled the scores for the largest of these programs as follows. In reading, 86% of homeschooled students were proficient or advanced as compared to 80.1% of public schooled students. In writing, 77.3% of homeschooled students were proficient or advanced as compared to 74.2% of public schooled students. In mathematics, 64.3% of homeschooled students were proficient or advanced as compared to 68.6% of public schooled students. In other words, these Alaska homeschool students outperformed the public school average in reading and writing and underperformed the public school average in math. This data does come with background factors, though the results of that analysis are too complex to discuss here. Of course, it should be remembered that this is only homeschooled students enrolled in the public school extension programs, and does not include those who homeschooled under Alaska’s minimalist homeschool statute. Still, the results are intriguing.

What else do we know? Well, we know that homeschooled students score slightly above average on the SAT and the ACT. However, a study of homeschooled students who take the SATs found that their parents were significantly better educated than were the parents of public schooled students taking the SAT, suggesting that homeschooled students’ slightly better performance on college entrance examinations like the SATs and ACTs may be largely attributable to their family backgrounds rather than to homeschooling. Further, homeschooled students are less likely than public schooled students to take the SATs and ACTs, suggesting that homeschool graduates may be less likely to attend college than public school graduates and that, as a result, homeschooled students SAT and ACT scores may be inflated by the fact that only the best of the best of homeschoolers take these tests.

Can homeschooled students succeed academically? Absolutely. Do all homeschooled students succeed academically? No. One question that never seems to be brought up is why successful homeschoolers succeed. First, homeschooled children in well educated middle class families would likely score well above average even if they attended public school simply because of their socio-economic status. Second, It takes hard work and dedication to succeed at homeschooling, it’s not like you can just apply the label “homeschooling” and get instant success. There’s also the fact that some students who would have wilted in public school thrive at home, and some students who would have thrived in public school wilt at home. Which educational method works best for which child may vary greatly, but many homeschooling advocates act as if homeschooling is some sort of universal formula that works automatically for every child.

HSI Grow up

HSI Involvement

This section relies on Brian Ray’s study of homeschooled adults. It should be noted that Ray is associated with HSLDA, the country’s leading homeschool advocacy group, and is not an unbiased researcher. For more, see this article. Anyway, in his study of young adults who were homeschooled, Ray did not use a random sample or correct for background factors. Further, his study was composed of highly leading questions and was not scientific in any sense (for more on Ray’s habitual use of leading and unscientific questions, see this critique of new study of homeschooled young adults that Ray is currently spearheading). In other words, Ray asked current homeschoolers to help him find homeschool graduates who would be willing to fill out a survey on their experiences. It’s not surprising that the homeschool graduates who would volunteer to participate in such a study would be those who most positive about their experiences. For a study conducted by Homeschool Alumni Reaching Out, a group dedicated to helping those homeschool graduates who experienced some form of abuse or educational neglect, see here. The group’s 2013 Basic Survey found that its participants ranked their academic experience in the home fairly low and experienced high rates of physical and emotional abuse. Neither Ray’s study nor HARO’s study used a random sample and neither can be assumed to be representative, but then, that’s kind of my point. 

The only random sample we have of adults who were homeschooled is in the Cardus Education Survey, which corrected for background factors like family income or parental education. The Cardus found that being homeschooled negatively impacted a student’s future educational attainment and earning potential, and made students less likely to attend college and, for those who did attend college, more likely to attend a community college and less likely to attend a prestigious university. Further, they were less likely to be politically engaged as adults. Homeschool graduates were also significantly more likely to report “lack of clarity of goals and sense of direction” and “feelings of helplessness in dealing with life’s problems” than any other group. Finally, they were also more likely to marry early, to have fewer children, and to divorce. Again, this study actually corrected for background factors and it used a random sample. While further research is badly needed, the Cardus suggests that Ray’s congratulatory survey is far from the whole truth when it comes to homeschooling outcomes. 

HSI Costs

Finally, the section on how much it costs to homeschool a child is highly misleading. If I were to leave my career and homeschool my own two children, that would cost our family tens of thousands of dollars a year in lost income, not the $500 per child listed in the infographic. When a family homeschools, they give up an income, because, in effect, they have to have a teacher—i.e., one of the two parents—employed full time teaching the children. And yet, that cost, that sacrifice, is routinely left out of infographics like yours. If a woman who made a salary of $50,000 chooses to quit her job and stays home to homeschool her single child, it costs that family $50,000 to homeschool that child, even before counting in books and supplies. If that woman quits her job and stays home to homeschool her five children, it costs that family $10,000 per year to educate each of those children, even before counting in books and supplies. Suggesting that homeschool parents’ average of $500 per child in school books is a figure you can compare to the average $9,963 public schools spend on educating children is highly disingenuous, and that’s me being gracious.

HSI Footnotes

Notice something about these footnotes? Very few are to actual good academic research on homeschooling. The history of homeschooling links are lay articles, not scholarly pieces. Lay articles have their place and are sometimes very good, but for something like creating an infographic it’s important to use sound academic data—and, well, this infographic is an example of what happens when you don’t do that. Now there is a link to the NCES data, which is some of the best data we have. However, the infographic relies on that data selectively and completely ignores it when it comes to areas like parental education (if you have any skill at reading charts, you might find these summaries of the NCES data interesting: herehere, and here). Further, there is not a single link to the International Center for Home Education Research, which was founded by Milton Gaither and Robert Kunzman and is probably the best resource currently out there on research on homeschooling. Finally, this infographic relies on Wikipedia for the history of education, rather than scholarly books and articles on the subject. This might help explain why its history section was so terrible, and it should be a sign that the people who put this infographic together did so without putting in a lot of effort.

This infographic does nothing but mislead readers and misrepresent data. It may be a nice pat on the back for those homeschoolers who want to make themselves feel good, but it’s not good research or accurate information. And you know what? This sort of terrible scholarship does nothing but hurt the image of homeschooling. These sort of infographics make homeschoolers look either ignorant or willfully deceptive. If you want facts about homeschooling, I suggest you go elsewhere.