How to easily reject results from Happiness research

Will Wilkinson is giddy over happiness research these days. One of the results he cited the other day was controlling for everything, kids make people less happy.

Here’s why that’s complete bullocks, as they say. Let’s say you have a survey that asks people whether or not they have kids (K) and asks people to rate on a scale from 1-10 their satisfaction with their marriage (M*) and on the same scale ask their overall happiness (H). Of course these survey instruments are more sophisticated then that, but the point is you have two types of questions, ones with a lot of measurement error (and actually is just a proxy) and one that has little or no measurement error.

Now, people can report whether or not they have kids with tremendous accuracy and precision. We’re great “kid counting” instruments. We’re lousy at measuring our own satisfaction (M), though. We’re inaccurate at measuring our satisfaction because its not clear what scale we should be using. We’re answering “satisfied – 8” on the wrong 1-10 scale when we should be answering “satisfied – 132” on the negative 17 to infinity scale.

Compounding the problem is that having kids probably changes our measurement of our own satisfaction with marriage. By this, I don’t mean that there’s necessarily a correlation between true satisfaction and having kids. I mean our measurement of satisfaction is correlated with having kids.

We’re also not precise instruments of satisfaction measurement. We would answer “132” on the correct satisfaction scale but we can’t really differentiate between a 132 or a 133. In fact, we probably have a whole range of satisfaction scores that we wouldn’t be able to differentiate between.

So what? Well, the problem is that this means the satisfaction with marriage score we get from surveys is a proxy for a true measure of satisfaction and it is correlated with having kids. Proxies correlated with other regressors introduce biases that make it more likely to find kids make us less happy in general even though in reality this isn’t true.

UPDATE: I think my post as previously written was confusing where proxy/measurement error comes into play. The original study regressed overall happiness on having kids and some other subjective, poorly measured variables (like happiness in marriage). There are two problems with the measurement of those subjective variables. The first is that the measure itself may be correlated with having kids. People with kids may believe kids makes marriages more happy. The second problem is standard measurement error (i.e. lack of precision and accuracy). I’ve rewritten a little in hopes of improving clarity.

UPDATE 2: Take 2. I replaced “happiness in marriage” with “marriage satisfaction” so as not to confuse that with overall happiness. YouNotSneaky! is so much better at this…

The math below the fold proves these point.


Let the variables be defined as above with overall happiness H. So true happiness in marriage is M, measured happiness in marriage M* and having kids or not K.

Alpha_1 is a measure of the accuracy of the happiness measure and epsilon is a measure of its precision. Alpha_2 is the degree to which the number of kids is correlated with the measure of happiness. None of these things are observable so you have to make guesses at their sign and magnitude.

Beta_1 is the true weight of kids on happiness. The claim in the research Wilkinson cites is that this is negative. Beta_2 is the weight of happiness on the squishy variable (happiness in marriage in my example).

bias.JPG

Equation (4) is the result of substituting the measured happiness into the actual (and unobserved) relationship. We see (among other things) that the bigger the true effect of marriage happiness, the more negative the effect of kids becomes. This is due to the correlation between kids and the measurement of happiness in marriage.

Also, equation (4) shows us that more variation than is due will be assigned to the more accurately measured variable. The alpha_1 in the denominator of the coefficient in front of marriage happiness shows us this. Also, the messed up error term (its smaller than it should be because we subtract a bit off of it) tells us the tests for statistical hypotheses won’t be rejected as often as they should be.

In net, we should be wary of results that compare easily measured quantities to less easily measured quantities. Its possible, via analysis like the above, to show the measurement biases may stack too much in a tested hypothesis’ favor.

BTW, this analysis applies to results on income and happiness, too. Income will be biased negatively, i.e. its importance will be under-weighted. I haven’t read Wolfers, but I bet he’s taken this issue into account.

13 thoughts on “How to easily reject results from Happiness research”

  1. I skipped the stuff below the fold. However, I do know that measurement error in a dependent variable is not generally a problem. That is, we don’t usually worry about it biasing our estimated coefficients on regressors. It’s not clear from your “above fold discussion” why the this is different.

  2. I don’t think I understand what you’re saying, Will. That having children changes the function that maps feelings-as-actually-experienced to self-reports? Is that it? And if so, what exactly do you suppose is the psychological mechanism that leads parents to rate the same experienced life satisfaction lower on surveys?

  3. Will, yeah, I’m contending the self-assessment of happiness in marriage (or whatever) is positively correlated with having kids or not. You are told that having kids makes you happier in marriage (after all, what’s a marriage without kids, grandma says) so the fact you have more kids makes you bias your measurement up.

  4. Jason, I’m not just talking about measurement error. And I’m talking about issues with the measurement of independent variables (the regression is overall happiness on a kids indicator and some other subjective measures which are poorly measured).

    Gabriel, I think the problem is my equations are hard to read. Click on them to get a better view.

    I’ve updated the equations to indicate (1) and (2) are my measurement and happiness models respectively. Also, (3) is just a rewriting of (1) and (4) is just (2) with (3) inserted. (4) is the thing that actually gets estimated.

  5. Here’s a mechanism that accommodates your intuition and fits the data: given grandma propaganda, you expect to be happier with kids. When you aren’t, you’re disappointed, and report lower happiness, even if there is no objective change in feelings (other than that one salient disappointment).

  6. Well, in fact we want reported happiness with marriage to be biased up (alpha_2 to be positive) to get the effect I’m talking about in this thread. (Admittedly I made this point opaque in the original wording of this post… I hope I fixed it.) This upward bias makes some of the overall happiness resulting from kids show up in the marriage satisfaction coefficient.

    We need a mechanism that supports a positive correlation between measured satisfaction with marriage and parenthood not actual satisfaction and parenthood. So grandma says marriage with kids is happier… If I’m married without kids, this will make me under report satisfaction in my marriage. If I’m married with children (great show btw), grandma propaganda will make me over report. I think to myself, “I was going to mark down a 7, but I’m not sure; I could be an 8 or a 6. Grandma’s over there telling me marriages with kids are happier and she’s been around longer than me and probably knows better, so I’ll mark down 8.”

  7. But doesn’t that problem only appear if you control for “marriage satisfaction” in your regression? I haven’t seen any papers who do that.

  8. Yeah, but in general this holds for any variable that’s squishily measured. There will be a spectrum of squishiness when looking at survey data.

    Funny story actually. I thought I saw satisfaction in marriage as one of the controls cited by Will from the happiness book he’s reading, formulated this objection over the weekend on a road trip and posted it today. Only after all that did I go back to check and notice that that wasn’t one of the listed controls. I guess some of the listed controls (like politics or religion) could be subjective (my mom doesn’t know if she’s liberal or conservative… she’s definitely conservative, btw), but not as much as another satisfaction/happiness measure.

    Here’s a well-cited paper that doesn’t have the above problem and finds a negative effect of children on happiness.

  9. OK, I’ll come out and admit it. I’m teh stupid. I don’t understand this post. I don’t really understand the different between M, M* and H and I don’t understand the equations.

    I never got to the measurement bias chapters of any econometrics textbooks. I fell asleep much before them.

  10. Well, you see in the next post that I admit this analysis doesn’t do much as a critique of happiness research. They probably don’t suffer from this “squishy” control problem.

    That said, these are important issues in measurement error, etc.

Comments are closed.