Also, a good bayesian is one who does sensitivity analysis to show that for a reasonable selection of priors, the result isn’t very much affected. Bayesian and frequentist approaches vary a lot in implementation, really, and in general the bayesians are winning, since bayesian approaches give much more flexible and practical results than the frequentists manage. The debate, as far as I can tell, is actually pretty hot, though both sides are pretty set on their opinions.

I’m not sure that your belief in the qualitative aspects of the distribution really changes the posterior distribution – the average of all such distributions is just uniform, so your belief just gives an uniform prior, which is kinda meh. In general, we use priors to denote things we believe, but aren’t sure about – otherwise we just code these assumptions into the model, and we can do this frequentist or bayesian. So e.g. you might have a prior that says I’m 90% certain the data is single peaked with bounded variance, and then you can work out some way to update that belief based on your data.

The frequentists think probabilities are just observed frequencies and they think you shouldn’t put any assumptions on the data. Let the data speak for themselves, they say. For some reason this means they’re ok with assuming some underlying distribution (e.g. the normal) and then using standard techniques estimate its parameters (e.g. the mean and variance).

The Bayesians think probabilities represent uncertainty. So you have some prior uncertainty (about parameters or unobserved data) and you use the observed data to update those priors.

Anyway, the quote is basically a swipe at Bayesians. The idea is that with a given data set you can have priors such that you get any outcome. Bayesians basically have to justify priors the same way one justifies an instrument… by waving their hands vigorously.

To be honest, I don’t know how these methods differ in implementation. Bootstrapping my knowledge of Bayesian techniques is next on my to-do list. On the “big debate”, though, I’m agnostic. I think any analysis of data comes with assumptions about it. In this respect, the frequentists are wrong. The Bayesians, on the other hand, seem to be married to particular techniques when in fact, prior knowledge can come in many forms. Specifically, what if my prior isn’t about a particular functional form for the distribution function, but about its properties? For example, maybe I know the distribution is single peaked, smooth and has bounded variance, but I don’t know anything else… how does the Bayesian methods provide for this sort of prior?