What if I told you that I can show you the difference between Bayesian and Frequentist statistics with one single coin toss?
Frequentist and Bayesian Statistics Defined
- Frequentist: This is a statistical approach that insists the parameter is not a random variable, which means there’s no probability. When a coin flips, a Frequentist will insist that there’s a correct answer. If the coin is heads up, the probability is 100 percent, and if the coin is tails up, the probability is 0 percent.
- Bayesian: In this statistical theory, the parameter is considered a random variable, which means probability expresses a degree of belief in an event. When a coin flips, a Bayesian will insist the probability of heads or tails is a matter of personal perspective. There is no right or wrong answer.
Before we go any further, the demonstration works best in video form, so don’t read the summary and spoilers below until you’ve seen the video below. In case some terms are unfamiliar, I’ve linked to friendly explanations to help you out.
Difference Between a Frequentist and Bayesian in Statistics
In the video, there’s a moment where I ask you, “What is the probability that the coin in my palm is up heads?” The coin has already landed, I’m looking at it, but you can’t see it yet. The answer you give in that moment is a strong hint about whether you’re inclined towards Bayesian or Frequentist thinking.
- Frequentist: “There’s no probability about it. I may not know the answer, but that doesn’t change the fact that if the coin is heads up, the probability is 100 percent, and if the coin is tails up, the probability is 0 percent.”
- Bayesian: “For me, the probability is 50 percent. For you, it’s whatever it is for you.”
It’s only by insisting that the parameter is not a random variable (Frequentist) that it makes any kind of sense to talk about your method’s ability to deliver the right answer. As soon as you let the parameter be a random variable (Bayesian), there’s no longer any notion of right and wrong. There’s only your personal perspective.
- Frequentist: The parameter is not a random variable.
- Bayesian: The parameter is a random variable.
One word, huge difference. Let’s take a closer look.
Frequentist vs. Bayesian
Frequentist and Bayesian Terminology
Which Words Tell You Who You’re Dealing With?
- Frequentist: Confidence interval, p-value, power and significance.
- Bayesian: Credible interval, prior and posterior.
What Are Their Goals?
What are they using statistics to change their minds about?
- Frequentist: Actions to take, otherwise known as default action.
- Bayesian: Opinions to have, otherwise known as prior belief.
What’s the Main Difference?
- Frequentist: The parameter is a fixed quantity, meaning there’s no probability about it.
- Bayesian: The parameter is a random variable, meaning there’s no right answer.
What Are the Advantages of Being a Bayesian or Frequentist?
What do you gain by joining their way of thinking?
- Frequentist: It makes sense to talk about your method’s quality and “getting the answer right.”
- Bayesian: This approach includes intuitive definitions, e.g. credible intervals are what you wish confidence intervals were (but aren’t).
What Are the Disadvantages?
What do you lose if you choose their side?
- Frequentist: The core concepts are harder to wrap your head around. For example, p-values and confidence intervals have counter-intuitive, wordy definitions and lazy thinkers frequently make a hash out of them.
- Bayesian: You lose the ability to talk about any notion of “right answers” and “method quality.” There’s no such thing as statistically significant or rejecting the null. There’s only “more likely” and “less likely” from your perspective. If there’s no such thing as a fixed right answer, there’s no such thing as getting it wrong.
So, Which One Is Better?
Wrong question. The right one depends on how you want to approach your decision-making. For example, if you have no default action, go Bayesian. Without a default action, the Frequentist approach is less practical than the Bayesian approach, unless you have special philosophical reasons for invoking the concept of truth in your calculations.
Those last three words are important. We’re not talking about the concept of truth in general, but rather about how it’s handled in the math that powers these approaches to statistics. The distinction between the two camps boils down to whether you treat the parameter of interest as a fixed constant or not.
Ok... So, Which One is More Objective?
Neither. They’re both based on assumptions, so they’re fundamentally subjective.
The key difference is how they assist decision-making once the decision context has been framed.
Wait, What About Sample Size? Isn’t Bayesian the Way to Go With Small Data?
If you’ve been hanging out with the “Frequentist if there’s lots of data, Bayesian if there isn’t” folks, you might be sold on the idea that you should let sample size decide which camp to go with. Alas, the reasoning behind their advice gets wobbly if you poke it.
Yes, it’s true that Frequentists spurn baby data sets. If you’ve got more fingers than examples, they’ll almost surely tell you not to bother.
Yes, it’s true that if you take a Bayesian approach, you can proceed with as little as one (!) data point. If the math checks out, sure, you can do it.
But should you?
Being allowed to proceed with a pittance of data might be a bug instead of a feature. There are circumstances where you definitely don’t want to be doing that. Statistics isn’t alchemy. We’re not making gold out of thin air. There’s the same amount of data in one data point no matter which school of thought you pledge fealty to.
The way to “require less data” is to make bigger assumptions. This holds for both philosophies, so, do take a moment to ponder the nutritional content of your conclusions when your main ingredient isn’t data but, essentially, some nonsense you made up. If you take yourself too seriously when working with tiny data, expert Bayesians and Frequentists alike will forget their differences long enough to join in a belly laugh at your expense.
Cassie, You’re Killing Us Here. Are You Bayesian or Frequentist?
Both. I choose based on how I’m framing my decision-making. It depends on whether the situation calls for choosing between actions or forming an evidence-based opinion.
Should I Pick a Side Between Bayesian and Frequentist?
I’d advise against committing to just one camp — unless you’ve spent a few years thinking about the philosophy of statistics, and you’re willing to die on this hill.
Honestly, it’s a little silly to declare yourself as one or the other unless you’ve pondered them very deeply. Having had the pleasure of doing my graduate work at Duke University, which is to Bayesian statistics approximately what the Vatican is to Catholicism, I noticed that the loudest loudmouths about the superiority of Bayesian statistics aren’t the professors, rather it’s the newbie students who are relieved not to have to memorize the definition of the weird Frequentist confidence interval anymore. The Bayesian credible interval is so much easier.
The professors understand that “better” depends on what you’re trying to do. They spend a lot of time thinking in the Bayesian way because it fits the kind of decision approaches they’re interested in. So, my advice? Don’t pick a side. See them as two different approaches that fit two different styles of decision-making and reasoning. Then, leave yourself the option of using whichever one suits the mindset or context you find yourself in.
