“Are you nuts? Pursue your dreams?” Joe gasped. I had just spilled the beans about a big decision to quit my job over a glass of chardonnay at happy hour. I was slowly getting used to this reaction.

“Maybe!” I laughed, my smile concealing my disappointment. But I felt then (as I do now) that I was making one of the sanest, calmest, most certain transitions I’d ever made.

You should’ve heard what other people told me (relatives included!) when I said I was giving up my high-paying sales career to be a full-time confidence coach and writer. The laughs I got, the wide-eyed reactions, the sniggers. Did I care? Yes. Were my feelings hurt? Certainly. But did I care enough not to do what my heart was telling me, just so other people would approve of me?

No. Freakin’. Way.

Put simply: What other people think about your life is not your business. Your business is to figure out what you most want in this world and then pursue it like a 6-year-old in sight of a Toys ‘R’ Us sign.

After life coaching hundreds of people, I’ve learned there is one mega-watt, sprinkles-on-top reason that people turn their back on their dreams: the fear of what other people will think. We can handle change. We can deal with some financial uncertainty. We are probably game to take a risk. But to do it in front of all the people watching, observing, perhaps even waiting for us to fail? The shame would be too big to bear!

Besides the fact that people take a lot less notice of you than you’d ever think, here are five very good reasons why you should follow your dreams even when (and especially when) others don’t approve.

1. It’s your life, baby.

Will the people who disapprove of your choices pay your bills? Will they be your therapist when you suffer from discontent and regret? Will they be there to hold you when you break down one random Tuesday, walking home (in the rain) from your unfulfilling job in five, 10, 25 years?

Didn’t think so. So don’t let them make your choices today. Why would you let two minutes of another person’s thoughts affect your entire life? And remember this when you’re tempted to judge others’ life choices: Only you know what matters to you, and the same goes for everybody else.

2. Naysayers are projectors.

People who are unsupportive are often in delusion or denial about their own decisions. They’re simply jealous of what you are doing—what they wish they had the courage to do. People will either support you, be indifferent to you, or disagree with you. Only the supportive people count. I beg you: Get around those people.

3. Something magical is brewing when resistance is strongest.

In The War of Art, Steven Pressfield says, “Are you paralyzed with fear? That’s a good sign. Fear is good. Like self-doubt, fear is an indicator. Fear tells us what we have to do. Remember one rule of thumb: The more scared we are of a work or calling, the more sure we can be that we have to do it.”

Fear is a good sign! Worrying about what people think is an extension of this fear. What we want is so important to us that it makes us terrified. Our fear seeks critics to be proven correct—that what we want is out of reach. This fear will trick you like a thief and rob you of your dreams. Don’t let it!

4. Statistics don’t mean sh*t.

Ever heard of the term “skillionaire?” Neither had I until I was doing research for a course on side hustle I was teaching. This word describes people who support the truth that success has no prerequisites. And there is a boatload of living, breathing proof that you don’t need an Ivy League education to succeed in anything. You don’t need connected parents to propel you forward. You don’t need any of the things people tell you that you need to achieve your dreams.

Case in point: Richard Branson, Rachael Ray, Pete Cashmore, Kelly Ripa, Michael Dell, Louise Hay, Simon Cowell, Tony Robbins, Chelsea Lately, Bill Gates, and Mark Zuckerberg all lack college degrees. Did that hold them back? Hell no!

If you want to be an entrepreneur, writer, entertainer, motivational speaker—whatever it is—and you keep hearing how you’ll never make a living, remind yourself of the above success stories.

5. “No one kicks a dead dog.”

Dale Carnegie famously said this in the 1930s. It means if someone is trying to put you down, it’s a disguised compliment. Why else would they bother? Critics, haters, and cynics prove you are growing. Name any public figure you love and admire, and I bet you my life savings I can find infinite, poisonous put-downs online they’ve been too busy to read. Smart move—do the same.

No one, besides you, gets the final say in your life. And no one can choose courage for you either. Following your passion already involves a lot of hard work, so don’t invite more of it into your life under the guise of “concerned,” “realistic,” “are-you-nuts?!” types of people.

Guess who called me the other day? My old friend Joe. A mere 18 months later, he wants to build his own business too but is lacking self-belief. He asked to meet to up for some advice. The irony was not lost on either of us, and we shared a good chuckle.

In a world packed with cynics, don’t be one of them. Be too busy doing your thing. Focus on your own goals. Do the work. And the only person you need on your side? It’s you.

This article originally appeared on Greatist.com