A chicken & egg dilemma

Or from the other perspective, if you have to be fit to have fun and have to have fun to be fit, where does that leave you now?
Lots of people ride mountain bikes. From pre-teens with their friends and parents up to aged grandparents and everyone in between. Some have been riding for years, many are new to the sport. Most, especially once you get past about 22 years of age, lean towards the XC end of the spectrum over the DH end. Yet across all these various people, most of them ride for the same reason - because it is fun. Not to race, not to win.
I know many recreational mountain bikers who are more than willing to offer that if they had a higher fitness level they would enjoy their sport more. It is not much fun to be the slow one on a social ride, nor is it great to have to stop mid-climb to find your lungs. It is a self-powered sport, the rider is the engine. High-performance vehicles are more fun than sedate ones and that applies to bicycles too. Having good legs means you can top climbs, ride for as long as you want, back up the next day for another ride and so on, all without worry.
The logical progression is to ride more. For fun, that works great. The more you ride, the fitter you get, the fitter you are, the more fun you have. Nice feedback loop. For racing it works too, but only to a point. You start racing, you get your butt kicked, you ride more and so on. Eventually you plateau and you will not improve much without specific training.
Lots of people today are time-crunched. They can ride some, but not as much as they need to move their fitness upwards. Every ride causes soreness and regret as well as the desire to ride more. Specific training can also assist these riders to improve within their limited time frame. Yet sometimes they can end up spending all of their time preparing to ride and none of their time actually riding.
My suggestion is to re-jig your life so as to have more time to ride. But I also realise this isn't practical for everyone.
It was pretty much this issue that led to my retirement from the game of tennis. The amount of time I had to put in to play as well as I wanted was not practical for my life at the time, and not playing as well as I wanted was more frustrating than it was worth. So I quit. More correctly, I diverted my attention to cycling. I had returned to the bicycle to get more fitness for tennis (ironically) and found that with only a little riding I was performing better on the bike than the court.
The bottom line with cycling is that more is almost always better.