Jarrett, As somebody who started out too old, and is now way too old to race for any other reason than fun, I can say that the secret to racing for fun, is to find a class of like minded riders to race with. The only times I've run into problems was while racing in classes where people were far more serous than I was, and were willing put others around them and themselves at far more risk than I was willing. Also, as someone who was self employed, of the last 15 seasons, I did not have insurance the first 10. I figured that if the time came I had to divert all those race fees to paying off a medical bill, it would be sign from above that I should discontinue. I race far more conservatively than I am capable of, even with insurance now, because I know that if I go down at my age, I'm not bouncing back up.
I found the ultimate refuge in racing with guys who are there to have fun, and don't much care about who wins, or creaming somebody to take home a plaque, and who will pull their own parts to help out their competitors. They are all different ages and backgrounds, and range from seasoned, over the hill veterans to beginners riding machines older than they are.
To me, the choice is obvious. You need to get a 350X since you prefer a 4 stroke, and become a part of OTC Racing!