I published my first short story when I was 39, and my first novel when I was 45. I would never have been able to write what I wrote without the life experiences that forty-five years had given me.
Is your book creative nonfiction where you're showing "real" events in an entertaining and story-like way? If writing in this literary genre do your homework and make sure you're actually doing nonfiction. It can get a little fuzzy when the truth is stretched too far on the creative side (thus turning it into fiction) but still represented as fact.
Starting can be the most difficult part of the journey. You know what you want to say, you've plotted it all out (or not), and are eager to finally make a go of getting it done. We all start somewhere, but there is an element of patience involved. Even for seasoned writers who know what direction they're going it can be a challenge.