The truth of your memoir has to revolve around fact, right? Otherwise, you may be writing something super, but it’s probably not memoir. At the same time, one of the greatest challenges to writing memoir is that facts also have a tendency to obscure the truth of your story at times. Sometimes you have to leave them out. Or even change them a bit.
So, definitely stick to facts, but don’t stick to too many facts. You’re welcome! Good luck with that. It’s a confusing bit of advice, I know, but writing memoir is about filtering through all the facts, choosing the ones that contribute to the truth of the story you’re telling and leaving the ones that are irrelevant or distracting in your memory, but out of your memoir.