In fact physicists insist it makes no sense that we can only remember the past. Why can't we remember the future?
Penrose (Sir Roger Penrose if you're nasty) already has peed in this pool or at least has stated his intention to jump in and try. Straining his tiny little scientist's bladder.
I will chide him in the next, fascinating installment of String Theory Backstory. I do a really good chide. Even if I may have to finally concede a point to the chidee. You'll love it.