Apparently, the laws of physics do permit time travel. There's a lot of paradoxes to grapple with, but a recent paper proposes a model in which they "may be ruled out by the weirdness inherent in laws of quantum physics," as summarized at NewScientist.com:
"If you travel into the past quantum mechanically, you would only see those alternatives consistent with the world you left behind you."
(But wait. Will I be able to act at all, or simply observe? If I'm travelling back to where the quantum waves recombine, will there be two of me, or will I be my past self? I need a nap.)