Ned, Arthur, and a single combat that didn't happen

Single combat has always seemed unlikely to me. But, based on Ned's memories of the event (or what seems to be the event), I have wondered whether it something other than combat, maybe steel and shadow coming together in some sort of ritual.
But my understanding is vague and half-formed and years of pondering hasn't produced a real theory. At it certainly wouldn't be below GRRM to have two noble warriors come together to have one stabbed in the back.
I keep coming back to that bit between Arthur and Ned, "And now it begins," .."No, now it ends."
Somehow, I don't think they were disagreeing on the facts or what was about to happen. For one, it was a beginning. For the other, it was and ending. But I'm not wedded that. Maybe there was a terrible surprise waiting for Ned in the Tower of Joy.
So what do you think was waiting in that Tower? and if Ned didn't know about it, why was he there?