Is he after rekindling things?

At one time I had a relationship with a chap, we were in love, but I realized that he was very possessive and I got out of the situation, there was a lot of aggro and hurt when I ended it, but after a while we became friends, and once in a while we have a cuppa in town together, nothing more. He's now said it would be nice to meet away from the cafe where we could chat more freely, but something tells me he's after being intimate again, and I've said 'no', he now thinks I don't trust him, and feels hurt, have I misjudged him? I trust the fact he'd never do anything to hurt me, but I wonder what his motives are, am I being too wary?