Should i give up and forget about this man?

If a guy really WANTS to see you, he will find a way. If he doesn't, well, there is your answer. If he wanted to see you he would have said so, not told you to "call back at 7". I wouldn't wait around for this guy any more. He's a waste of time.