If he calls then thank him for his concern but you do not wish to meet with him as you are busy. If this changes you have his number and will contact him. you are not permitted to receive personal calls at work and under no circumstances should he try to contact you on your work number. Thank you again for your call. Good bye
Put the phone down, don't get into a conversation, keep it firm but friendly. You do not have to explain yourself, or comply. Focus on what you want to say and give him no way in. He doesn't know you he is a total stranger to you and therefore it would be inappropriate to meet with him.