Hang in there and make sure that you send him cards and letters. They will probably get thrown away; but, eventually he will be old enough to pickup the mail himself and then he will know you care and may decide he wants to know you.
It worked for my wife and I, the two granddaughters the we were not allowed to see for their first 9 years and only saw 3 times in the next 9 years are now closer to us than their parents.
Philip