The question is not whether or not you believe it, it's whether he believes it. Whether or not these things actually happend to him is something else, but he may in fact believe it did happen to him.
As to whether or not it affects the relationship, it just depends on your relationship. Some people seem to be really good about compartmentalizing their relationships so that other things doesn't seem to be relevant and therefore doesn't affect the relationship, but I for one couldn't be partners with someone that is mentally unstable or live in their own world, nor do I really want to socialize with someone that is in the same state. So it simply depends on whether you have an issue with that or if he has an issue with you not believing him. I use the word relationship in a functional way, although I state my preference based on relating to the whole person I do not mean to imply this is the only way or the right way. You'll have to decide what is acceptable depending on how you two relate.