Choose the course that will cause the least long term damage. If you can survive "giving in" and he can't, give in.
If "giving in" will be decision that you cannot live with and will haunt you for the rest of your life, don't wreck your life for anybody.