Why did his marriages end? I am not looking for you to answer that here. That is for you to think about. When someone has 2 failed marriages, perhaps there is something wrong with him and his commitment to marriage and his ability to communicate. That is something you need to think about.
Does he have children from either relationship? If not why? How does he treat your children? How long did those marriages last and why were there no children?
Again, please don't open your relationship information here. This is all for you to think about and think about the answers.
As for your children: Where is their father? Is he in their life? Are they looking for a father figure?
Being the child of divorce I was hoping my mother would get re-married and I would get a father since my dad re-married and had a new family and seemed to throw us to the side.
She dated and dated and brought various men, JWs and worldly home. Each time I hoped. Each time my hopes were dashed. I gave up. When I least expected it she eloped with a man she barely knew. When I was 20. He beat the crap out of her. Numerous times. While I was initially happy she was married, her husband quickly turned into a beast. Divorcing him was a nightmare. So much for a new daddy.
So, you need to know why he is divorced twice, etc. Even in a small town.
As for your family and it being a small town: if you move in, they will gossip. They will vilify. They will give your kids a hard time. Are you and do you think they are prepared for that?
Plus, you have a lease. If you break it, it could affect your credit and your landlord could make things financially difficult. That is also something to think about.
If this man truly loves you, he will wait for your lease to end and you can go from there.