2pink it's really horrible when your own flesh and blood show loyalty to a cult/publishing company rather than their own family. I know as I personally am experiencing myself, my husband and son being shunned by our JW families. I was told I was satanic, a liar, dead as far as they are concerned etc - all this vile stuff was said to me by my own father in our last phone conversation 18 months ago. And it hurts. However, even though they have cut us off completely my situation is a bit different in that my son is now 20.
So having experienced first hand how our own parents can say such horrible and untrue things about us to our face and no doubt plenty more behind our backs, I personally would be very reluctant to let my very young children be alone with their JW grandparents. It is very possible your children will be told things like "mummy and daddy are going to die at Armageddon", or "mummy and daddy are bad because they don't love Jehovah" etc (I'm sure you get the picture). As your children are very young, being told this kind of stuff can have a real impact on them, both now and in years to come. And that is the last thing you would want now you are out and living a normal family life with your children.
You very much have the right to say that you will be with the children during the visits and that there is to be no JW talk and no running down of you and your husband. After all there has been articles in the Awake magazine quite a few times where grandparents are told they have to respect their childrens right to parent their own children their own way.
Wishing you all the best in this difficult situation