If the grandparents want to be part of your baby's life, then they have to include you too. If they are going to disrespect you by shunning you and excluding you, then they don't get to see the baby.
Don't allow unsupervised visits. Ever. They will exploit unsupervised visits and use the opportunities to take your baby to the Kingdom Hall and in Field Service™.
Real grandparents take their grandchildren to the movies, to the park, to the zoo, and other fun places. JW grandparents take their non-JW grandchildren to the Kingdom Hall and Assemblies.