In addition to freedom of religion protections, there's also the Constitutionally-protected freedom of association, which includes the right to NOT have someone associate with the group, as long as their exclusion is not based on factors related to civil rights (race, color, creed, sex, handicapped/disabled, etc).
The funny thing is, Levitical laws excluding the disabled or injured from worshipping in the Temple would be SOOO much the grounds for an Americans with Disabilities Act lawsuit today... So much for superior Biblical moral principles....