which "physical" are we talking about? the perceptions we "seem" to be sharing? [data?] or something you believe to exist as the foundation of those perceptions [symbols in your mind]?... for which you have nothing more than belief or faith as to their existance, or am I mistaken and you have found a way to get outside your mind and see the source of data by some other means?
the universe may be made of objects which are the source of our perceptions of them.
the universe may be made of minds made of something completely unknown and unknowable by us --sharing data which originates in those minds.
in either case the ONLY universe we know is the one we perceive... so you can have faith in the "physical" model if you like, but I would point out that quantum physics has made that model a bit more complicated by saying that matter and energy is different depending on the test the observer uses to detect it...sometimes appearing as a wave and sometimes as a particle, and often times making no "classical" sense what so ever. [that is the idea of objective things does not make sense of the results of all tests].
some physicists reject the notion that the observer plays a key role in the things observered...but have not been able to demonstrate another solution. Einstein, btw, hated this aspect of quantum physics and fought it with the remainder of his life-- if I am not mistaken, he was quoted to have said, if I had known it would turn out this way, I would have been a plumber....
his famous quote-- "God does not play dice with the universe" was a rejection of quantum physical rules which demonstrated the possibility of true randomness having a place in the universe... he held out hope that some hidden determinism might be found... which some string theorists claim they have discovered... assuming a universe of more than nine dimensions.