As I understand it, panpsychists don't typically argue that "rocks are conscious". Galen Strawson, for example says it's a misconception that he "thinks the chair is conscious".
Rather what we're talking about is whether matter is experiential in some sense, so that awareness is a continuum rather than a special property of some beings.
So for example we can agree that humans are conscious. And most can probably agree that cats and dogs possess some form of consciousness. Probably the same with mice. But where do you draw the line? Are spiders and flies conscious? We might say they probably have a very low level of awarenss, but not consciousness as such. What about tad poles? Single cell animals? Plants? They are not conscious but do they have any level of awareness? And obviously when we talk about plants or even rocks having "awareness" we are not talking about high order "thinking". The basic point is that awareness is a property of all matter, not that all matter has the same level of awareness. Like how a candle and the sun are both combusting, but the gulf between the two is unimaginably huge. This might illustrate the difference in awareness between a human and an ant. The gulf between an ant and a rock may be larger still. But the point is that consciousness isn't something that arises magically when matter reaches a certain order of complexity. Rather awareness is a property which is already present in matter to begin with, so that the potential for consciousness is already there, and we need not invoke mysterious notions of "spirit" as in dualism, or "radical emergence" as in materialism.