Sometimes if you walk near Yoshiro, she acts as if she's been suddenly spooked by something until she realizes it's Soh and then bows her head in greeting. Complementing the idea Soh can only manifest because of Yoshiro and so can only be seen by Yoshiro.
And in cutscenes and in-game, the villagers don't notice Soh at all. When we meet the carpenter, he's talking to Yoshiro only then, which explains why Soh looks to Yoshiro for her reaction to the carpenter.
They get spooked the same way Yoshiro does. But Yoshiro can see Soh...so I'm assuming Yoshiro enters some kind of meditative mind-state to perceive Soh.
I'll record some better footage soon and compile spooking videos.