"I wasn't sure how literally SW meant "filthy hands" when he was talking to Ghost Face (or whether it was like a "getting your hands dirty" kind of idiom), but it does come across as Extremely Literal and Specific in that context, you're right. :-/"

Yeah, I think the only person who is being metaphorical (or at least offhanded) here is Ghost Face. (And Shen Wei is reacting to him doing this with his literal hands: "Wholly unafraid, he glanced down and smoothed out Zhao Yunlan’s clothing, which had been blown about haphazardly in the wind.")