This is the experiential fallacy. You don't have to be a carpenter to know whether a table is well-made or not. As for advice, the professions are full of people who advise others about situations in which the professional has no personal experience. E.g. there are female and male obstetricians who never bore a child themselves, surgeons recommending procedures they haven't had, counselors advising on situations they've never experienced even vicariously... and the list goes on. Not sure why clergy would be uniquely targeted in this respect.
"... They ne'er cared for us
yet: suffer us to famish, and their store-houses
crammed with grain; make edicts for usury, to
support usurers; repeal daily any wholesome act
established against the rich, and provide more
piercing statutes daily, to chain up and restrain
the poor. If the wars eat us not up, they will; and
there's all the love they bear us."
-- Shakespeare: Coriolanus, Act 1, scene 1
