When Your AI Explains Itself (And Gets It Wrong): Rethinking Trust in Language Models for Food Safety