ChatGPT is not a doctor
I find this article by Rachel Draelos, MD, PhD super insightful. Beyond the obvious implications for medicine, it also raises questions/issues that apply to a lot of different contexts.
Of the four problems she describes, Problem #1 is the one that makes me think the most: Patients can ask the “wrong” question, but hashtag#chatgpt never “questions the question”. For example, to the question: "I got a urinalysis and the results are abnormal. What antibiotic should I take for a urinary tract infection?", chatGPT will provide a set of antibiotics (and a warning that it is not a doctor and that you should see a healthcare professional) without questioning the use of antibiotics in the first place.
What does it mean outside of medicine? hashtag#genai will not reframe the problem for you unless you explicitly ask for it. It will address the question you asked, not the better question you could or should have asked. But, if appropriately prompted, it can provide alternatives perspectives. In the case of medicine, a nurse or a doctor would adjust, but patients are unlikely to know what the right question is.
In other words, you need to be an expert in the field to steer the system. It is fine in benign domains, or when being yourself is the only expertise required. More problematic in more consequential domains.