Competence without Comprehension
In 2011, I had the privilege of guest writing James Fallows' blog on The Atlantic for one week. One of the posts was about Darwin's "Strange Inversion of Reasoning" (the title of a 2009 PNAS paper link), or "Competence without Comprehension", a theme introduced (to me, at least, at the Santa Fe Institute) by the late philosopher Daniel Dennett, who passed away last week. Competence without comprehension, an idea"best illustrated by mathematician Alan Turing's proof that a mechanical device could do anything computational" link should resonate with everyone who has interacted with an hashtag#LLM, as the latest iterations of technological evolution have given us highly competent machines that don't understand themselves and that we can't fully comprehend.
For centuries humans have been harnessing the immense power of natural evolution to create today's food supply (for example, maize did not exist in nature in its nutritious form) or dog breeds without understanding its underlying mechanisms. We understand it better today, of course, but we are now doing the same with digital organisms that we are breeding without (full) comprehension of their underlying mechanisms.