A lot of people struggle with this: Reality is variable between observers. It is utterly impossible to arrive at a single inclusive answer. It’s not a question of what is, but of how we must approach the question, and what we should expect to find. How do I talk about such a thing?
If you approach scientific research into physical reality with the assumption that, in theory, everything can be eventually nailed down precisely as to how it acts, then you will never arrive. This is particularly true when you start to press down into subatomic levels, as well as the other extreme, to galactic levels or larger. The effort to measure will break down.
In practice, reality itself is inconsistent at some level. It is not possible to correct either the observer, the instruments, or the process itself, so as to overcome this invisible barrier to investigation. The barrier itself is elusive; it cannot be identified except in terms of effects after the fact. A lot of stuff in science fiction will ever remain fiction, because it’s not possible to nail things down to the point that you can begin to manipulate matter and energy with sufficient precision to get the results you want.
The major flaw is in the underlying assumptions about reality. I’ve written about that often enough. Reality is intentionally flawed, and so are we as a part of it. It is not possible to step outside into some imaginary objectivity as long as we are in our mortal frame. To step outside means leaving this mortal frame permanently. As long as we refuse to consider that there is a realm outside this reality, we cannot begin to grasp the nature of things. We are inside of a bubble, and the time-space boundaries are themselves a universal deception. But in our native form as mortals, we cannot ever hope to escape.
This is part of why algorithms inevitably break down. No algorithm can cover all the factors in the first place. But the variability-in-effect of reality will eventually break down even what the algorithm does cover. All algorithms are generated from flawed humans, and the product cannot avoid having the same flaws. The flaws are in the source, the process and the product.
Thus, no AI will ever exceed the flaws. The mythology that foresees an omniscient AI will never become reality. We may have the appearance of such an AI, but at some level of output, it will stumble over the inconsistency of reality. Calling it “entropy” is not going to answer the fundamental question. That fancy word simply acknowledges the effects, not the cause. The flaw is in this reality itself.
Any attempt at AI will always pull up short of human dreams, whether benign or malevolent. It’s not a question of matching the capabilities of human will and imagination; AI cannot escape the limitations of reality’s inherent flaws. I have no doubt that we will eventually see one or more attempts to seat an AI as human government of some kind, and it will probably seem successful. But it will eventually fail; the problem of guiding humans will always break down.
This is how our Creator does things. It won’t matter whether you see it as God’s active intervention or a matter of inherent design; both approaches will miss the point. The point is that you cannot in mortal form ever grasp the answer. The more you wonder about it, the harder you seek to understand, the more you will run into the effects of something none of us can overcome.
Nor would I say we have nothing to worry about, in terms of human suffering. Mankind will always overrun the limits and make this life worse than it has to be. Still, this existence is supposed to suck; it’s a deception in the first place. God didn’t make it that way. We chose it and we keep choosing it. It’s when you make the final decision to turn away from this life and turn to Eternity that you understand how to live while we are here.
If you can absorb this to the point it’s fundamental to how you approach everything, it opens the door to divine peace. It is human nature to pursue everything but that one thing.
