Just as many aspects of human behavior are impossible to explain in detail, perhaps it won’t be possible for AI to explain everything it does. “Even if somebody can give you a reasonable-sounding explanation, it probably is incomplete, and the same could very well be true for AI. It might just be part of the nature of intelligence that only part of it is exposed to rational explanation. Some of it is just instinctual, or subconscious, or inscrutable.”
https://goo.gl/pwMh9T
@gideonro it's not just an "AI problem" but a technology problem, IMO.
We often aren't able to explain why various complex programs (high speed trading, control of nuclear reactors, real time statistics based stuff) do what they are doing. Sometimes with hindsight, after something went wrong (Chernobyl , again).
