All this talk of &#x27;intelligence&#x27; gets confusing - it seems like the core distinction people are talking about most of the time with &#x27;general&#x27; or &#x27;strong&#x27; AI is really something more like Artificial Consciousness.<p>This has issues too - if consciousness is an emergent property of a neural net with the right feedback mechanisms and training material then even if you have the right feedback mechanisms you could still create an artificial consciousness that&#x27;s stupid.<p>You see this problems in humans - depending on a lifetime of &#x27;test data&#x27; exposure (parents, peers, environment) and the underlying brain neural net &#x27;hardware&#x27; you can get people that believe a lot of stupid things.<p>Maybe consciousness doesn&#x27;t have to work that way and we&#x27;re just dealing with a local maxima of evolution (or some reproduction&#x2F;sex drive constraint), but we might end up stumbling on the ability to create a neural net with the ability to emerge a consciousness before we can craft the type of consciousness we&#x27;d want.<p>Actually understanding how the system works is harder.