Because if we build it correctly, it won&#x27;t be a person at all, and won&#x27;t have a value function orthogonal to our own.  Building something that can self-improve and solve problems doesn&#x27;t mean we need to build something that has independent will, motivation, or values.  And it&#x27;s a lot safer <i>not</i> to.