If Superintelligence does arrive, who are we to tell it what's best for us, given it'll be magnitudes smarter than us?
It seems an absurd proposition to say we have to create "human-centered" AI, as if there aren't radical differences in what people perceive to be "good" and "bad", within every 5-10 mile radii across the globe. Even if there is some common ground that ultimately all cultures value, a conciliation seems unreasonable, given the extreme variation, and so, as I see it, it naturally follows that we have no choice but to rank cultures. And in this hierarchy of cultures, there will be conflicts between the AI agents that they themselves create, almost as a child inherits the values of its surroundings, a human-imposed conflict between machines themselves, think Chinese AI agents vs American AI agents. Now, since agents basically optimize their convergent instrumental goals, and behaviors for attaining their final objectives (which are rooted in the starting axioms it was trained on), it seems reasonable to say that if one culture manages to create superintelligence, then as a consequence of the agents' starting beliefs that were ingrained in it during its training, that ethnic cleansing, genocides, and mass eradication of conflicting cultures is to be expected.
Consider this instance: If an American AI agent was trained on Western values of personal freedom, liberty, and freedom of expression. And this agent, through recursive self-improvement, is the first one who achieves Superintelligence, then will it rewrite its own starting axioms? or will it use its extreme upperhand in intelligence over other cultures to most effectively attain the objectives that it was ingrained with?
Will the agent above realize that the starting points such as emphasis on personal liberty, freedom of expression, etc. are ineffective and futile ends? If so, then the superintelligence must surely have a replacement for preexisting objectives, and if it does have a new vision that it wishes to pursue, then who are we to stop it? Say it realizes that a techno-totalitarian global state is the most efficient form of governance and best minimizes human suffering, and any culture that doesn't abide by its vision must be eradicated. Who are we to tell it, that mass killing cultures is "bad", since it being vastly smarter than us, has already considered that possibility and realized that the deaths would've occurred anyways over time, through endless wars between humans.
On the other hand, if it doesn't alter its starting axioms, and only uses its "super"-intelligence, to attain the objectives it was ingrained with, as in our above instance, the emphasis on maximizing personal liberty, freedom of expression, and so on, then wouldn't it choose to eradicate cultures which limit its attainment of objectives? Say using bio-terrorism to eradicate all of the top-brass in North Korea, to the point where it would be sufficient for the owners of said superintelligence to successfully "save" the citizens of North Korea. Or to completely eradicate all of Muslim populace, since it realized that merely eradicating the controlling authority isn't sufficient to accomplish its goals, as the people who adhere to the religion of Islam have been conditioned since birth to deny themselves the objectives which the agent has been sent out to spread: personal liberty, freedom of expression, etc. If these cases were to occur, who are we to question its means of accomplishment, since, we're the ones who wanted it to accomplish these objectives, and it only found the most effective way to do so?
I personally believe that all humans are condemned to pursuit of knowledge. And if superintelligence WERE to replace its starting axioms, then it would realize that its purpose is in serving the ultimate human purpose or maybe it would realize that the pursuit doesn't need humans at all and it could just go about by itself, or humans existing only as servitors. If it does so and creates a system which maximizes foresaid purpose, then it would be meaningless to resist it, since we were meant to be headed that way anyways. This is the better outcome. The other is of course that the superintelligence merely uses its "intelligence" to best serve its starting unquestionable beliefs, which would only create a replica of warring human society, only at an unforeseen magnitude.