Nehaveigur

Let’s hope AI doesn’t turn the universe into paperclips and remains aligned with its masters.

Even so, this could lead to suboptimal outcomes. Imagine AIs running corporations, replacing its employees, either from the bottom up or from the CEO down. If those AIs are well aligned with the company’s missison, they will be profit maximizers.

Corporations already are supposed to be profit maximizers, and this can lead to problems when profit maximization is incompatible with acting morally or even legally. For a good discussion of this problem, and why it’s not easily solvable, I recommend the recent conversation between Jim Rutt and guest Kristian Rönn as they talk about Rönn’s book The Darwinian Trap.

CEOs may be incentivized to maximize profits, but because they have to act through human managers and employees, there are limits to what they can achieve. There are immoral or illegal actions that they can’t pull off, or at least not pull off efficiently, because employees are either openly going to rebel, or more often, intentionally slow-walk actions they don’t agree with. This last barrier to profit maximization would go away with profit maximizing AIs taking over.