I believe that the consequences of machine intelligence are in some ways irrelevant, because I see machine intelligence as inevitable. I believe it is human nature to always seek for ways to do things faster, more efficiently, and with less effort. We can attribute this drive to virtually every human accomplishment. It is what brought us out of the caves, made civilization possible, capitalism, the industrial revolution, you name it. This drive is so powerful it means that if machine intelligence is possible, it is unavoidable. We will build it regardless of the consequences because we must increase production, and machine intelligence offers us a way to automate and optimize nearly any task.
I also think it is inevitable. And although I am generally an optimist when it comes to technological development in the future, I can't help but be scared and frustrated sometimes. I am currently in graduate school and I'm afraid that by the time I'm done there will be a robot that could be programmed to know everything I know about my subject (and probably every other subject) and also be able to do independent research more efficiently and better than my own. That leaves me scared of what my future is. I think it is quite analogous to what blue collar workers must have felt to varying degrees throughout the 19th and 20th centuries as machines "took their jobs" (south park voice).
Do you think that the jobs that require the reading of human emotions will be less likely to be overtaken by machines? Jobs like sales? In sales, the most successful practitioners are those that can quickly identify what someone's motivating factors are. I wonder how a machine could do this? That said, I'm sure that someday they will be able to. But, how is it going to make you feel as a consumer knowing that you are being spoken to by a machine that has been programmed to know how to sway your opinion? Right now we have people that are highly trained to do this but yet because they are "people" we seem to be okay with it to an extent. These types of jobs and positions may be the last to be overtaken by machines in my opinion. But perhaps that's just me being a protectionist.