I can see that now. Our "common sense" output is only due to experience with a number of factors, which an AI may not possess or even be able to simulate in a way humans experience it.

The difference between newborn humans and AIs, therefore, is a big portion of information that we process without thinking or even being able to notice. We subconsciously understand a lot of stuff from trial and error, and we utilize it without applying conscious effort. It's quite difficult to program what we don't understand working; in fact, it may be up to chance for us (or the self-improving AI) to do so right now, because neuroscience is nowhere near that far in explaining our thinking.

Do you think it's possible to create an AI that's whole purpose would be to crunch through data (with the reward being more data connections made)? As in, is it possible to create one that won't loop onto sheer data-crunching or stuck with one level of data being crunched (if that's even possible)? This sounds like it would boost our progress significantly, if only because of how much data an artificial (and bigger/more responsive) neural network might go through in a unit of time.

One example of an algorithm acting weird is in those "teaching a bot to play NES games" video.

I read about it some years ago. Good to finally meet the maker of this demi-mythical beauty - the game that decides not to play because it's the best it can get to winning.