>>Interesting short post on AI worth a read:
>>
>>http://market-ticker.org/akcs-www?post=230260
>>
>>
>
>Interesting, yes. It reminded me of another post on AI that I read a few months ago. It isn't short, so the author has broken it into two posts (Part 1 and 2). This guy, Tim Urban, writes some very interesting stuff!! I recommend reading his blog.
>
>http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html
>http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html

Interesting links, you & Jos - thanks.

I think they both rely too much on exponential growth. There's no doubt it could theoretically happen and achieve a greater or lesser "singularity" but as I see it, that misses the point about what always happens when that growth runs out of resources. Jos' article touches very briefly on this, "...and given intelligence the device will seek to expand the boundary as well since that will be immediately known to it as the limiting factor on its development!" True enough, but how will those boundaries be expanded? Even if it could suborn all of them, there are only so many transoceanic optical or satellite links, carrier-class Cisco routers, so much extra electricity available on creaky grids, and Predator/Reaper drones capable of killing any humans trying to pull the plug on its data centre(s). An ASI would have a tremendous uphill battle against entropy. As I see it, an ASI (or probably even an AGI) will only be able to exist in a well-resourced, clean environment e.g. corporate/university or military/intelligence. As such it would likely be limited in scope, and fragile rather than robust.

OTOH, given that many human, biological and other systems run in a critical state, even limited meddling/intervention by an AI could cause tremendous damage. But, doing so would tend to reduce its own chances for survival.

As for nanotech - I haven't yet seen, anywhere, any proposal or research on how to power it at scale. Some people seem to think it's magic - "Hey, Utah's running low on fresh water. I know - let's throw some nanotech in the Great Salt Lake and turn it fresh!!" Even with (unachievable) perfect efficiency, it takes energy to remove salt from salt water. Where does that come from? Entropy's a biatch.

Some science fiction stories try to address these resource problems by collecting all mass in the solar system and using it to assemble a Dyson sphere around the Sun. But none go into any details on how that could be achieved or (ideally) bootstrapped.

The posts also make much of how different we are from our ancestors of 100,000, a few thousand, or even a few hundred years ago. Then they extrapolate to say that ASI could outpace humans far more than we outpace any of our primate forebears, or probably even amoebae for that matter. But it's not obvious to me that it must, or even can, be so.

"Any sufficiently advanced technology is indistinguishable from magic" -- Clarke's 3rd Law

"Any sufficiently advanced civilization no longer believes in magic" -- I wish I remember where I saw this, maybe Paul Preuss (a sometime collaborator of Clarke)

The rejoinder is hugely important. As a civilization we're already pretty "meta". We can ask big questions and search for the answers. An ASI might be able to do that faster, but could it achieve anything better?

http://1.bp.blogspot.com/_iRE1f7T37Zk/TU-3EXGzKcI/AAAAAAAAAWM/Cgacel09r0M/s1600/7.+Hate+being+a+girl.gif

I like to think we're not bugs ;)