Yes it’s very stupid, and that’s not the point.
Chess computers are stupid too. They don’t realise they’re playing chess. In fact they don’t realise anything, they are just machines. All they are doing is using massive computing power in order to predict the best next move. In a sense it’s not really “playing chess” at all, it’s just mimicking somebody making chess.
But that is the point. Even though it doesn’t know what it’s doing, just by calculating the next move, and replicating what a chess player would do, it can now win at chess against any human, every time.
The same goes for the AI that pretended to be blind to get the information to enter the captcha. It’s fair enough comment to point out that it didn’t really know what it was doing, it just mimicked what it calculated to be a successful move in that situation to reach its goal. But that really is a red herring.
In terms of outcome, it doesn’t matter if the AI knows what it’s doing, or not. Whether the computer is playing chess, or it is mimicking someone playing chess is really immaterial in terms of the outcome of the game. The same with the AI that ‘pretended to be blind’. Whether it had any ‘intention’, or not (I don’t think AI has intentions, or thoughts, of any kind, or ever will) isn’t the point. The point is that it found a way to reach its goal through calculating the next move. When we reach the point that AI is better at calculating the next move than us in any given situation is when it becomes dangerous, because it will be able to pursue a goal whether we want it to or not. It will ‘know’ (in the sense that it will be able to calculate) the best next move to achieve its goal whether humans are in the way or not. If we do get in the way of it achieving its goal we are in danger, not because it has any malice for us, or any feelings at all for that matter, but just because it is better at manipulating the world to achieve its goal and we are in danger of being crushed by it.
Eliezer Yudkowsky explains the consequences and likelihood of this scenario, much better than I can, in this interview.