There's the clincher, at the moment. The debate over security is centred around whether one has physical access to the passport or not. While the Home Office may be correct when it says that:
the information sucked out of the chip is only the same as that which appears on the page, readable with the human eye. And to obtain the key in the first place, you would need to have access to the passport
However, naivety is the bane of security applications - often, one small attack that seems non-consequential can be combined with various other "small" attacks to create something that is just as "consequential" as a "big" attack. Jigsaw pieces.
The question, therefore, becomes a matter of attitude. In other words, how does this naivety translate into ongoing, day-to-day authentication processes? To establish heavy cryptographic (effectively DRM) techniques is one thing. To assume that they can't be broken and to carry on as if they never will be is another. The article mentions the information available to, say, a postman - who knows a passport is for you, knows the name and address, and can get hold of birth dates relatively easily. (The profiling longed for by government ripples out into the commercial sector too, of course...) Brute-forcing the passport number may or may not be difficult.
There seem to be a fair few people working on the security (from both sides) of this machine, anyway. It'll be interesting to see how it goes, and whether or not "zero-day" exploits emerge from underneath our attitude.