Strangely enough, latest advice from security experts (circa early 2006 I believe) are advising that writing down very-secure passwords is actually better then having non-written weaker passwords.
Anyway, yeah, I agreed with that part of your post - which is why I ignored it and only focused on when you were talking about the physical aspects of security.
My point is that while they may be hackable, the best way to insure it doesn't happen is to make it take so long that they don't have enough time to do it. Poll workers can't watch it all the time, but they sure as hell can realize something is up when a voter is taking 30 minutes in the polls.
Security is a two-sided coin. You need both the technological aspects and the administrative aspects.