This appears to have been added in v4.0 as the v3.3 builds Google was on before didn’t have the feature—that version played via the earpiece only. Now you can just start a voicemail and place the phone to your ear to listen via the earpiece (based on the proximity sensor). Pull it away, and the sound switches to the speaker. You can even go back and forth during a really long message (ugh).
I tested this on a few phones and it seems to work without issue. Some of you have probably noticed feature this before, but just so we’re all on the same page, this is pretty cool. Anything to make listening to voicemail more tolerable.