However, it can be immune to a passive attack, i.e. simply listening to communications go back and forth. To break it one would need to reverse-engineer the Yahoo program.
Not necessarily. Just a thought - the libcrypto library being used is free software, so why not just restrict our tracing to what goes into and what comes out of MD5_* functions. That should give us a fair idea of how the hash is being generated. I've done a preliminary analysis, and judging by the number of calls to MD5_Update, I'm guessing that this is almost identical to how pam generates passwords. What's the probability that yahoo would use the same tried and tested algorithm rather than develop their own?
Hmm.. if they're using libcrypto then I suppose it should be possible to trace it and break the protocol... basically catch the stuff before it gets hashed! Could you send me details of your analysis?
Since Yahoo's been deliberately trying to kill clones, I'd assumed they'd used some kinda statically linked crypto functions... something fairly trivial to do.