The speculation comes from the lack of consistency. For example, Ralph's "absolutely not", followed by Marv's "maybe." Ralph's "Mike is staying", followed by Mike's "I'm leaving." If all of this had happened in a matter-of-fact way, there would be no second guessing.