If the photons from a source have some non-zero angular distribution (i.e. different photons take different paths), then the half-silvered mirror isn't doing "different things on different occasions!" ...