That Sadducee who was in the neighborhood of Rabbi Joshua ben Levi etc. - it is not the way of the world. Even though the Sadducees etc. "we lower them down but do not raise them up." This refers to being done by a person, but by the hand of Heaven it is not the way of the world to punish them, trouble them, and kill them by the hand of Heaven in an unnatural way unlike how human beings act. And if we read here "idol worshipper" it makes sense, as it says there (in Avodah Zarah) regarding an idol worshipper and shepherds of small animals that we do not raise them up nor lower them down.