In R. Moshe Yoshor's Chafetz Chaim : The Life and Works of Rabbi Yisrael Meir Kagan of Radin it is related that a 17-year old yeshiva bachur of German origin was arrested during WWI by the Czarist police, as were many German nationals in Russia, under suspicion of spying for the enemy. The boy was put on trial and faced death. His lawyer wished to enlist the Hafetz Hayyim to testify as a character witness. Before the Hafetz Hayyim took the stand, the lawyer introduced him to the court as a man of great moral stature and he related a story about him. One time, said the lawyer, this rabbi was robbed. As the robber ran away R. Kagan chased after him shouting, "I forgive you, I forgive you" for he didn't wish that even his own robber should face divine retribution on his account. The judge, not sure if he should bite or not, asked the lawyer if it had really happened. The lawyer admitted that he wasn't certain that it had, but "Your honor, they don't say things like that about me or you."
(The boy was convicted and sentenced to death, by the way, although his death sentence was commuted to ten years hard labor due to his age [and, presumably, the fact that the case was entirely concocted]. The Russian Revolution happened a few weeks later and he was freed along with many other Czarist political prisoners.)
An anonymous commenter on Godol Hador's blog writes "it is said that Rav Shach had invited [R. Elya Ber Wachtfogel] to be [Rosh Yeshiva] of Ponovevz. Whether that's true or not, the fact that they tell that story indicates his stature within that community."
The "story" about the Hafetz Hayyim in court is well known and likely the source of the phrase "they don't say that about you and me" and its variations and how its used.