From the Jewish Encylopedia of 1906:
The Kol Nidre has been one of the means used by Jewish apostates and by enemies of the Jews to cast suspicion on the trustworthiness of an oath taken by a Jew. This charge was leveled so much that many non-Jewish legislators considered it necessary to have a special form of oath administered to Jews ("Jew's oath"), and many judges refused to allow them to take a supplementary oath, basing their objections chiefly on this prayer. As early as 1240 Jehiel of Paris was obliged to defend the "Kol Nidre" against these charges. It can not be denied that, according to the usual wording of the formula, someome might think that it offers a means of escape from the obligations and promises which he had assumed and made in regard to others.
From Artscroll's biography of Rabbi Samson Raphael Hirsch (of the same name):
There was one instance where Rabbi Hirsch did omit a traditional prayer. In 1839, Rabbi Hirsch deleted the recitation of Kol Nidrei in Oldenberg....Rabbi Hirsch explained it in writing to the correspondent of the liberal Allgemeine Zeitung des Judentums: "Although last year the Kol Nidrei was deleted for halachic reasons, neverthless I came to the conclusion that this change, although halachically grounded, would better not be instituted by an individual rabbi. Therefore I requested that the congregation recite it, but only once, not three times." In any event, he reinstated Kol Nidrei the following year.An interesting cultural artifact is the centrality of the Kol Nidrei in the alleged first talking picture, Al Jolson's The Jazz Singer from 1927. To me, that seems to indicate a refreshing lack of prejudice against Jews in the United States, this despite the more benign country club variety antisemitism that was quite prevalent. Surely it would have been unthinkable for such a film to be produced overseas at that time.
More importantly, today many Jews are completely unaware that there ever was a controversy. While it says something about their own Jewish education, it also says something about the society in which we now live (David Duke notwithstanding).