The 16th century Tishbi lexicon of R. Elijah Bahur (Levita) has some really fascinating entries. On Gan Eden (Garden of Eden) he first explains its usage in rabbinic texts, and then addresses the obvious question: can't you visit?
"The Sages called the enjoyable place where the righteous receive their reward "Garden of Eden," after the the garden which the Lord planted in Eden and placed the first man in, since it is the choicest place on earth, without a doubt. Therefore the named this enjoyment the "Garden of Eden." Some ask, if we know that it is in a place called Eden, and we were given various signs, such that it is east of Eden, why is it that no man has seen the place and its garden? The response is that without a doubt many people have gone there! But all who go do not return, because of the great pleasures there. Who is the fool who would leave the Garden of Eden? All remain there. Or, perhaps the reason is the ever-whirling flaming sword (Gen. 3:24) is still whirling and no one can enter."
R. Yeshaya Berlin-Pick's note here (2002 edition) says that it is certainly the ever-whirling sword that prevents people going. He adds that Levita's first answer is only a joke.