The Raiders already have a huge fan base in Southern California, and for much of that population, a drive to Vegas will be little more than their morning commute. The worst that should be likely, vis a vis fan participation, is that the ratio of home vs. visitor fans at games will be the lowest in the league, thereby giving the Raiders the worst home field advantage in the league.
As far as the town, itself, being a huge distraction, are you referring to gambling? I would imagine that the Raiders coaching staff would impose strict prohibitions on such activity, and with the exception of the real gambling junkies, it really isn't such a huge draw for those living there-- it gets old fast. As for the nightlife, it is true that there are numerous clubs and such, and lots to do after hours. But, the same can be said for Miami, New York, and Chicago.