No, you're right, salary really depends mostly upon specialty, type of practice (private practice vs. other), and location in the country. Private practitioners with a heavy practice load can pull in a huge salary, although it's much more (in general) for subspecialists than for GPs. Salaries out east are also generally much better than those here in the west (except perhaps the west coast, although I don't know for sure).
As for me, I picked the lowest-paying specialty, in an academic center, in the west. That's one hell of a combination.