Absolutely, it is a calories in/calories out equation. However if the calories in are healthy muscle building calories like protein and healthy carbs, then you will see results provided you're adding in exercise.
I'll tell you what I did. When I was 220lbs and 25% BF, I was eating garbage and barely exercising. I was repping 135x5 on the bench the few times I went to the gym, this was at the age of 33. I would complain that I never had time for the gym or to prepare meals, I was full of $%^#. I started waking up at 430am, eat some oatmeal with nothing on it and drink a protein shake, then hit the gym. I rotated chicken, beef, fish and turkey with whole wheat pasta or brown rice throughout the day, getting about 3500 cals with a 40/40/20 split of protein/carbs/fat. It sucked at first and it took some getting used to but after a few weeks I lost my appetite for the junk. I'm 35 now, I weigh 191lbs, I'm 10% BF and I rep 225x15 on the bench. I didn't do any magic pills or steroids, I didn't buy into TV products and I stayed focused.
Having a day or two of "pigging out" is actually good for you're metabolism. No rationalization needed.