It's not so much an issue of whether the neighbor is doing the work for you --- it verges on squatter's rights. If you allow that, there's many people who won't hestiate to plant things there, or park their car there, or.... The issue is that it's your property, not theirs.
My uncle had a similar problem w/ neighbors mowing into his yard. One day while he was mowing and the neighbors were out in their yard, he veered off into their perfectly-cut, straight-line lawn, did a squiggly line across, did a loop and steered back and finished his own yard. What was so funny is he did it very nonchalantly. 'Course, he's a Vietnam vet on full mental disability, so not caring and being a little lot crazy helps insofar as potential repercutions.
You should've seen the look on the neighbors' faces.
My neighbor is a lipstick-and-makeup state trooper with a great posterior. She can do whatever she wants.