Who would be made to move?

The non-residents. But it should be whoever agrees on moving. YOu could always just gently tell them that you only want a better path of getting to your driveway, and you could discuss the solutions.