The housekeeper naturally spread this order to other servants. So they did their job, treating them as invisible even if they saw them inside the mansion.