Now that you mention it I think Ultron was just embarrassed they showed up before he was ready and was too shy to ask them to leave, his cover lie was full of holes.