Elrond may have had a fairly large 'manor house' with space for many people of his household, not just family, but it seems unlikely that the actual Last Homely House was the only dwelling place in the valley. He is leading an army in retreat when he founds Rivendell. That's got to be thousands...