High Lotrmith,
A couple of observations on that statement from Gandalf:
First, it is 'contradicted' by Elrond during the Council. When Galdor asks, "What power still remains lies with us, here in Imladris, or with Cirdan at the Havens, or in Lorien. But have they the strength, have we here...