The number of seal pups being born on an important breeding ground on the Norfolk coast has fallen as the animals search to give birth elsewhere.