“……………starting down/All his young days into the harbor where/his ball went…’ Do you think the boy has had the ball for a long time? Is it linked to the memories of days when he played with it?

Yes, the boy has had the ball for a long time i.e. from the very beginning of his childhood days. Since it has been a part of his life, he admires playing with that ball. He has numerous memories of the days when he played with it.