Why did the peddler decline the invitation?

As ironmaster mistook peddler as an old regimental comrade and invited him home. The peddler declined the invitation because firstly he was afraid that he had not confessed that he is not a regimental comrade and secondly he was carrying money he had stolen from the crofter. He knew that if Ironmaster will come to know about his identity, he would be arrested. Therefore for him going to the manor house is like walking into lion’s den.