And let's not forget why a knight like him is in the middle of nowhere.
He was selling poachers on his lands to slavers and that's illegal in the kingdom. Ned Stark found out he did it and was going to deal with it, though Ser Jorah fled and ended up with the Targaeryns. I think the penalty was execution for slavery.
I think that's how it went down, so please correct me if I have some of it incorrect.
You've got the gist of it.