Nah you can at least get 2 since I have Pears, Oranges and Apples but don't visit other people.
I wasn't clear on the other sources, they include friends or letters from your Mom in game. So if you wait long enough, Mom might send you all of them.
So you have
- Native Fruit
- 1 Fruit from NMT (your native and first NMT fruit will be all you'll ever see on NMT trips)
- Remaining fruit from Mom or friends.