Fragmentation is a problem with hard drives where the head needs to seek back and forth, seek times are near instantaneous with an SD card, so even extreme fragmentation won't affect transfer times.
Its much worse with old slow-seeking media like floppies and zip disks.