My Xbox One X is wired and even I will see slow speeds like you mentioned on game updates. I will have one update go full speed 300mbs and then another crawl below 1mbs. I assume there are different servers that get accessed for different games? It has nothing to do with my connection. I doubt it is your configuration either.
Do you have any games running in the background? I've noticed if I don't 'quit' a game downloads happen at about 1/3 normal speed. When I quit all running apps (game, store, etc.) DL speeds pick back up.
Typically the first thing I do when I turn on my console is to check for game/app updates. Any time there is an app update, the amount of bandwidth used to download those updates is almost exclusively measured in Kbps. This is the case on my Xbox One X (wireless) and my Xbox One S (wired).