So you have a box of water containing a number of fish. You have filters containing 85-95% of the bacteria necessary to keep the tank in check for the given bioload (amount of waste produced by said number of fish).
If you move to a larger box of completely new water, but move both those fish previously existing and the 85%-95% of the bacteria that support them, nothing has really changed. You should be concerned about water temperature perhaps, in some cases maybe pH, keeping it within 1 of the previous pH is reasonable. Otherwise you're good to go.
I've done this a number of times when upgrading fish to larger tanks. I've also done it when switching to smaller ones (broke down a 100 gallon of haps into 2 29 gallon tanks for a period of 3 days and just split the filters and bioload in 1/2 between the two tanks.
Yes, the substrate contains some potential for bacteria. So do the glass sides of the tank. But ideally you have some form of biomedia that has a large surface area to volume ratio in the filter. Having it in the filter insures good water flow, something that can't be said for the entirety of the substrate.
No worries doing what you suggest IMO.
Times when I would be concerned? Doing what you suggest, plus adding a bunch of new fish to the tank and no additional filtration containing bacteria sufficient for the bioload of the new additions. In such as instance you would encounter a spike. The significance of which would depend on the quantity of additional fish added (AKA in some cases it wouldn't be a big deal).
