The passed modules to amp.initialize will get patched based on the used opt_level. If you are not passing all modules to it, autocasting might not work and/or the code might break in another unexpected way.
Note that apex.amp won’t get any new features anymore, as the current development is focused on the native implementation.