If you build in difference to reach higher jhana won't that make it hard to enter jhana? If i have to review how gross the jhana factors are won't that make it hard to enter the jhana as I don't care for the factors?
The jhana of the Buddha is born from letting go (SN 48.10). When the peace of letting go is always known, the vibrations of rapture will feel unsatisfactory compared to the peace of letting go (i.e., the Nibbana element). Thus, the mind will not cling to the vibrations of rapture and the rapture will lose its energy as the body (rupa) relaxes even more. 

The rapture of jhana is merely nervous system bliss arising from the dissolution of sankharas or stress previously stored in the physical body. That is why they are called "rupa jhana"; because the bliss is based on the relaxation of the neurology of the physical body that stores and is stressed by unwholesome mentality.