It is not the material desires, it is the offences against the holy name that is the problem .....because the process of chanting Hare Krishna will easily remove the material desires but if we commit offences against the holy name then the effect of the process is delayed and thus will take longer for the material desires to be removed. And if the offences are of a serious nature then even new anarthas will be established in the heart and thus new material desires. So the essence is to avoid the offences, avoid the ten offences of the Holy Name.
(Kadamba Kanana Swami 03/2009)