The fate of supernova remnants near quiescent supermassive black holes