While I can't answer your question directly, I can tell you that I do the same thing with my MF towels except I toss them in a garbage can rather than a storage drawer.
It's a clean garbage can used only for dirty MF towels and the only thing I do is if the towel is wet, I will hang it over the side of the can till it dries or I wash a load. A week is not uncommon for them to remain unwashed.
Been doing it for years and haven't noticed any problem.