Luckily, Martha Stewart has a trick for taking care of the grinder that doesn't involve any hands-on cleaning. All you have to do is add bread to your grinder and let it run.
Second, the starch in bread has the effect of wicking up any oils and odors left over from either coffee or spices as the bread absorbs everything that's in there.