Growlers are one of the best ways to get fresh local beer from the brewery to your house, but they have to be taken care of correctly. If you don't treat your growler right, it won't treat you right.
After you finish drinking the delicious beer from your growler, you should wash it out with warm water and NO SOAP. Soap will cause issues with your beer when you refill the growler and it can possibly give you a serious case of intestinal distress. You should use the same warm water and wipe the cap out with a paper towel or clean cloth, then rinse with hot water again. Turn the growler upside down and allow it to dry thoroughly Do not put the cap back on it until it is totally dry (or just don't put it back on). If you put it back on and the growler has not fully dried you can start to get mildew or mold.
If you happen to take your time before washing your growler, you may get something growing in there. This is bad, there will be a unique aroma coming out of the growler, and it won't be good. If it happens to you, clean the growler out with a little bit of bleach mixed with warm water(or sanitizing acid used in brewing if you homebrew). Cap the growler and shake it up to try and loosen anything that may be clinging. Then rinse it out again. If you still can't get it out, a bottle brush should reach into the bottom of your growler allowing you to get a better scrub.
As long as you rinse it out well and let it dry you should not run into these problems.