This isn't just a matter of aesthetics, as plastic jars can degrade over time and leach into what's in the jar, giving your sourdough starter a funky taste.
Glass jars are often wide-mouthed, making removing the starter a lot easier than with narrow, plastic containers. They are also easier to clean than plastic jars.
However, glass jars are more fragile and less forgiving when it comes to breakage. If it takes a tumble, glass can get into your starter, and you'll need to start from scratch.
If you opt to store your starter in plastic, use containers like the ones the deli serves soup in. They have wide mouths and are inexpensive, so you can replace them periodically.
Regardless of your preference, avoid using metal for your starter. According to baker Carroll Pellegrinelli, the fermentation process can corrode metal.