vanilla white and chocolate brown

Can anyone tell me when vanilla became associated with the color white? I was thinking about it and really, vanilla beans are brown. So is vanilla extract. I tend to like all the dirt colored flavors (chocolate, cinnamon, coffee, star anise, pepper, ginger, etc...) and I kept thinking vanilla but kept associating it with white and not putting it in that category. I think it's stupid. Vanilla is brown.

random thought.