I didn't worry about it for about a year of open source development. By the time I realized that I should be concerned, the address is out in the wild. With that said though, I haven't had too much spam coming in on that account, perhaps a spam message or three gets through gmail's filter per month.
If you're concerned about this sort of thing, perhaps make a separate email address for your opensource contact, and then just forward everything. If you're using gmail, I'd advise using their "plus addressing""plus addressing".