this being the case, can anyone explain why not?
The optimizer cannot compensate because the optimizer works at compile time. It cannot know how your collections will be used, so it cannot optimize the collections to the operations that are most common. Worse yet, the common operations may not be the important ones. Say you do 1000 inserts, but all those exist to let you do one quick lookup later. The inserts might not matter because they're done in the background while your app loads. There's no way for the optimizer to know.
would this be a massive performance hit
As with all things, it depends. Let me try to provide a quick, high level explanation why.
The reason that there cannot be one universal data structure is because of tradeoffs. If there existed a structure that was quick to insert, quick to search, quick to delete, quick to sort, memory efficient, thread safe, with no resizing penalties... well we'd certainly use it. The problem comes that if you optimize a data structure for say, quick insertions - something else will suffer. This table (thanks to this question) shows what all those tradeoffs.
And those tradeoffs are pretty significant. Having code do something common in
O(n) (or worse) rather than
O(1) is what takes the runtime of your app from milliseconds to seconds and from seconds to minutes or hours.
So yes, it is necessary to have all of these collections.
That said, it often doesn't matter if you use them. In C# for example, there's
List<T> which is "good enough" for any basic collection and
Dictionary<K,V> which is "good enough" when you need a lookup. Few apps need to be highly performant, and in those apps a small fraction of the app accounts for the majority of the runtime. Using the default collections until your profiler tells you differently is a fine approach for beginners, as long as you know that the other collections exist, and why they exist.