This really all depends on how you are defining a CPU. If you are talking about the embedded CPUs found in many of the devices you use, then yes, they often contain both the RAM and the processing units that would make up a common computer. If you are talking about a CPU that you would find in a consumer desktop/laptop you would see that the cache you are referring to is usually very small and only used to buffer instructions to the core, thereby reducing downtime, as calls to the main memory (RAM) are very expensive. They would be very very limited in what you could do if you ONLY had the cache on those cpus, but they could technically still be called whole computers.
This all comes from the fact that in the days of old, the mathematical units were separate from the memory manager, which was separate from the memory, which was separate from the logic units. each of these units by itself was incapable of doing the whole job itself, and thus was not a 'computer.' as we think of it today. Without memory, your logic and mathematical units wouldn't have anyone to get instructions from. Today's computers are very far removed from being a collection of separate entities on a chip, as the more integrated you make them, the more performance you can achieve.