The idea of VERSION CONTROL (misnomer: source control) is to allow you to roll back through history, recover the effect of changes, see changes and why made. This is a range of requirements, some of which need binary thingies, some of which don't.
Example: For embedded firmware work, you will normally have a complete toolchain: either a proprietary compiler that cost a lot of money, or some version of gcc. In order to get the shipping executable you need the toolchain as well as the source.
Checking toolchains into version control is a pain, the diff utilities are horrible (if at all), but there is no alternative. If you want the toolchain preserved for the guy who comes to look at your code in 5 years time to figure out what it does, then you have no choice: you MUST have the toolchain under version control as well.
I have found over the the years that the simplest method to do this is to make a ZIP or ISO image of the installation CD and check this in. The checkin comment needs to be the specific makers version number of the toolchain. If gcc or similar, then bundle up everything you are using into a big ZIP and do the same.
The most extreme case I've done is Windows XP Embedded where the "toolchain" is a running Windows XP VM, which included (back then) SQL Server and a stack of configuration files along with hundreds and hundreds of patch files. Installing the whole lot and getting it up to date used to take about 2-3 days. Preserving that for posterity meant checking the ENTIRE VM into version control. Seeing as the virtual disk was made up of about 6 x 2GB images, it actually went in quite well. Sounds over the top, but it made life very easy for the person who came after me and had to use it - 5 years later.
Summary: Version control is a tool. Use it to be effective, don't get hung up about things like the meaning of words, and don't call it "source control" because its bigger than that.