One important difference between SVN and Git is that in SVN, all files reside on a central server and only some revisions are checked out. In contrast, Git downloads the complete repository. While this allows Git to function as a decentralized version control system, it also means that it is preferable to keep the repository small.
Because size matters, checking in (large) binaries into Git is not a great idea. They will add up over multiple versions to significant storage. Whether or not this is going to matter depends of course on the total number of binaries and their size – run the numbers first to see whether this matters.
It is therefore correct to store the binaries externally. E.g. Git-LFS (large file storage) is a widely supported git extension to do exactly this. However, the server might need additional configuration. In particular, hosting services like GitHub charge extra for LFS storage. Git-Annex is another implementation of this idea.
Your idea to use a separate repository doesn't quite solve your problem but just moves it to a different repository. Sometimes that is good enough – I've done it as well. As you are here using Git as a storage mechanism and not for source control management, it is completely legitimate to ignore normal best practices. Lots of orphan branches? No problem.
But because you are not making use of Git, there's no strong reason to use a Git repository to out-source your blobs at all. If you don't want to use LFS, another possibility would be to upload the binaries to some server (but never modify these files!) and check in the names/URLs into your repository. Your build process would then download the binaries, possibly caching them locally. This is quite similar to how LFS works, except that LFS is part of Git and not part of your build process, and that LFS uses content-addressable storage: instead of specifying names/URls, the hash of the object is used.