Should you add the source? YES
Should X add the source? DEPENDS
Here comes the why...
Back in the day, compilation time was an issue even smaller projects had. Compiling your sources and never worrying about caching compiler results was definitely appealing to some. That's one point for libraries irrelevant to you.
Another important one is versioning. Do you really need to version each library separately? Run tests against each one? Distribute it amongst many team members? Libraries are great if you do, and convenient to move around, but again, seems you don't care about this either.
Final point here is, it's an added overhead, and dropping the source files is easier in your case, which gives a very strong point to dropping in the sources rather than using libraries. As you have noticed, once you make a single compiler setting change, you have to chase all the dependencies otherwise.
I know all this from experience:
For Swift projects, I definitely use frameworks (libraries) and link against them, since it's easy to configure using Xcode. I also really need the versioning, tests, and decoupling there, so that's why.
For Mono (C#) projects, for Unity, I started with the hip approach of breaking down the project into libraries, compiled and tested each one, which was great... but once I dropped the libraries into Unity, all sorts of issues happened, from the hacked version of Mono Unity uses, to simply the sometimes different behavior the code exhibits when changing platforms. Not having a single IDE here to manage all the libraries was a true pain, so putting all source within Unity was a huge win for productivity.
Finally, most relevant to you, a C++ game project I worked on. A game engine, network realtime client, network HTTP client, AI, and a persistence store were written for this game, just on the client side. What did I opt for? CLion + Libraries. Even though I was using libraries, it didn't feel like I was. All the sources were in the CLion IDE project, and by composing CMakeLists, I was able to trigger all builds and link them in a single stroke.
As a conclusion, I would say using libraries is a future-proof solution, but also a premature optimization if not needed. As far as I could ascertain from your situation, switching from MSVC to Xcode will be a pain if you are gonna have multiple build target. So, just drop it in and maintain as much isolation as possible for when the time when you might need to use libraries.
P.S: I am having a similar dilemma these days with docker. Should I compose? Should I just run locally? .. etc. Also Elixir, as it allows you to build Applications within the same application.. Should I do that? Or separate the app into the so-called micro-services? ... etc. There is no silver bullet, always measure yourself, as YMMV.