You can do a LOT in game development with only the most shallow knowledge and understanding of Computer Science. You don't need CS knowledge to do graphic design, nor do you need it for most 3D design. You don't need CS to tell a story, which is essentially what most quest games do. You don't need CS to think about game-play and usability, and design the ways a user interacts with a game, and so on...
Also, today you have so many tools available (such as the Unity, etc) that allow you to create really cool things without going too deep into what happens in the background.
Also, most of the problems you face in day-to-day software development, even when they have some CS elements, are very trivial, and almost always have been solved before in a well known and documented way. You can always come here and ask :-)
The way we, humans, understand and describe the world around us is with mathematics. The way computers work is solely founded on mathematics (and some fancy sand i.e. Silicon). When I say "the world around us, i don't only mean physics, and the "physical" world. Chess, checkers, soduku, most card games, and many more are just fancy math problems. To understand, manipulate, and be able to create, things like a Chess AI, a visual 2D\3D engine, or anything beyond a MOD to a First Person Shooter, you must have a deep and profound understanding of the mathematical elements behind them.
So basically my answer is - Your mileage may vary. It depends on what do you want to do, where, and how much understanding do you want of what you are doing. Even if you want to do things that really require the Math, today you can learn most of the things you need yourself on the internet.
But, as someone who worked for about 7 years in the industry with only 6 months of professional training, I can say that the easier way is going to collage and learning it there...