So do we need to increment the build number when the source code changes?
No, you need to increment it when you build the application. Build numbers increment per build. That's why it's called a build number.
Generally speaking, builds refer to compilations done by a dedicated build agent, and it does not include any local compilation a developer may do on their machine while trying to debug the application.
Additionally, build agents are generally only directed to start a build if any changes were made, or when manually told to build. It makes no sense to rebuild something that hasn't changed.
Even if you only build daily, I expect build agents to be configured so that they only start the daily build if anything has been committed since the last daily build.
That being said, in the olden days where application installs were still done by having a developer compile and copy the output to a production server, it was customary for each build to increment said build number.
"Different build numbers might be used when the processor, platform, or compiler changes."
... so build number shouldn't be incremented when the source code stays the same
You're saying the exact opposite of the information you just referenced.
Then of course the build number is a relatively large number as we recompile our source code a lot of times.
I think you are vastly overestimating how many builds you're going to be doing, or undervaluing how much builds can be housed in a number with relatively few digits.
Even if you build for every compilation, and compile the application every single minute, 24/7, it would take just under 70 days for it to become a six digit number. To reach a seven digit value, it takes about two years total.
Accounting for 8 hour work days, that's 210 days to reach six digits, and almost 6 years to reach 7 digits.
If you reduce it to hourly builds, and you assume that every hour (remember, 24/7!) you make a change, it takes 11.4 years for the build number to reach a six digit figure. It would take 114 years to reach seven digits.
On top of that, whenever the major/minor version is changed, the build number should reset. Depending on how often you release minor updates, this naturally caps how large a build number can get.
The amount of time it takes to add an extra digit to a number increases exponentially (10amount_of_digits, to be precise).
I can keep doing the math on how long it takes for big numbers to get big, but I think I've made my point.