Is there anything I am missing here?
Yes. You are missing the fact that you are human and lack perfect foresight. One day you will find that you did many things that were just plain wrong or that prevent you from making forward progress.
When that happens (and it does happen to almost every software project of sufficient complexity), you will have to break backwards compatibility. You need some mechanism to tell your users that (for example) release #13 provides a number of new features, fixes a huge number of formerly unfixable bugs, but does so at the cost of requiring your users to rewrite some of the software that uses your package.
The build number, which monotonically counts from one (or zero) hand pwards doesn't communicate much. You need something that tells the users of your package how much work they need to do to take advantage of your latest release. At a minimum, I'd recommend you use a
major.minor numbering scheme. This may suffice if the project remains small. Note that the build number is absent from this scheme.
If your project has the chance of growing beyond a one person project, you might want to advance to a
major.minor.patch numbering scheme. This is the approach taken in semantic versioning. Once again, the build number is absent from this scheme.
Telling your users that they may have to rewrite significant chunks of their software that uses your package (a bump in the major release number) conveys a lot of information. Telling your users that they need to watch for behavior they relied on that was in fact a bug (a bump in the minor release number) also conveys a lot of information. Telling your users that behavior that everyone has avoided up until now because everyone knew that your software used to choked (a bump in the patch number) also conveys a lot of information.
The build number, on the other hand, doesn't convey much information at all.