API writing is indeed one of the most difficult tasks in computing. Obviously it's possible to simply expose everything you have and let people use whatever they like. That prevents your second problem: a task the the code clearly could do, but that you can't do via the API.
Unfortunately, that is the only advantage of exposing everything. The drawback is that the API becomes more difficult to understand the bigger it is, and the less clearly it is organized. An API that is confusing to use isn't used by many people (unless it has some "unique selling point"), so code authors try to make their libraries easy to understand. But smallness and clarity are almost impossible to achieve with an "everything is public" approach, so you have to choose what to expose, whether to allow One Right Way to do anything or Orthogonal Building Blocks that are easy in themselves but can be combined without restriction.
This is a very difficult call to make. Entire programming languages follow different principles on this point (Perl officially claims that There's More Than One Way To Do It, while Python prefers that there be One Obvious Way to do it), so it isn't surprising that individual coders disagree as well.
As for what to allow at all in the first place... may of the most successful programming projects start out as something that the author does for their own immediate requirement. When something works well, looks coherent and seems something that others might want, then people start wondering whether it might make sense to publish it as a library. At this point very different things can happen. Some coders just take the set of things they use the code for and publish that. They may respond to community feedback by adding more API calls, or they may completely ignore them. Others interview possible users in advance and try to find a set of methods that will allow them to do what they want, and is still reasonably well-designed. (Still others interview possible users and include absolutely everything they suggest...)
(It should also be noted that API design is almost never taught to beginning programmers in the way that structured programming, OO principles, or even project management and revision control are now routinely taught. Apparently the task is considered either to difficult or too rare to be taught to programmers in general.)