As a beginning programmer, should I favor building my own libraries over using 3rd-party libraries?

Question

As a beginning Python programmer, is it a good idea to build and understand my own libraries before jumping to advanced 3rd-party libraries that contains the functionality I need?

Some projects (e.g. web frameworks like Django) are probably too large for this approach. But other projects (e.g. Web Crawlers, graph libraries, HTML parser) seem to be feasible.

I worry that early reliance on 3rd-party libraries would stunt my growth.

Note: this question and this question seem to focus more an experienced programmers, who are probably more focused on the efficiency of reuse than the learning benefit. My question, I think, is focused on beginners.

Answer 1

This is always a trade-off.

As a beginning programmer, you should ask yourself two questions when considering reusing code vs. reinventing the square wheel:

1. Will I learn more about the problem I want to solve by writing everything from scratch, or by focusing on the problem domain and putting aside complexity not critical to the problem that I am interested in?
2. Is it more important to me to solve the problem at hand or is it more important that I understand some fundamental concepts?

If you don't have to finish your project, it's fine to spin your wheels on complex problems that other people have already solved, because you'll learn something. But you'll probably move on to something else before you "finish," which may or may not matter to you. Other projects will start to look shiny fast when you get in over your head on a complex domain that looks simple until you start trying to solve it yourself.

Don't obsess about giving up control because you're deferring to someone else's way of thinking; focus more on what you're trying to accomplish.

If your goal is to write an HTML parser because you want to understand how parsers work, go for it. If your goal is to write an HTML parser because you want to sanitize user input or transform some random bits of HTML, you're probably focused on the wrong thing, because you probably are more interested in the application of parsing rather than the parsing itself. If you feel like writing an HTML parser because you don't want to take time to understand someone else's library, you're probably wasting your time, because, at least in this case, I guarantee someone else has spent more time figuring out how to solve this problem effectively than you'll have. In really trivial cases, you may save time by not reusing code, but in complex ones, unless the library you use sucks or your ability to read documentation and code samples sucks, you'll just waste time.

On the other hand, I would say that it's worth writing your own graph library, since you'll be more focused on transferable, fundamental algorithms and data structures that you'll be able to apply to other domains, even if you end up using someone else's library when you work on those problems.

Answer 2

Most beginning programmers vastly underestimate the complexity of a problem. This leads to many unpleasant situations, but I'll focus on just one: they dismiss a lot of third-party libraries for being "bloated" or "too complicated". Then they try to write their own code to do the same job, and make a complete mess of it, because they didn't anticipate all the stuff that they'd have to handle.

Take web crawling and parsing HTML. You seem to have a pretty casual attitude towards those tasks - which leads me to believe that you never tried to do them on anything but a very small scale. Here's just a short, non-comprehensive list of the potential problems that a real web crawler needs to deal with:

• Malformed HTML.
• Multiple character encodings (the whole world doesn't run on ASCII).
• Links that are generated or modified by JavaScript.
• Links to multi-gigabyte binary files.
• Incorrect mime-types.
• Crashing in the middle of a crawl session.
• Tar pits.
• SEO consultants.
• Microsoft.

Answer 3

I worry that early reliance on 3rd-party libraries would stunt my growth and make me nothing more than a library integrator rather than a programmer.

That's false.

Early reliance on 3rd party libraries will teach how good libraries work.

Premature development is always a waste of time. Writing your own libraries without carefully studying (i.e. using) existing libraries dooms you to reinventing the wheel -- badly -- for the rest of your career.

I've spent many billable hours cleaning up those messes.

Answer 4

This sentence, which is the first one is very logically problematic for me:

As a beginning Python programmer, is it a good idea to build and understand my own libraries before jumping to advanced 3rd-party libraries that contains the functionality I need?

How can you believe you can effectively write advanced (your word) functionality libraries as a beginner that will be as correct, feature complete and tested as existing libraries?

This doesn't even mention the massive amount of time you would waste re-inventing a wheel that you don't understand as a beginner, will eventually abandon when if you are lucky you realized you are over your head, and have to learn the exiting library anyway.

Take HTML parsing for example, you mention this as feasible, there is no way you can do a better job than Beautiful Soup as a beginner. Plain and simple you will fail. The things you think are feasible have huge complexity issues that you are not taking into account, you don't know the problem domain in enough detail to understand the complexity, you are doomed to failure just from that.

My opinion is learn what other people have done and already solved your problem for you, until you can't find something that addresses your problem.

Answer 5

This is just my opinion, but I'd suggest building your own libraries for pretty simple, re-usable code. For anything more complex, you can use 3rd-party libraries, which have been built and tested thoroughly (hopefully!). Creating a robust library can be very time-consuming, depending on what it is. I'm sure you'll get plenty of programming practice just by building your application!

Answer 6

Check out this similar question, the answers are relevant:

https://stackoverflow.com/questions/991487/how-to-assemble-a-project-with-software-products-and-your-own-code