Hacking Python “Requests” library.. How to start as an intermediate pythonista?

Question

My question is conceptual, yet extremely vital for me.

I'm an intermediate Python developer. I know fair enough about it, and actually use it on a daily basis. Now, I would like to transform my capabilities from a library/API user to "designer" one.

I found that "Python Requests" library by Kenneth Reitz is an extremely good place to explore Python patterns and idioms. I need to learn to split a big problem into its essential Modules, define required classes/functions, and use them to work together.

I lack such a discipline, and I'm starving to gain this knowledge.

How should I dive into this library? It is big, with many modules (with logical names, of course), classes, and methods. I started to read the code, but I don't know where I am and what this portion of code is designed for.

Answer 1

Get focused. Think of a question, then try to answer it. For example:

• how does the library encode a dictionary into a form suitable to submit to a web server?
• how does the library store authentication information?
• how does the library convert a dictionary of headers into a web request?
• how does the library implement logging?

... And so on.

Answer 2

You start by following common tasks; find out how a regular GET request is handled, for example, or how the .json() method works once the data has arrived.

It does help to be familiar with the subject matter modelled by a body of code. Knowing how HTTP works from a client perspective helps; if you don't know what a 302 redirect is or how HTTP uses headers to exchange info, you will have a harder time understanding all that goes on.

Luckily the requests library does keep it's concerns nice and separated; you don't need to dive all the way into the urllib3 library innards for example.

Answer 3

As @Martijn Pieters has pointed out, you need to know what problems that library solves, which in practical terms means, that you have to understand the problem domain.

then you will be able to have an idea on the choices the author made, appreciate some, criticize others.

if the library is too big to get an immediate overview, I would advice to start in some place where you have an (at least approximate) idea what the code is doing, like e.g. requests/api.py and

1. understand the objects you find there, and
2. understand how the code you are reading modifies those objects.
3. see if (and which) objects are created and
4. what those objects are used for.

if you do this from several starting points, you will get an overview of the whole. it is somewhat like exploring a unknown territory.