I'm starting a pet project and would like some advice. I need to perform the following:
- Scrape JSON Data from a REST API
- Save the Data into my Postgresql table
- Paginate through the rest API and continusously bulk insert rows into my Postgresql table.
I have some specific questions regarding 2 and 3.
Question regarding #2
How do I bulk insert? When I make an API call I get 500 dictionaries, each is a row I'd like to insert. I've set up my Postgresql table and the code that handles persisting to the database. The only issue is I'm using
... INSERT INTO ... VALUES ... ...
Which is totally inefficient when I'm receiving 500 dictionaries in each API call.
Question regarding #3:
I'll be collecting trade data that is over 4 years old. Eventually I'll get to present day's data. How can I detect this, and not continuously call the API, receiving the same data as the previous call and inserting duplicate data into my table?
P.S. If anyone has any design tips, that would be great. I have a FeedHandler class that does the inserting, a DataCaller class that does the querying. I wanted to know if anyone has any tips on building such a system.