Timeline for How can I efficiently diff a CSV file against a database?
Current License: CC BY-SA 4.0
21 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 7, 2022 at 19:37 | comment | added | MonkeyZeus | @Isvara I see. I just figured that total time to diff might be lower if you get the table as a CSV. | |
| Sep 7, 2022 at 16:33 | comment | added | Isvara | @MonkeyZeus Even if I could (it's multiple tables), downloading and parsing a CSV file would just give me the same data as fetching the data directly from the table. | |
| Sep 6, 2022 at 12:44 | comment | added | MonkeyZeus | So can you download the Postgres table as a CSV and diff the CSVs? | |
| Sep 5, 2022 at 3:00 | history | tweeted | twitter.com/StackSoftEng/status/1566622332688506881 | ||
| Sep 3, 2022 at 0:13 | comment | added | Isvara | @MonkeyZeus In the case of a manual upload like this, the CSV file would be the master list, but it's still worth diffing, because lots of rows will be unchanged, and the changes are being sent out to third parties, both smaller companies who are likely to have less resources, and to services like AdWords. | |
| Sep 2, 2022 at 14:30 | comment | added | GammaGames |
You could load the csv with the file_fdw module and run queries against it like any other table: postgresql.org/docs/current/file-fdw.html
|
|
| Sep 1, 2022 at 18:07 | comment | added | duckbrain | Is your CSV file sorted by some unique value, such as the ID? If so, you could query all of the rows, ordered the same way, and stream the result and stream reading the CSV, keeping one of each in memory at a time. | |
| Sep 1, 2022 at 9:59 | answer | added | jpa | timeline score: 2 | |
| Sep 1, 2022 at 9:15 | history | protected | gnat | ||
| Sep 1, 2022 at 8:58 | answer | added | Daljeet | timeline score: 3 | |
| Aug 31, 2022 at 18:29 | comment | added | BurnsBA | can't timestamps filter out rows you don't care about -- i.e., records since last comparison | |
| Aug 31, 2022 at 16:02 | answer | added | Daniel Koszta | timeline score: 9 | |
| Aug 31, 2022 at 15:26 | comment | added | MonkeyZeus | If the database contains something which the CSV does not then should it be deleted? If so then the CSV is your master list; the table can possibly be cleared and repopulated. You can achieve this with some sort of temp table rename scheme. | |
| Aug 31, 2022 at 12:35 | review | Close votes | |||
| Sep 5, 2022 at 3:04 | |||||
| Aug 31, 2022 at 12:12 | history | became hot network question | |||
| Aug 31, 2022 at 10:44 | vote | accept | Isvara | ||
| Aug 31, 2022 at 8:56 | answer | added | Christophe | timeline score: 22 | |
| Aug 31, 2022 at 6:45 | answer | added | Kilian Foth | timeline score: 39 | |
| Aug 31, 2022 at 6:44 | comment | added | Philip Kendall | How big are each of your rows - or more directly, why can you not just load all ~3M items into memory at once? | |
| Aug 31, 2022 at 6:28 | comment | added | Ben Cottrell | Could you create a temporary table and import the CSV content? -- stackoverflow.com/questions/2987433/… | |
| Aug 31, 2022 at 4:11 | history | asked | Isvara | CC BY-SA 4.0 |