I have a text file of ~600 CIDR notation IP blocks which, when expanded, amount to ~17.5M IP addresses. I need to socket connect to each one. If it connects, I add it to a "live" list, if it returns an error/refusal, to a "dead" list. Then the socket is closed. I don't need to read from it, I don't need to write to it. Obviously, this is a problem of scale, if we generously assume that the connection takes only one second to return success or failure, it would take months to complete, but likely several years. I need to get it down to <24 hours.
Right now I'm using Python to expand/count each of the IP addresses, because it is trivial to do so. I am writing a simple multi-threaded C program to address the above problem. There are a few ways I have thought of to tackle this:
Purely using C: I have not found a way to expand a CIDR block in C, (handling strings in general is a pain). I could probably cook something up, but if something already exists I'd love to hear about it. Will I be able to spawn enough threads? Even if I spawn a thread for each block, that 600 threads! I feel like I need to shrink the stack space allotted to the threads to do this maybe? Even so, I need to be able to handle a large number of strings because the blocks need to be expanded. Regardless, I have looked at the list by hand, and one of the blocks has a /10 CIDR notation, which amounts to >4M IPs by itself. This would still take far too long.
Spawning C processes from Python: This would trivialize the string problem, and each individual IP could be sent to an instance of a C function called from Python, which would then end. The question I have is: when Python calls an external C function, does it continue running with the C process in parallel? Or does it wait for the C function to complete? I know Python does not allow multi-threading (or rather, it does, but it's somewhat of a joke since only one line is interpreted at a time), so is this the correct way to "export" multi-threading?
Vice versa: As above, but with C calling Python code, is this "more" correct? Which is to say, can C initiate multiple Python processes and continue to do it's own thing?
Something completely different.
Any questions, suggestions, or concerns are welcome. Please point out anything I might be missing or incorrect assumptions I have made.
Thank you for your help.