Hi I'm bulding an ML pipeline with PyTorch to support various tasks and am looking for some advice on efficient ways to store processed data.
The main frame work was 3 layers
[data prep] -> [data loading] -> [training/inference].
The dataprep module is responsible for taking raw data (medical data in this case) and storing it in an organized and efficient way to be handled subsequently by the dataloaders. Data preb is ideally only done once initially for each dataset and data loading/training may be done many times.
The main insights I'm looking are:
- Video files: is storing them as raw pixels
.npyfiles an okay method or am I really missing out on optimizations in standard video formats (mp4/avi...). For further info most videos are around 100-200 frames.
- Segmentations: A segmentation can be a contour of 20-40 x/y points. My plan is to save the contours all to a single json file which can be loaded directly into ram in the constructor of a PyTorch dataloader. Then every call to
getitemwould take the next contour and call something like
skimage.draw.polygonto convert it into a binary mask. Wondering first if loading all the segmentations in the constructor is naive and if I should store them as npy files, or directly as binary masks from the start them load them individually with the dataloader on each call to
- Images I think just using
pngis fine and loading during calls to
Any insights on these would be appreciated, it's difficult to find agreed upon best practices for these kinds of thing on google.