I'm not sure if this right location for this question, but here it is.
I currently have about 30 different CSV or TXT product data feeds. I am trying to build a tool to
1) upload each feed
2) break apart each feed line by line
3) extract every title, category, price, etc. into a product object, and...
4) either create or update each product in the database.
Here is my problem. Each feed has a different name for header attributes like
price, etc. Also each feed's categories are unique as well.
So far the approach I have been leaning toward is: (Note this is a rough overview)
- Admin fills out UI form. The Admin selects a file type & attaches the product data feed
- Upon submit, the product data feed is sent chunk by chunk and saved to disk via socket.io
- Once the file is saved a job is added to our job queue to process the data feed
- Once the job is called, we first check for new header data attributes, if a new attribute is discovered, we put the job on hold and set a UI prompt for the Admin to define what the header attribute maps to, Fe. the Admin would map
Classification -> category,
Name -> title, or
Product Name -> title.
- Once all header attributes are recognized, we break the file down line by line using streams.
- With each broken down line we grab the
price, etc. We then check to see if the product exists, if so, update the product, if not, create the product.
- Once each line is processed we delete the file and mark the job as complete.
What would be a good way to map incoming feed categories to my db categories? I was thinking again to set a UI prompt, Fe.
Mens Footwear -> Mens Shoes.
Am I going in the right direction? Should I consolidate all the feeds and then import? Or am I just going about this all wrong?