1

I have to develop a c# application that will read large xml files. XML files will have data about computer systems (i.e. Hardware, software, network, bios information). There will be separate xml file for each category i.e. hardware xml file, software xml file. A computer id is common attribute in all these xml files. It might be possible that a computer id present in one xml file but not in other. After the reading part is done, and storing data locally, it needs to be written in different csv i.e Hardware.csv, software.csv. Only those id's need to be written that are present in all xml files. I am thinking of the below approach to do this,

1) Read each xml file, store its computer id in data structure like Dictionary (with key being its computer id and value is Arraylist>). Each object will have associated attributes. For Example - Hardware xml will be read and stored in dictionary as <1, hwarraylist>. Each hw object will have attributes as hwmodel, hwmanufacturer etc

2) While reading the second xml file i.e software xml file, if the id already exist in dictionary then a swarraylist will be appended to its value. Otherwise a new computerid is added to dictionary.

3) After all reading is done, I'll write arraylist of only those id's which have number of arraylist = number of xml files. Separate csv for separate arraylist.

Is this approach correct ? Will it be feasible for millions of rows in each xml file ?

One more thing to add, the xml file is as per the below format:

    <result_sets>
      <result_set>
<cs>
  <c>
    <wh>123</wh>
    <dn>Computer ID</dn>
    <rt>1</rt>
  </c>
  <c>
    <wh>112</wh>
    <dn>Name</dn>
    <rt>1</rt>
  </c>
  <c>
    <wh>124</wh>
    <dn>System</dn>
    <rt>1</rt>
  </c>
</cs>
<rs>
  <r>
    <id>8820</id>
    <cid>230</cid>
    <c>
      <v>230</v>
    </c>
    <c>
      <v>Windows</v>
    </c>
    <c>
      <v>system32</v>
    </c>
  </r>
</rs>

Is it possible to parse them using XMLReader or I should treat then as text files and do the processing. Please advise

  • Regarding feasibility: in your step 2) Why read (and store) subsequent entries when the id is not present in all previous ones? This would only make sense if you process the files in parallel (as then it is unknown if the current entries will actually make it to the CSV files). Additionally you could even remove all previous data belonging to an id not present in the currently processed file. This should depend on whether you expect most files to contain most ids, i.e. whether removing unneeded entries does outweigh the search effort. – CaringDev May 13 '15 at 16:48
  • "Is it possible to parse them using XMLReader" - anything which is valid XML can be parsed using an XMLReader, what else do you expect? – Doc Brown May 14 '15 at 15:32
  • The XMLReader will do the parsing by searching the nodes and retrieving their values. In this case, the "cs""/cs" tag contains the keys and values are within the "rs""/rs" tags. There can be multiple "rs" "/rs" tags. – Anky May 14 '15 at 16:26
  • I think the only way to parse such kind of file is to treat them as text file and read them line by line. Do you think of any other way to do this. – Anky May 14 '15 at 16:27
1

millions of rows - you have to understand the memory requirements of each row stored, if you have 1 million rows and each row is 1k (including object overheads) then that's a gig of RAM used. Strings can get big, so you might run into memory constraints quite easily.

What you are trying to do here is a classic merge problem (you should write it in COBOL - the classic COBOL program is one that takes a list of employees, and another list of work and merges them together to run payroll!) that works best if the input files are sorted.

If they were sorted in id order, then you can open each and read an entry from each, 1 at a time, if the IDs match write out a row to the output file, then repeat until all input is exhausted. Memory usage is minimal, efficiency is great (as you do not need to store and remember all IDs, they will come to you in order)

So I would work on a sorter for the xml inputs first.

0

Is this approach correct ?

If you mean "does it match the problem description" - yes, seems to me.

If you mean - "did you understand the requirements correctly" - who knows but you.

If you mean - is there a better approach - there is always a better approach, but maybe what you wrote is sufficient for your case, you will probably find out by trying it out.

What I probably would not do is just appending your category lists to an ArrayList . If you are not very careful, you will end up not knowing which of the different categories is at which index in your list. Using a Dictionary<string,CategoryList> can fix this - where the key is the category name, and the value is a supertype of your different category lists.

Will it be feasible for millions of rows in each xml file ?

That depends on the correct implementation and your available hardware/main memory. If your XML files are very big, you should avoid to read them completely into an XmlDocument, because that it very memory expensive. Better use something like an XmlReader, which processes the file sequentially. Nethertheless you have to test by yourself if holding all the CategoryLists in memory will work in your case.

-1

Millions of rows !! Think performance first :-)

1) You can probably think of creating a WCF service or Windows service to process the XML parsing & CSV generation steps.

2) Use SSIS package to parse the huge XML file into data to be stored into the database. This will help you to churn the massive large XML file into records. Then you can use to generate the CSV file from .net code. Recently, i used SSIS package to process huge record set and it took very less time to churn it.

Since there will be million of records in the XML file, i think it will add plenty of constraint on the memory while parsing the XML into object form.

  • 3
    Premature optimization is the root of all evil -- DonaldKnuth – mcottle May 13 '15 at 8:50
  • Since the scenario requires to handle millions of rows, so i suggested to use SSIS. – Karan May 13 '15 at 8:58
  • "think performance first" and "WCF service" are not phrases that go together. – gbjbaanb May 13 '15 at 9:05
  • Although I hate SSIS personally, This is an approach many companies would be happy to use. and you should think of performance first for this kind of thing – Ewan May 13 '15 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.