1

I was in a conversation with someone who codes for me. He was frustrated with my approach to a particular problem. I am entirely self taught and very pragmatic - I am not a professional developer instead I sometimes write code to get something done and then pass it on to him to clean it up - make it robust etc.

We are reading in large txt files that use sgml tags to mark boundaries between individual documents that are related to one another.

<opentagfor txt file>
<document>
<maybe some attribute tags of the document>
the
document
could be a uuencoded
could be html
or something else
can tell what type by the attribute tags
</document>
<anotherdocument>
<maybe some attribute tags of the another document>
the
document itself
could be a uuencoded
could be html
or something else
can tell what type by the attribute tags
</anotherdocument>
 .
 .
 .

 <closetagfor txt file>

The individual documents get consumed downstream. Sometimes they get written out to a file structure for indexing, other times some of their content is extracted and saved as 'data'.

So my approach was to create a dictionary of dictionaries.

 document_dictionary = {document_position_index:{attribute-tag-1:value1,attribute-tag-2:value2,document:the\n    document itself\n    could be a uuencoded\n    could be html\n   or something else \n  can tell what type by the attribute tags}, . . .}

document_position_index is the index position of the document in the original txt source file - has no relevant meaning at all.

now if I want to do something with the documents I can do things like

for document in document_dictionary
    if document['some-attribute-tag'] == someValue:
        do something

The problem is this bad coding (specifically using an integer (document_position_index)) was frustrating to my developer. He said that it is bad form. I still after Googling for a while do not understand. So I would appreciate either a better explanation then I got from him or some pointer to the right direction.

Is this the type of question that should be asked here?

2
  • "document_position_index" as which number from the start the document is (eg. first, second, nth)? Or line number in file where the document starts?
    – Euphoric
    Sep 7, 2013 at 18:33
  • number from the start
    – PyNEwbie
    Sep 7, 2013 at 19:48

2 Answers 2

1

Why not just save it in a list? You still get positional information of what number the document is. Your usage will look exactly same. And you don't need to worry about overhead of using a dictionary and saving your own index. And you will still be saving them under numbers 1,2,3,4,etc..

3
  • you are saying to save it as a list of dictionaries I presume.
    – PyNEwbie
    Sep 7, 2013 at 19:47
  • @PyNEwbie Yes, right now it seems to be dictionary of dictionaries.
    – Euphoric
    Sep 7, 2013 at 20:16
  • Also, if you use a list you can use map()/reduce() and/or filter()/reduce() to process the documents, which might make your code easier to maintain.
    – david25272
    Feb 7, 2017 at 23:24
1

There are a few different ways to access the data and your needs to access it should dictate which format you use. Right now, with a dictionary of dictionaries, if you want to find the nth document, you have to do a linear search through the outer dictionary to find the one with the index you want, whereas with an array, it would be a single lookup.

However, if positional data is not normally used and you may be inserting other documents into this structure, then using an array or list may not be as useful and could slow down access to the data you want. In that case, a dictionary might be a better choice.

You can learn more about how different data structures affect the memory use and running speed of various algorithms by searching for space complexity or time complexity. Some structures make it easy to add and remove items, while others make it faster to search for items. So the best one depends on what operations you're going to do the most.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.