1

I am working for a company which has an abundance of Word and Excel files festering away in the depths of their file server, probably like most other companies. The problem is that each of these files contains nuggets of information that at the very least would give some valuable insight into past performance. There is no 'offical' method available to the company to ever retrieve these files, never mind review them.

So, I'm preparing a report for the company which basically says, "You treat your files as information graveyards, and your folders like document coffins!" As you can imagine, I'm trying to find a more diplomatic way of saying so!

As an example, consider a traditional contract document between a company and its customer, written up in Word. It may contain a bunch of text, maybe some images. In short, it is mainly a brochure waffling on about how great the company is, and saying the same thing that the previous contract said, but the differences are the customer's details and the contract value. Ideally, we'd be able to interrogate such a document so that we can extract values such as these and use them in our aggregate financial reporting.

Now, you are probably thinking, "Just use a template file and populate it with the details that are different! Store your keys and their values in a database!" As software engineers, we intuitively know to suggest such things. What's not intuitive to the office staff is how to recognise that they are archiving these nuggets of information away for ever, and that an alternative exists.

  • What do I call this problem?
  • What methods or patterns exist to help decompose a natural language document (like the contract above) in order to determine which key-values are present - in other words, how to determine the scope of the application domain?
Improve this question
4
  • 1
    This isn't a problem that you can attach a vocabulary word to and obtain enlightenment, nor is it a problem that will succumb to a pattern-matching exercise. This is, quite simply, the reason that you have folks like Software Engineers and Architects. It takes a coordinated effort by everyone in the organization to identify the problems, develop software solutions and establish new policies and procedures so that everyone is singing to the same sheet of music.
    – Robert Harvey
    Oct 31, 2018 at 18:13
  • 1
    As to your specific technical requirement described in bullet 2, there are technological solutions that will provide the necessary leverage, like the Open XML SDK.
    – Robert Harvey
    Oct 31, 2018 at 18:17
  • Do you have a real task (not something you invented by yourself) where you need things like contract information from lots of contracts in machine-readable form? Otherwise, it seems to me you are trying to find solution to a non-problem?
    – Doc Brown
    Oct 31, 2018 at 20:15
  • @Doc, yes, plenty of real-word examples. In fact, pretty much every time I take on a new project. The most interesting (for me) are those projects where the developer says "what if" rather than the client approaching with "we'd like to convert 'n' Excel sheets into an app..."
    – boatingcow
    Nov 1, 2018 at 8:19

3 Answers 3

Reset to default
1

Standardizing documents like contracts in a way the contained information becomes machine-readable requires some effort. You or your company don't get it for free. And this is not an end in itself, it should be a means to an end. So before you ask for methods to implement this, you better ask your management if they think this will be actually worth the hassle.

Said that, there is one method I know of to achieve what you are suggested, but it is probably not kind of answer you are expecting: make your company buy and roll-out a company-wide ERP system like SAP. That should provide the infrastructure for storing most "important" information in a structured database, and forces everyone in the company to keep contracts, orders, invoices, payment information, or other business related data in that place.

Regardless if you like such a solution or not, ERP systems will typically allow to create business intelligence documents like the financial reports automatically from the database, something which cannot easily be automated when business data is stored exclusively in unstructured Word files.

Improve this answer
4
  • That's basically it - although imagine there is no SAP or the like, and we are now proposing such an ecosystem from scratch (not that we are). The question is perhaps how to reverse-engineer the process which ended up with a contract for example, so that you end up with key-values and a template instead.
    – boatingcow
    Nov 1, 2018 at 8:21
  • 1
    @boatingcow: there is no "standard way" of reverse engineering structured data from unstructured documents. Best thing you can do is to look at the documents in your company, see if a reasonable amount of them have "enough" structure to allow developing a solution for the specific cases.
    – Doc Brown
    Nov 1, 2018 at 9:44
  • ... But in general, you cannot expect to recreate a cow from minced meat. That's why the standard solution for making business data accessible is to put them in a database at the time when the data is created, and keep the structure. Once that structure is thrown away, recreating it requires manual effort, there is no easy way around this.
    – Doc Brown
    Nov 1, 2018 at 9:45
  • I think that's ultimately the problem - software engineers know to "look for" data and are able to recognise it when it pops up, and simultaneously suggest that now is the time to codify it and stick it in a database. "Regular" users are not trained in information science and don't realise that they are inadvertently creating these information graveyards. Perhaps the answer is "training" (seems it always is) to help colleagues spot these occurences and alert their "information professional" colleagues as and when it occurs?
    – boatingcow
    Nov 1, 2018 at 10:13
0

Extracting key/value pairs is a very hard ask on non-structured documents. However there are plenty of tools for building indexes on the content. I suggest a google of something like "open source document management word excel" This should give you a way forward.

Improve this answer
0

The overall field you are interested in is called "Document Management". In an environment that has good document management practices, documents are treated as first class entities. Documents have life-cycles, processes and governors.

Documents have metadata associated with them. File system metadata such as file name, creation time, file type and so on, are the traditional metadata. In a Microsoft Office environment, additional metadata is often created in the form of document properties.

Document management processes are usually done within the context of a document management system, manual or automated.

Governance practices for documents are policies and procedures required of your organization to manage life cycle of the data in your documents. You may be familiar with governance if your organization is required to conform to Sarbanes-Oxley or other government-mandated processes.

Improve this answer
2
  • I think that's the distinction - is this document a first-class entity, or is it an unstructured container of 1st class and lower entities. UK-based, we don't have SOX, but we still have industry-sepcific best-practice and GDPR to take into account.
    – boatingcow
    Nov 1, 2018 at 8:25
  • 1
    Document management aims to take "unstructured" data and extract the structure. Document management systems, such as Alfresco, Documentum or Microsoft Sharepoint bring a lot of tools to the table. It all starts, however, with a determination by senior management that their data is, indeed, a precious resource.
    – BobDalgleish
    Nov 1, 2018 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.