0

I'm a very new programmer developing my first application in Java as a side project for my employer. I'm a part-time student working full-time hoping to eventually score a developer job, and my employer encouraged this by allowing me to set aside some time daily to work on it. For the sake of anonymity, we'll call my employer ABC.

Problem Background

ABC stores documents electronically, and receives many electronic documents each month. ABC also receives a fair amount of paper documents each month. Some of these paper documents are duplicates of electronic documents received and shredded once they've discovered this. However, some are found to be novel (not duplicates!) and need to be manually entered (read: hand-keyed by data entry workers) into the e-document system, scanned, and then shredded.

These documents come from different sources and can have slightly different formats, but contain the same information that is going to be input. Data in received documents can be used to distinguish between unique and duplicate documents - specifically, dates and other values. If enough of these match to an existing document (easily implemented in an matching algorithm that scans a "document group overview" page), then the current document can be marked as a duplicate and skipped.

The e-document system is a locally-hosted server, but accessed through a browser. Documents are stored in this system are grouped and organized based on specific associations, tethered electronically via an ID number that is used internally - but not present on the documents when received from external sources.

Use Case

I decided I wanted to develop an application that would automate this process with four phases, with the following tools in mind:

  1. OCR of the paper documents after an initial scanning (Tesseract/Tess4J)
  2. Identifying necessary data during OCR and storing it locally
  3. Identifying if the paper document is novel or duplicate (Selenium)
  4. Automatically entering novel documents into the system (Selenium)

Classes

My intent is for no less than three or four classes:

  • A: an OCR/analysis class that fulfills phases 1 & 2
  • B: an automation class that minimally, fulfills phase 4, possibly phase 3 also
  • C: an initialization/control class that performs housekeeping tasks
  • D: (optional) a search/identification class that fulfills phase 3

A calls a Tesseract instance on an image file passed in from C, analyzes the document line-by-line, and stores data in local variables to be passed into the automation class.

This class is fairly straightforward - the information that I want is the same from document-to-document. The only complicating factor involved is that documents received from different sources come in different formats - the data of interest is stored in different places. Making A an abstract class (with Class extensions for each document source) is a possible solution here, but it would have to be fairly simple. Tesseract processes images line-by-line, and since each line will have different information depending on the document source, processing methods will need to vary for each.

Is this a good approach to this class?

B is a potentially complex class, and one I've been struggling with in regards to design strategy. It needs to perform the following functions:

  • receive passed-in variables from the OCR/analysis object (analyzed image file)
  • **automate searches for the document group the analyzed document/images belong to
  • **identify the instance of a duplicate document, and in the event of such, stops the automation process
  • perform basic navigation functions (to search pages, to "add document" pages, etc)
  • navigate to each sub-page of the "add document" page
  • enter information belonging on each sub-page via forms/selectors

I'm struggling with this in regards to methods/class design. There are a lot of repeat behaviors (ie: lots of selectors, fields accessible/unblurred only after a selector value is chosen) as well as a select few of random behaviors (a field that generates a JavaScript popup after a value is keyed in, in which a link must be clicked). There is also a key data type/sub-page that, based on information within the paper document, will vary in terms of fields available.

Is it possible to produce generic methods (ie: a selector method for selector/blurred field combinations) that can be used broadly? For the variable sub-page, would it be considered bad practice to put all of that automation code into a single method? Also, should I separate out the search/identification functions into D?

C is our initialization/control class. This does all of the "fun" housekeeping tasks.

This class sets the Selenium WebDriver parameters, starts a webdriver instance, and closes the instance when the tasks have been completed. It will perform basic navigation actions such as the initial navigation to the user login portal, where it will wait for the user to login. It should likely contain (or initialize from another class) a Swing GUI file chooser, enabling our non-computer friendly end users to easily select the document image files to be passed into A. It should contain a control loop that enables the user to scan multiple documents in a single login session (maybe with a GUI using a selector or form to tell the application how many documents are being scanned in the current session).

This class should instantiate objects from the OCR/analysis class and the automation class for each image file selected. I'm not really sure of operation time requirements yet, so there may or may not need to be a wait to allow for processing between analysis and automation. There likely should be a wait for the automation process to complete before analysis of a new image begins.

This class has the potential to be quite large (but nowhere near as large as B), so I'm not sure if I should break it up. I definitely could use some suggestions on that.

Again, feel free to offer suggestions for anything and everything. I apologize in advance for being vague and not presenting my current code - the data stored in the documents contains confidential information, so I'm trying to avoid identifiers. The code for the web application that accesses the electronic document system is also not public, so by revealing my code I might be breaking confidentiality rules that have not been fully explained to me yet. If I get the green light from my supervisor on any of this, I will share what I'm able to.

  • Without an identifier that is common to all duplicates, regardless of format, you will end up storing dupes. What suprizes me is that you seem to consider automatically weeding out dupes the primary concern of this system. I would have thought making it easy for users to find things later was the primary concern. – candied_orange Jul 14 '16 at 23:30
  • I realize that my in trying to vague to preserve confidentiality, I went a bit too far in that I didn't explain things clearly enough. I'm adding some specificity back in to clarify things. But to touch on your specific points: – drs Jul 14 '16 at 23:43
  • Yes, there are data in received documents that can be used to distinguish between unique and duplicate documents - specifically, dates and other values. If enough of these match to an existing document (easily implemented in an matching algorithm that scans a "document group overview" page), then the current OCR'd document can be marked as a duplicate and skipped. The primary concern of this isn't so much as making it easier for users to find things later (the document storage system already does that by default) but to automate a tedious task (manual data entry) and remove human error. – drs Jul 14 '16 at 23:54
  • This would be a complex project for someone with 7+ years of Java development experience. How many years (decades) is your employer giving you to complete this project? – Gilbert Le Blanc Jul 15 '16 at 23:10
  • As long as it takes, or as long as I want. Ideally, I want to design a program that another software engineer can, down the road, look at and improve if my employer decides to pass the code along. For the interim, it really just needs to work. The project itself isn't as complicated as you think - the format of input files is consistently falls into a few layouts, and with some decent regex I should be able to rip information that I want with ease. The complexity, as I see it, is entirely dependent on how well I want to write it. – drs Jul 18 '16 at 19:15
0

The road to Damascus was built in sections, and so shall yours. No one builds a 500 mile road all at once, all or nothing.

Beware the Dunning-Kruger Effect


Define data formats. Sometimes they're not obvious. These will always simplify the using code. Even small and simple structures. Data will be a primary way the software parts will communicate. And I don't mean just those classical structures we learn in school: tree, stack, queue, etc.


  • What is an abstract class?
    • Is it possible to produce generic methods ... that can be used broadly?
    • There are a lot of repeat behaviors

My intent is for no less than three or four classes Those naive of OO design are easy to spot. Upon seeing a design they immediately say "that's too many classes."


B is a potentially complex class

Good OO design/programming manages complexity by making simpler things that interact. "B" is not one class.

The Single Responsibility Principle is very, very powerful but can be groked only through diligent practice; constantly looking for coalescing abstraction and subtle concept.


a field that generates a JavaScript popup after a value is keyed in, in which a link must be clicked

This is an implementation detail. At design this is virtually irrelevant. At this point in your design any user interactive GUI is virtually irrelevant. Sure, think about it. Even sketch some of it out. But do not let it drive design.


Wishful thinking design

Design what you need. Then design whatever it takes to interact. For example:

  • Here is a class: the information that I want is the same from document-to-document.
  • Here are several classes: documents received from different sources come in different formats
  • Now integrate. Possibilities:
    • Inheritance
    • Class(es) that transposes data

Spend LOTS of time in design

... and then spend some more.

True story. Creating 2 from-scratch pieces in the same larger project. "A" was very thoughfully designed. Everyone, especially management, said we're taking too long designing. And lots of "there are too many classes." "B" was designed fast, fast, fast because "we had no time."

Final results:

"A" went through formal testing in 2 days. "B" was thrown into testing before coding was complete because it was proving very hard to code; it was a self reenforcing "we had no time." Then it spent 3 months in a "test-fail, fix/hack, retest-fail" cycle of hell. No kidding, 3 months.

Analysis

Coding "A" was fun. Coding "B" was torture.

Make no mistake about this. "A" was objectively much more comprehensive and complex functionality than "B". NOT THE CODE, the functionality.

"A"'s design expressed the business domain in business domain terms and concepts. We played "what if" and "how will this work" a lot before a single line was written. Function was integrated/fitted to the desired GUI. "B" was designed only based on the various sections of the visual form. There was no regard for the underlying business domain.

Integrating the bits of "A" was easy because they were all commonly understood business classes. Integrating "B" was like a whole extra effort of hacky patch code because everything was arbitrary grids of numbers and individual, seemingly arbitrary, values on the screen. There was no coherence behind what the screen displayed.

"A" was heavily unit tested. "B" had none. Indeed, coding "B" was so hard that "we had no time for writing tests."

  • Wow, thanks for the great answer! Based on your suggestions, I've expanded things considerably. B is now represented by an abstract class that is extended by five sub-classes, each representing individual sub-sections of the document. Each of those classes will eventually be extended by an automation class specific to the Selenium behaviors that its need to enter data. I got the go-ahead from my superior's superior that my code will be OK to share provided confidential information is not present in it, so if you would like to see my code (git) and a current class structure, feel free to ask. – drs Jul 18 '16 at 22:41
  • The controlling class(es) have also been expanded further to encapsulate housekeeping functions such as setting webdriver run parameters. I'm pretty sure I could take things a step further with what I've done with B, but I'd like to first see how my design shapes up and how the code feels. That said, any advice you or others have along the way is very welcome. Thank you for all the advice you've provided so far, it's been very helpful! – drs Jul 18 '16 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.