1

I am currently dealing with an app that has several classes which are used to compare files in various formats (xls, csv, xml, html, pdf...). They are all implementing an interface that is defined like this:

public interface ReportComparator {
    // compare files (1 - match, 0 - no match, -1 - both files empty)    
    int compareTwoFiles(InputStream fileA, InputStream fileB) throws IOException;
    // gets a list of format-specific differences between two files
    List<String> getDifferences(InputStream fileA, InputStream fileB) throws IOException;
}

The specific comparators are instantiated through a class Report and are later used by a GUI application, that prints the results of the comparison. Report:

public class Report {
    private final String name, format;
    private final ReportComparator comparator;

    public Report(String name, String format) {
        // produce comparator specific for the format
    }
    ReportComparator getComparator() {
        return this.comparator;
    }
}

The problem is, when I am dealing with large files I quickly run out of memory when calling getDifferences() (increasing memory is currently not an option). Therefore, I thought of using something like a python generator, but I am having trouble visualizing it in java. Since lambdas allow deferred execution they would be a candidate for that, but would it be possible keeping this API or at least not having to refactor a lot of code (practically writing it from scratch again)?

6
  • Would changing ReportComparator::getDifferences to return a stream be okay? You need a return type which isn't required to be strict here
    – Phoshi
    Sep 7, 2017 at 8:58
  • That seems worth looking at. I will try it, thank you!
    – DCzo
    Sep 7, 2017 at 10:05
  • 4
    As @Phosi said, the thing you really need is lazy loading, which is exactly what the ’Stream` abstraction provides. This isn't about lambads or method groups, it's about the fact uou only want parts of the files to be processed at a given time. Note you should make sure this is possible over all the formats you require to compare, sometimes you may need the entire thing in memory. Sep 7, 2017 at 12:06
  • Thank you for your answer. Unfortunately, when I rewrote it using Stream I get StackOverflowErrors, because I add the differences using Stream.concat in a loop - and it is long. Can you think of a solution to that problem?
    – DCzo
    Sep 8, 2017 at 11:12
  • After some research, I suppose I could use a Supplier to 'feed' the Stream. The problem now is: how do I keep current position in the file that I am parsing?
    – DCzo
    Sep 8, 2017 at 14:27

1 Answer 1

2

I am not sure whether this is idiomatic Java, since I'm not really a Java programmer, but a nice way to do it involves rewriting your comparator as a state machine, keeping the state in some object. For example:

public interface Coroutine {
    List<String> getPrevDiffs(void);
    String getDiff(void);
}
private class SomeSpecificReportComparison implements Coroutine {
    private InputStream a, b;
    private List<String> prevDiffs = new List<>;
    private int state = 0;
    SomeSpecificReportComparison(InputStream a, InputStream b) {
        this.a = a;
        this.b = b;
    }
    List<String> getPrevDiffs(void) {
        return prevDiffs;
    }
    String getDiff(void) {
        // diff generation goes here
        // add diff to prevDiffs
        return diff;
    }
}

The state variable is for storing what state the getDiff state machine is in, and any other variables that need to be preserved must also be instance variables. Then, your comparator function would just return an instance of this class.

For more details, see the article from Simon Tatham about coroutines in c, which uses a similar techniche.

5
  • Thank you, but that doesn't solve my problem - I still need a list to keep the diffs. I want to be able to hold in memory just one diff at a time.
    – DCzo
    Sep 14, 2017 at 10:39
  • Sorry, @DCzo. Forgot about that requirement, you don't need the list for that to work, it is for keeping previous diffs if you need them later. Sep 16, 2017 at 13:43
  • Thanks for the clarification. Now the problem is, how do I keep track of my position inside a file or rather a Stream. Starting every time from the beginning would definitely take too much time...
    – DCzo
    Sep 17, 2017 at 12:49
  • It's technically in the answer, but it may not be clear. The idea is to keep all necessary state in instance variables. Sep 18, 2017 at 16:24
  • I don't know how i could have missed it :). Thank you.
    – DCzo
    Sep 23, 2017 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.