Why should someone use a stream processing engine like Apache Spark, Flink, Hadoop instead of just a normal backend worker which works on something and returns the results as soon as it's done?

Credit card fraud checking example is given when we talk about these solution so what is the problem by just write a program and put it as backend service which does this for us and returns the result?


Spark and similar engines are built with processing massive amounts of data in mind. They enable you to run your calculation on dozens or hundreds of machines in parallell. This is not so easily implemented from scratch.

So, for data which can be processed on a single machine, you are probably better off by writing a piece of code completely by yourself, but when the data grows and cannot be processed on a single machine due to the amount of memory and CPU power required, Spark or another such engine, are a much better choice. It leaves to you writing your processing logic, but handles the "logistics" of spreading the load between machines and combining the partial results into a single result.

  • The thing I don't like about it is, it doesn't let you do it in the language you are most familiar with, like for example a Ruby developer can't write the processing logic and let spark/flink to handle the logistics, specially with flink it doesn't even provide python's full support. Let me know if I am wrong here, thanks – Muhammad Feb 5 '19 at 11:34
  • 1
    @Muhammad Spark runs on the JVM, so it's natural to code in JVM languages such as Scala or Java. However, there are bindings for some other languages: spark.apache.org/third-party-projects.html I'm not sure about Ruby but for Python, there is PySpark for example – Michał Kosmulski Feb 5 '19 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.