0

I try to validate an arhitecture like below, but I'm not sure if is the right way. What I have here:

  1. A web application which have an Excel Module (import/export files, csv, xlsx) and display them in web pages.
  2. This web app consume a api (Scala API)
  3. When I want to import a file, Client sent the file to Scala API, the second save the file on a temp folder and call another api (Spark API) to start the process

I know the API Spark can be a job/cron and to listen a queue, but I need an answer from them (200 - found file in temp folder and start the process, or 404 - file not found, or 500 server error) to return to Scala API, to return the received answer to client.

When the file is found on Spark API, this return 200 and start the procees on another thread, to not be awaited.

class SparkService extends ISparkService {
  override def importFile(path: String) {

    val tempFile = new File(path)
    if (tempFile == null)
       BadRequest("File not found")
    }

    // this is not awaited
    Future {
      processFile(path)
    }

    Ok("started")
  }

  def processFile(path: String) {
     val file = SparkContext.context.textFile(path)
     val rows = file.take(10).map(mapper)
     // ...
  }
}

My problem is ... if is correct to do the process in a new Future. I know SparkContext is builded hardly, so this is instantiated as an object (or singleton class) when the application is started, and injected in services to be used.

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.