4

I have a mainly Scala application and I am interested in approaches to integrating Python code into this application in a way that is proven by you personally to be successful.

In this context

  • integrating means allowing Scala code to call Python code, somehow, and use the results or access the exception
  • successful means the approach was used more than once because it allows the two languages to work together to deliver business value and the approach was used a second time by the same team

Hopefully this question admits factual answers rather than discussion because I am asking for factual observations. However if this is too subjective then as you close this could you please suggest one of,

  • How I can ask this question in a non-subjective way OR
  • Point me to a site where I can ask this type of question.

Thanks.

closed as too broad by gnat, Robert Harvey, Lie Ryan, Greg Burghardt, Philip Kendall Dec 28 '17 at 14:10

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    As Scala is running on the JVM, I would consider Jython at jython.org a possibility. Leaving just a comment only as I have not actually used Jython with Scala. – juhist Dec 22 '17 at 16:53
  • 1
    Caveat, I have not done any Scala development. However, the most reliable way that I have used (across languages) for invoking something that has a completely different runtime engine is to start a process and monitor that process for completion. This has worked reasonably well for a large number of applications. I tend to resist calling complicated third party software of unknown quality within the main application process because if that crashes it can crash in a way that kills the main app. The systems I've worked on need to be resilient to such events. – Berin Loritsch Dec 22 '17 at 16:57
  • 1
    Example of a bad crash: calling a tool to import a shape file where the GDAL metadata is not properly configured. The GDAL libraries could easily crash the dotnet runtime environment so hard you couldn't even catch an exception. That particular issue is mostly resolved now, but wrappers around C libraries that don't set up the environment correctly are particularly notorious. – Berin Loritsch Dec 22 '17 at 17:01
4

Capture results via a variable and capture exceptions via STDERR as follows:

in Scala:

import sys.process._
def callPython(): Unit = {
    val result = "python /fullpath/mypythonprogram.py" ! ProcessLogger(stdout append _, stderr append _)
    println(result)
    println("stdout: " + stdout)
    println("stderr: " + stderr)
}

and in Python:

try:
    throws()
    return 0
except Exception as err:
    sys.stderr.write(f'Exception: {err}')
    return 1

For further review see the package process here. Also visit ProcessBuilder here and ProcessLogger here

  • Thanks for the answer. Have you used this on a project and if so does it meet the definition of successful from the question? – Janek Bogucki Dec 28 '17 at 12:38
  • 1
    This is a “trimmed-down” version of what I have used on many projects. The structure is the same for many languages such as Perl, Ruby and Bash making a system call to another language such as Java. – tale852150 Dec 28 '17 at 13:13
  • Could you explain how any exception resulting from the Python script is made accessible to the Scala code? – Janek Bogucki Dec 28 '17 at 14:49
  • @JanekBogucki - write exceptions in Python code to stderr using sys.stderr.write() and capture in Scala using ProcessLogger(stderr append _) – tale852150 Dec 28 '17 at 15:15
  • 1
    @JanekBogucki - updated answer with more detail. – tale852150 Dec 28 '17 at 15:30
3

There is not going to be a single answer to this because each case is different and all approaches has been used successfully by different teams.

In general there are three approaches:

  1. If both languages has the same runtime, you can compile both languages to that runtime and use a foreign function interface. In this scenario, you use Jython and Scala, which both runs on JVM. This is generally the fastest and least overhead, but you'll have to deal with some impedance in the way each language treats objects in their language, and you don't have any isolation so poorly written code in either language can crash the other. Additionally, it can be cumbersome to scale this to multiple machines.
  2. You can spawn subprocesses when processing each request in the main app. The main process communicate with the subprocess by streaming data using stdin and stdout, and possibly other pipes or any other OS specific IPC. This is generally best if the subprocesses are a filter type of program that can be used in a pipeline. There's going to be some overhead in creating subprocesses for each main request, but if you're doing this on a Unix-based system like Linux, creating new process is really fast as the system is optimized for it.
  3. You can create an use inter process communication and communicate with a microservice using a message passing API. Example of this is to run a microservice running an HTTP application server or communicating with domain sockets. With this approach, you have some overhead due to rendering, copying the messages, and parsing the messages, so it's best used when the messages are coarse-grained rather than making lots of small calls. You will need to design an explicit API, but this approach is usually simpler in the long run and can be made more robust since crashes are separate and won't affect the other processes. This approach is also much more easy to scale as having an explicit API makes it more straight forward if you ever need to run the processes on different machines.
  • Why is the first option difficult to scale? – Max Dec 28 '17 at 12:23
  • Thanks for providing this top level approaches. Can I check with you that each approach strictly matches the question requirements? In the question I have changed "someone" to "you personally" to further reduce the admissibility of answers that might be construed as subjective or speculative. I am after facts won from experience! – Janek Bogucki Dec 28 '17 at 12:57
  • 3
    An API designed for use with foreign function interface usually takes advantage of features like foreign objects that are designed with the assumption that foreign calls are relatively cheap (as round trip latency is almost zero) so it may do lots of smaller fine grained calls or it may make use of features like shared memory which isn't really possible with multiple machines. Additionally, when you do explicit message passing, you tend to become more conscious about the number of round trips that the program are doing and you'd naturally design to minimize it. – Lie Ryan Dec 28 '17 at 14:18
  • 1
    I definitely favor option 1. I've employed it on the CLR using IronPython and C#. Despite the heavily mismatched syntax, being able to communicate in terms of fully structured transparently self-referencing object-graphs is a big win. – Aluan Haddad Dec 28 '17 at 21:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.