2

I have a relatively simple front in good ol' JQuery, that makes a POST Request to a Flask Python web server, that runs a Python script using the subprocess module. Here's the whole route for illustration:

@app.route("/adyen_request_download", methods=["POST", "GET"])
def adyen_request_download():
    merchant_manager = MerchantManager()
    error = None
    if request.method == "POST":
        pfjs = []
        prefixes = []
        extensions = []
        ftpd = "/var/ftp/ADYEN"
        date_type = "specific"
        specific_date = ""
        range_from = ""
        range_to = ""
        for result in request.get_json():
            if result["name"] == "pfj-select":
                pfjs.append(result["value"])
            if result["name"] == "prefix-select":
                prefixes.append(result["value"])
            if result["name"] == "extension-select":
                extensions.append(result["value"])
            if result["name"] == "date-group":
                if result["value"] == "specific":
                    date_type = result["value"]
                elif result["value"] == "range":
                    date_type = result["value"]
            if result["name"] == "specific-date":
                specific_date = result["value"]
            if result["name"] == "range-from":
                range_from = result["value"]
            if result["name"] == "range-to":
                range_to = result["value"]

        dev = True # Checar pra False em modo de produção.
        logging.info(f"Requisição de download manual de arquivos realizada.")
        logging.info(f">> PFJs: {str(pfjs)}")
        logging.info(f">> Prefixos: {str(prefixes)}")
        logging.info(f">> Extensões: {str(extensions)}")
        if date_type == "specific":
            logging.info(f">> Data Específica: {specific_date}")
            if prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif extensions and not prefixes:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --ext {" ".join(extensions)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif not prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif prefixes and extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --ext {" ".join(extensions)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
        elif date_type == "range":
            logging.info(f">> Período: {range_from} | {range_to}")
            if prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif extensions and not prefixes:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --ext {" ".join(extensions)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif not prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif prefixes and extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --ext {" ".join(extensions)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
        subprocess.run(run_command, shell=True)

    return json.dumps({"result": "ok"})

That program I run has an extensive logging operation using the logging module. The thing is, it's quite vital for the user on the front to see this log, as this program downloads files and uploads files to a FTP. The problem is how requests work. Would I need to run a loop on the front making AJAX Requests with parts of the log file? Is there a "stream" way of sending data from the server to the client in real-time? I don't want to implement something weird and inefficient, so I'd like to see some suggestions. What do you suggest?

3

You can use chunked transfer encoding to respond in multiple fragments. This allows you to stream the response without having to introduce drastic changes. However:

  • This will require support from your web framework
  • Your server will still be effectively blocked until that request completes. Python also has a bunch of web frameworks that can make use of Python's async features, such as aiohttp.
  • You will have to write client-side JavaScript code to update your site when the next chunk is received. JQuery is not going to help here.

Alternative approaches are:

  • Model the log as an explicit resource that can be accessed over an URL, although this log may not yet be complete at some point in time. Clients can then poll the resource to see if it has changed. This may be easier to implement.

  • Use web-sockets to push events to the client. However, this requires that you have a websocket-capable server.

In any case, do not use the logging module for user-visible output. If some output is meant for the user, model it explicitly.

  • 1
    Good 'ol fashioned AJAX is the answer here. Specifically readyState === 3 fires each time a chunk is received. Then just strip the new part of the responseText (which means you need to keep track of the last version of this property each time a chunk is received). – Greg Burghardt Nov 16 '18 at 16:02

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.