An approach to send the contents of a file on the server to the client in real-time

Question

I have a relatively simple front in good ol' JQuery, that makes a POST Request to a Flask Python web server, that runs a Python script using the subprocess module. Here's the whole route for illustration:

@app.route("/adyen_request_download", methods=["POST", "GET"])
def adyen_request_download():
    merchant_manager = MerchantManager()
    error = None
    if request.method == "POST":
        pfjs = []
        prefixes = []
        extensions = []
        ftpd = "/var/ftp/ADYEN"
        date_type = "specific"
        specific_date = ""
        range_from = ""
        range_to = ""
        for result in request.get_json():
            if result["name"] == "pfj-select":
                pfjs.append(result["value"])
            if result["name"] == "prefix-select":
                prefixes.append(result["value"])
            if result["name"] == "extension-select":
                extensions.append(result["value"])
            if result["name"] == "date-group":
                if result["value"] == "specific":
                    date_type = result["value"]
                elif result["value"] == "range":
                    date_type = result["value"]
            if result["name"] == "specific-date":
                specific_date = result["value"]
            if result["name"] == "range-from":
                range_from = result["value"]
            if result["name"] == "range-to":
                range_to = result["value"]

        dev = True # Checar pra False em modo de produção.
        logging.info(f"Requisição de download manual de arquivos realizada.")
        logging.info(f">> PFJs: {str(pfjs)}")
        logging.info(f">> Prefixos: {str(prefixes)}")
        logging.info(f">> Extensões: {str(extensions)}")
        if date_type == "specific":
            logging.info(f">> Data Específica: {specific_date}")
            if prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif extensions and not prefixes:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --ext {" ".join(extensions)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif not prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif prefixes and extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --ext {" ".join(extensions)} --date "{specific_date}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
        elif date_type == "range":
            logging.info(f">> Período: {range_from} | {range_to}")
            if prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif extensions and not prefixes:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --ext {" ".join(extensions)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif not prefixes and not extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
            elif prefixes and extensions:
                run_command = f'python ../InputDataController/acquirer_file_downloader.py --acquirer "adyen" --pfj {" ".join(pfjs)} --prefix {" ".join(prefixes)} --ext {" ".join(extensions)} --range "{range_from} {range_to}" --ftpd "{ftpd}" --force' + ' --ftp "localhost eric 1234" --ftpd "/adyen"' if dev else ''
        subprocess.run(run_command, shell=True)

    return json.dumps({"result": "ok"})

That program I run has an extensive logging operation using the logging module. The thing is, it's quite vital for the user on the front to see this log, as this program downloads files and uploads files to a FTP. The problem is how requests work. Would I need to run a loop on the front making AJAX Requests with parts of the log file? Is there a "stream" way of sending data from the server to the client in real-time? I don't want to implement something weird and inefficient, so I'd like to see some suggestions. What do you suggest?

Answer 1

You can use chunked transfer encoding to respond in multiple fragments. This allows you to stream the response without having to introduce drastic changes. However:

• This will require support from your web framework
• Your server will still be effectively blocked until that request completes. Python also has a bunch of web frameworks that can make use of Python's async features, such as aiohttp.
• You will have to write client-side JavaScript code to update your site when the next chunk is received. JQuery is not going to help here.

Alternative approaches are:

• Model the log as an explicit resource that can be accessed over an URL, although this log may not yet be complete at some point in time. Clients can then poll the resource to see if it has changed. This may be easier to implement.

• Use web-sockets to push events to the client. However, this requires that you have a websocket-capable server.

In any case, do not use the logging module for user-visible output. If some output is meant for the user, model it explicitly.