I have a Python-Flask app in which users can place files into a folder. As of now the directory structure is something like:

/app
    /storage
    /templates
    .
    .
    .
    server.py

The user can create directories and subdirectories within storage to organize files.

So far I am making AJAX calls to the server to get details of folders etc. and populate a dropdown list from which users can traverse the tree. It works, but it just doesn't feel right. What other approach could I take?

The major issue that I'm hoping to address (and what I'm doing right now):

There is a lag between when the user selects a folder to when it's contents are shown. Of course I don't expect it to be as fast as opening directories on a local file system but it'd be nice to improve UX there.

Current solution: I'm loading information about the contents of all the subdirectories as soon as a directory is selected. I can then show contents of the subdirectory as soon as it is selected.

Suppose the directory structure is:

/foo
    /bar
        /sub
        hello.py
        world.py
    /car
        hello.c
        world.c

I'm fetching information about files/subdirectories within bar and car as soon as foo is opened. This helps in showing them contents as soon as bar or car is selected.

Advantage: UX is good

Disadvantage: Transfers too much unnecessary data from the server. And as I'm actually accessing the disk to open each subdirectory this may become slow if many subdirectories are present!

Also, I'm not sure how well this will scale, if it is ever necessary.