Return to Question

The manual upload via the website is trivial and is not really the focus of this question. Users would log in as they currently do, and upload the file via the standard multipart/form-data interface in their web-browser. Once the upload is processed then they would receive feedback of any errors or a success message, along with auditing to show the history of uploads.

As part of the file-upload process the user would need to signal to my Windows service that the upload is complete, either by renaming the uploaded CSV file, or by uploading a signal-file (eg. csv-xyz.complete) once the CSV file is uploaded.

The manual upload via the website is trivial and is not really the focus of this question. Users would log in as they currently do, and upload the file via the standard multipart/form-data interface in their web-browser. Once the upload is processed then they would receive feedback of any errors or a success message, along with auditing to show the history of uploads.

As part of the file-upload process the user would need to signal to my Windows service that the upload is complete, either by renaming the uploaded CSV file, or by uploading a signal-file (eg. csv-xyz.complete) once the CSV file is uploaded.

The website already exists, however the feature to upload CSV files is still in the planning stages. The website currently enables users to login and manage their data (ie. it has privilege control at the user level) and its all managed through an SQL Server database. The website runs ASP.NET on IIS - this cannot be altered. If you need more information about the website, please ask.

1. Users must be able to upload files manually themselves through the website
2. Users must be able to upload files automatically at scheduled intervals from their own PCs
3. Only CSV files (or the equivalent data a CSV file contains) will be processed into my website's database
4. The entire process MUST occur without intervention from my IT support (and preferably their IT support too, though that may be out of my control, depending on other chosen solution)
5. The file upload process must not introduce any security risks that would not also be present when submitting data through a form in the website
6. The file upload process must be compatible with IIS
7. Whatever solution I implement must be well supported for at least the next 10 years into the future (as of 2016)
8. The chosen solution must use standard software that is already cheaply available to users (preferably at no cost)
9. There MUST be automatic feedback to the user when bad data is uploaded so they can correct it and re-upload. The shorter the time delay between uploading and receiving feedback the better.

6. Can IIS handle say 10,000 different FTP users each with their own secure quarantined directory that none of the other users can access, and which they cannot break out of and access other areas in the system? Is it possible to set up SFTP/FTPS directories with .NET? At a glance it seems so, however I would like to know if there are any limitations?

7. How likely is SFTP/FTPS to be around in 10 years? It seems like an extremely popular technology today, but I would like to know if it is about to be superseded or is going obsolete without my knowledge.

The website already exists, however the feature to upload CSV files is still in the planning stages. The website currently enables users to login and manage their data (ie. it has privilege control at the user level) and its all managed through an SQL Server database. The website runs ASP.NET on IIS7.5 - this cannot be altered. The server is not NATed but it is firewalled. If you need more information about the website, please ask.

1. Users must be able to upload files manually themselves through the website
2. Users must be able to upload files automatically at scheduled intervals from their own PCs
3. Only CSV files (or the equivalent data a CSV file contains) will be processed into my website's database
4. The entire process MUST occur without intervention from my IT support (and preferably their IT support too, though that may be out of my control, depending on other chosen solution)
5. The file upload process must not introduce any security risks that would not also be present when submitting data through a form in the website
6. On my website's side, the file upload permissions must be configurable on a per-user basis using either IIS, ASP.NET or a table in SQL Server and the firewall must not block the uploads
7. Whatever solution I implement must be well supported for at least the next 10 years into the future (as of 2016)
8. The chosen solution must use standard software that is already cheaply available to users (preferably at no cost)
9. There MUST be automatic feedback to the user when bad data is uploaded so they can correct it and re-upload. The shorter the time delay between uploading and receiving feedback the better.

6. IIS can only handle FTPS, it cannot handle SFTP. Also, SFTP runs on a single port, whereas FTPS runs on multiple ports, which is probably no good with my firewall (I will test and update this). Are there any SFTP programs which run outside IIS that can be configured via ASP.NET? Is it possible to set up SFTP/FTPS directories with .NET? At a glance it seems so, however I would like to know if there are any limitations?

7. How likely is SFTP/FTPS to be around in 10 years? It seems like an extremely popular technology today, but I would like to know if it is about to be superseded or is going obsolete without my knowledge.

Manual upload - A user logs-in to the website and uploads a CSV file

Automatic upload - The user somehow configures their CSV generating software and their PC to automatically upload CSV files to my website. Again, I have no control over how they choose to do this, but I would imagine that they would use a program to generate the required CSV files into a directory (ie. new CSV files would be appearing in a directory about once a day) and then these would be uploaded automatically to my website by some form of client kicked off by a scheduled task (again, I have no control over the client the user chooses, though I can offer advise). This is by no means a requirement - they can do this by manually clicking on their CSV-generating software, creating a file, and then manually running the client themselves to upload the CSV into my website if they want to. However the whole point of the automatic upload is that they can just leave it to run by itself so that IT support can use their time more productively. The upload would generally be on a daily basis, but I will not be imposing any limits through my site on the number of uploads done per day. If they want to upload a new CSV file every hour then that's fine by me. However, a frequency of daily is probably just what will be most convenient for my users. If the automatic CSV upload fails then my website will notify the user of this fact. I haven't decided exactly what form this notification will take yet since it really depends on the chosen solution. In the case where the upload fails due to the uploaded CSV file containing bad data then the user will need to manage this by exception (ie. a human will have to get involved - this is not something that software can fix). They will need to investigate the reason for the fault and correct the CSV file, probably by correcting the data in their database then using their software package to re-generate the CSV file. They can then upload the corrected CSV file at their leisure - uploads are not time-critical, and bad CSV files will be simply ignored by my website, rather than being partially loaded into my database.

Manual upload 

A user logs-in to the website and uploads a CSV file

Automatic upload 

The user somehow configures their CSV generating software and their PC to automatically upload CSV files to my website. I have no control over how they choose to do this, though I have spoken with some of my users and they have told me that their software is able to automatically generate the required CSV files into a directory (ie. the new CSV files appear in a directory about once a day without human intervention). They could then uploaded these CSV files automatically to my website by some form of client kicked off by a scheduled task (again, I have no control over the client the user chooses, though I can offer advise). This is by no means a requirement - they can do the steps I have described manually by clicking on their CSV-generating software, creating a file, and then manually running the client themselves to upload the CSV into my website if they want to. However the whole point of the automatic upload is that they can just leave it to run by itself so that IT support can use their time more productively. 

The upload would generally be on a daily basis, but I will not be imposing any limits through my site on the number of uploads done per day. If they want to upload a new CSV file every hour then that's fine by me. However, from my discussions with users it sounds like a frequency of daily is going to be most convenient, and the time of day doesn't matter. 

If the automatic CSV upload fails then my website will notify the user of this fact. I haven't decided exactly what form this notification will take yet since it really depends on the chosen solution. In the case where the upload fails due to the uploaded CSV file containing bad data then the user will need to manage this by exception (ie. a human will have to get involved - this is not something that software can fix). They will need to investigate the reason for the fault and correct the CSV file, probably by correcting the data in their database then using their software package to re-generate the CSV file. They can then upload the corrected CSV file at their leisure - uploads are not time-critical, and bad CSV files will be simply ignored by my website, rather than being partially loaded into my database.