1

I've got a console application which pretty much interacts and prepares some data in a SQL Server Database, exports a table, encodes, gzips and uploads to an AWS S3 bucket. The application is complete and runs as expected however I need to schedule this application to run daily (Weekdays) in the evening...

Please can you give me some insight into what is the current "trend" to schedule console applications to run. Obviously there are multiple ways and Windows Task Scheduler seems to be the most obvious way however I would like to know if this is still the current trend or is there a 'new' 'better' 'shinier' way of scheduling applications to run?

At the end of the day it really doesn't matter but I just want some insight into the current trends in 2019 and practice up to day industry standards.

Thanks in advance :)

New contributor
Dylan Delport is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 1
    Is the intention for this console app to run in the cloud, or run locally and connect to the cloud? – neilsimp1 Sep 12 at 12:06
  • The App will be stored and run locally on a EC2 instance. I am trying to explore options of running on the cloud however Lambda won't work as the interaction with the DB goes well over the 15 minute execution limitation. – Dylan Delport Sep 12 at 12:21
  • Do yourself a favor and use whatever works (as long as it works well), not what looks "shinier". Even the old "at" tool under Windows or "cron" under Unix/Linux can be sufficient. – Doc Brown Sep 12 at 12:23
  • Thanks Doc, very valid point. – Dylan Delport Sep 12 at 12:24
0

If you don’t mind using another cloud service, I like Azure Webjobs with built-in cronjob system for this.

See: https://docs.microsoft.com/en-us/azure/app-service/webjobs-create

  • Thanks Dirk, I'll look into it. Already linked up with AWS so I wonder if there is an AWS equivalent. – Dylan Delport Sep 12 at 12:18
  • You can use a cloud based service without compromising security as you need access to all resources from the point where you access the data (where the service runs), uploading to AWS is the last thing you do, also will be the slowest as you need to move all data to the cloud before you zip it... – PPann Sep 12 at 17:51
0

You need to consider a few things

  1. From where can I access the data and my tools(SQL server, file storage, ETL tools or even BCP);
  2. How do I maintain security, doubt you like or are allowed to store proprietary user account data on a 3rd party server;
  3. How will I monitor the success of the task's steps;

First SQL server has a scheduling and task based system, costs nothing and can do all you mentioned. Then you can schedule windows Jobs (type task at search bar of windows) also costs nothing.

As your data is in SQL Server I would suggest starting with SQL server, it has been around since … for ever and is not going away soon. you can schedule all kind of jobs, not just TSQL also start apps and run dos and powerhell scripts.

SQL allows you to try again for N times and create a flow of good and bad case and use different methods of authentication on each step. Best of all you can script the task to TSQL and install it on another server any time you like.

You can query the status of your job by looking in the MSDB database on your server with Excel or write a Little app if you like. Also jobs will be picked up by enterprise monitoring tools and you can write errors to mail, sms, windows eventlog (if hosted on windows) or all of them.

New contributor
PPann is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • We originally had it scheduled through SQL Agent jobs however we ran into permission issues on the clean up stage. Thus we looking at other options and Task Scheduler was the next one on the list. One of the things the application does is cleans up old files. Seems that SQL Server agent didn't have the right permissions for the folder and although we provided the agent "user" with full permissions, we kept getting the same error: `Executed as user: NT Service\SQLSERVERAGENT. Unhandled Exception: System.UnauthorizedAccessException: Access to the path 'D:*\2019_09.csv' is denied. – Dylan Delport Sep 13 at 11:09
  • @DylanDelport, that's an easy fix, use a domain user say MyDomain\BatchJob1234 for the disk related tasks. the permission issue you are having is a OS issue, not a SQL. have a look at dba.stackexchange.com/questions/60984/… – PPann Sep 13 at 11:26
-2

I would go with "Don't"

The trouble with running standalone processes on schedules is that really the processes need to know about each other and you need to know about them when they are not running.

  • Say for example your process crashes half way through. Should it be triggered again at the next scheduled time?

  • What if it takes longer to run than the delta between schedules? should you run a second instance while the first is still running?

  • What if its half way through? Can you monitor it and find out if its still working or if its hung?

  • What if it's currently between runs, can you query it to see when it last run and if it errored?

Instead of having a stand alone application running on a schedule, write a service which runs continuously but has an internal wait between "runs".

The service can report its health when queried, knows when the process is running, how long for and if it's making progress or blocked etc.

Ideally you can move away from scheduled "big jobs" and have the service constantly process small chunks of the big job

  • The question is specifically about "modern methods to run a Console Application on schedule", not about pros and cons of scheduled jobs against alternatives. – Rumen Georgiev Sep 13 at 8:04
  • presumably a question about modern "ways to ride my horse to work" wouldnt want any mention of cars either. – Ewan Sep 13 at 8:14
  • Task scheduling is by no means obsolete, it's a valid approach in many cases. Nearly every major platform like DB servers, CRM/ERP solutions, operating systems, cloud environments, etc. have some kind of implementation. Those implementations are not developed for "people who want to ride their horses to work" :) – Rumen Georgiev Sep 13 at 10:21
  • it is my contention that that is exactly why they are developed – Ewan Sep 13 at 11:01

Your Answer

Dylan Delport is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.