I am working on Application which goes and get's market data from different sources, I am using polling mechanism where vendor/broker put his data file and my application would go and hit that ftp location multiple times during the day to get data.

Right now, we hit location every 5 mins and if data is not there then again after 5 mins. In it's current software architecture system using lot of resources between time window of 3 hrs where we do this polling mechanism.

I want to improve performance of application and so what would best practices to solve this kind of software problem?


Some might say that this problem is engineering and should not be post here but my target audience is software guys working in finance and so i have posted this question here, if you feel that this is not right location for this then please feel free to migrate it to stackoverflow or programmers.stackexchange.com

  • I would recommend a migration to ProgrammersSE
    – SRKX
    Commented Feb 7, 2012 at 12:45

1 Answer 1


If you cannot have the information "pushed" to you, you have to build your own "push".

That is, you should have one application that does your FTP polling and the other, not directly connected to the first one, that retrieves the new data and treats it. Your second application should "subscribe" to the first one in order to receive a notification for the availability of a new file. This way, you should use very few resources since your main application will only launch/work when receiving a notification. Disconnecting into at least two applications is also more flexible since with the notifier/subscriber architecture any other new application you write would only need to subscribe to your polling service to be able to work with it.

Also, to use even less resources, you can poll with an increasing time frame. For example, let's say your polling time frame is between a minimum of 2 and 10 minutes. You start with 2 minutes. If the polling application does not find a new file, the time is increased by 1 minute, until you reach a maximum of 10 minutes. If the polling application finds a new file, the time is set back to 2 minutes. You can even imagine a decreasing time frame...

Finally, to improve your architecture even more, you can even break the problem into 3 applications: one for the polling, one for retrieving the data and storing it somewhere (like a cache), and another application that treats the data (and therefore has no relationship whatsoever with the "source" of the data, be it FTP, HTTP, or a web service)

Edit: below, an example of an architecture based on web applications and web services for my first proposal (two applications).

The Poller Service can be a web application composed of:

  • One Web Service composed of two methods:

    • Subscribe for allowing a client application/web service to register for receiving information that a new file is available. One of the parameters of this web method should be the HTTP address where this "client" could be reached at.
    • Unsubscribe for not receiving anything anymore.
  • One service triggered every x minutes for polling an FTP for new files. When a new file is found, the service calls every subscribed "client" Web Service's NewFile web method (which I describe right below)

Now, a client application can be really anything but it should have a web service facade with at least the following web method:

  • NewFile, for allowing the poller to push the information on the availability of the new file. Within this web method the poller may also send some more information on the file such as the size, when it was modified and of course its exact location.

  • When this application starts, it calls the Subscribe method of the Poller web service.

  • When this application exits, it calls the Unsubscribe method of the Poller web service.

Edit: you asked how it would be with REST services.

Well, the definitions of the services do not change. The Subscribe REST service should receive an URL that is in fact the location of the NewFile REST service on the client web application. As for how the services should be written:

  • the Subscribe and Unsubscribe on the server web application should probably accept respectively POST and DELETE HTTP requests (for both I guess POST would work too).
  • the NewFile REST service on the client web application should accept POST HTTP requests.

When a new file is detected, the server web application posts the location of the file to each URL that has been registered as an active NewFile compatible REST service. That's it.

  • I really like your suggestions. I would like to learn more about this kind of architecture, is there any other related articles or information that I should be looking for online or any book chapter or any blogs that explain this kind of architecture.
    – Rachel
    Commented Feb 7, 2012 at 16:39
  • @Rachel Well, it is part of so-called SOA architectures, but you don't need to build anything huge such as an ESB... If you build your solution with web services you should be fine. I edited my answer to given you an example of architecture with web services.
    – Jalayn
    Commented Feb 8, 2012 at 6:49
  • hey jalayn, thanks for your suggestions, i would like to understand your architecture more in detail considering REST webservices, can you elaborate how we can achieve this?
    – Rachel
    Commented Aug 8, 2012 at 16:20
  • I elaborated a little bit, but really there is not much difference, I still hope that helps you.
    – Jalayn
    Commented Aug 9, 2012 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.