-1

I have an application that process portfolios. I must aggregate 8 years of returns of aproximatelly 8K securities. (about 2016 doubles (8 years of data) for 2 dates (for comparison) 4032 for 8K securities.)

This is consuming aproximatelly 2GB of memory. I wonder if there is a better way of processing this other than holding this in memory.

I use Dictionaries to hold this data.

Improve this question
0
2

It sounds like what you are doing is better done by a good database than directly in C#. Using MS SQL Server you can store all of the data and use queries/stored procedures to transform the data however you like.

Database engines are designed exactly for this purpose. They have optimisations in place to cut down on memory usage and improve speed. Theoretically you could write code which outperformed a database engine for your specific data, but it would be hard and time consuming and generally not worth doing.

Improve this answer
6
  • I partially agree. I am more into using database to store data only. My application allows the users to group the records in any manner they desire. This demands my calculations to be performed again. (It changes the sub groups of data). What I have to do then is to get 8 years of data of each subgroup and calculate the aggregation and that is why it is more convenient to keep data in memory. – Felipe Almeida Sep 9 '15 at 23:10
  • 1
    Why aren't you using queries to aggregate your data? – Stephen Sep 9 '15 at 23:10
  • 1
    This is a financial complex calculation. It is hard to do the same calculation in the database. I created one CLR library and hosted in the database to calculate for each security individually. But when it comes to calculate for a group of securities (porftolio) with dynamic grouping with as many levels as the users want. This is much more complex – Felipe Almeida Sep 9 '15 at 23:16
  • 1
    It still sounds like you're better off writing a query to do the job than to try and do it manually. Databases are designed to transform and manipulate data. – Stephen Sep 9 '15 at 23:21
  • I found this link memory mapped files and this framework TeaFiles that feels like solves the problem I am having. – Felipe Almeida Sep 9 '15 at 23:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.