I am currently working on a C# Azure Function that is triggered by an HTTP request. The function's purpose is to extract raw data from Azure Table Storage and output it to the user in the form of a large JSON object. However, I have observed that as the amount of data I am processing grows, it takes an increasingly long time for the API to complete the extraction process.

To provide some context, I have IoT sensors that send messages to my storage once every minute. When I attempt to extract data spanning multiple days, I have noticed significant performance issues compared to extracting data directly from Azure Table Storage.

I am now seeking input from others who have encountered similar issues. Specifically, I am wondering whether using a List object could be contributing to the slowdown. I have read that List objects can be resource-intensive compared to simple preset arrays.

However, I am hesitant to switch to using an array, since I do not know the size I will need to allocate in advance of downloading the data.

Any insights or advice would be greatly appreciated. Thank you in advance for your help!

This an example of my code:

public static List<AzureTableSensorPlotData> Read_Azure_IoT_Device_For_App_Graph_View(bool average_data,string tableid, string partitionKey_start, string partitionKey_end, string rowKey_start, string rowKey_end, string id, ILogger log)
    CloudStorageAccount storageAccount;
    storageAccount = CloudStorageAccount.Parse(Environment.GetEnvironmentVariable("AzureTableAccount"));
    CloudTableClient tableClient = storageAccount.CreateCloudTableClient(new TableClientConfiguration());
    // Create a table client for interacting with the table service 
    CloudTable table = tableClient.GetTableReference(tableid);
    TableOperation retrieveOperation = TableOperation.Retrieve<IoTDeviceEntity>(partitionKey_start, rowKey_start);
    string PartitionKey_filter1 = TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.GreaterThanOrEqual, partitionKey_start);
    string PartitionKey_filter2 = TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.LessThanOrEqual, partitionKey_end);
    string PartitionKey_combinedFilter = TableQuery.CombineFilters(PartitionKey_filter1, TableOperators.And, PartitionKey_filter1);
    string RowKey_filter1 = TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.GreaterThanOrEqual, rowKey_start);
    string RowKey_filter2 = TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.LessThanOrEqual, rowKey_end);
    string RowKey_combinedFilter = TableQuery.CombineFilters(RowKey_filter1, TableOperators.And, RowKey_filter2);
    string total_combinedFilter = TableQuery.CombineFilters(PartitionKey_combinedFilter, TableOperators.And, RowKey_combinedFilter);
    log.LogWarning("Azure Table Query:" + total_combinedFilter);
                    TableQuery<IoTDeviceEntity> myquery = new TableQuery<IoTDeviceEntity>().Where(total_combinedFilter);

    //List<IoTDeviceEntity> Read_device_data = new List<IoTDeviceEntity>();
    List<AzureTableSensorPlotData> Read_device_data_plot_data = new List<AzureTableSensorPlotData>();
    foreach (IoTDeviceEntity item in table.ExecuteQuery<IoTDeviceEntity>(myquery))
        AzureTableSensorPlotData temp_data = new AzureTableSensorPlotData();
          temp_data.Timestamp = item.Timestamp;
          temp_data.iot_battery = item.iot_battery;
          temp_data.iot_signal = item.iot_signal;

    return Read_device_data_plot_data;

List<AzureTableSensorPlotData> data_device = AzureTable.Read_Azure_IoT_Device_For_App_Graph_View(average_data, "AzureDevice" + device_plot, year_month_start, year_month_end, End_ticks, Start_ticks, id, log);
var result = data_device.OrderByDescending(d => d.Timestamp);
jsonToReturn = JsonConvert.SerializeObject(result);

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();                      
            return id != null
                ? (ActionResult)new OkObjectResult(jsonToReturn)
                : new BadRequestObjectResult("Please pass a name on the query string or in the request body")

I am downloading about 20-30K rows of sensor data. Each Day will have 1 row for each min

  • It sounds like you need a collection-object of some sort. There're lots of options that have various performance-characteristics for different things such as adding new elements, seeking elements, space-efficiency, etc. Often, just randomly picking a C#-List<> can be sufficient if the use-case isn't performance-sensitive, but it sounds like yours is. That said, a better choice might depend on exactly what you need from the collection -- for example, a linked-list would tend to be efficient to add to, though it can have trade-offs such as no default-indexing and higher memory-usage.
    – Nat
    Mar 22, 2023 at 14:44
  • 2
    "wondering whether using a List object could be contributing to the slowdown" - I'm inclined to think that's not the root cause of the issue. Can you provide more detail on how you're consuming this data and what you're doing with it? "When I attempt to extract data spanning multiple days, I have noticed significant performance issues compared to extracting data directly from Azure Table Storage" - I didn't quite follow the "extract data spanning multiple days" part. Extract it from where? Is there some intermediate/auxiliary storage involved? Mar 22, 2023 at 15:36
  • last time i checked a c# list is an array underneath
    – Ewan
    Mar 22, 2023 at 17:58
  • 1
    If you get your data over http and write it to a database, worrying about what data structure you put it into in memory is a really weird place to start optimizing.
    – nvoigt
    Mar 22, 2023 at 20:52
  • Yes, it is performance sensitivity, as I could be having thousands of rows of data to collect and present to a JSON object, collection object. If I were to comment out adding my object to a list, e.g., Read_device_data_plot_data.Add(temp_data); the API call finishes very quickly, so that indicates to me the issue is with the List object. Please see the updated post, with my code example for your reference. Mar 23, 2023 at 15:20

3 Answers 3


For a function which involves

  • heavy data transfer over a network

  • external storage access to all the data processed

it is extremely unlikely switching between lists and arrays in C# will result in any measureable performance difference.

However, the question contains barely enough information to give you a more specific answer. I was actually inclined to vote for closing it ("needs more details"). Then I reframed it as

What is a good strategy to approach such performance problems?

The best advice I can give you is: measure the performance of the individual steps! As you wrote, your function has at least three different parts:

  • extract data from from the storage

  • process the data to create a JSON object

  • send data to the user.

So you could give the function a debugging parameter which allows you to run only the extraction, the extraction with processing to JSON (but without sending the result), or all three steps. By comparing the running times for all three variants for the same input, it should be easy to identify the slow section in the code.

Your real code will probably have more details, which will allow you to make a more finegrained analysis. Once you figured out the slow section, you can try out different implementations by yourself, or isolate the code and ask a question about it on Stackoverflow.

  • Please see my updated post with my updated code example. If I were to comment out the .add method, then extracting and reading the data from my database does not take much time. It seems like updating the table multiple times is slowing the overall performance Mar 23, 2023 at 15:24
  • Is there anything that can be Visual Studio, that can help find the cause of the problem? Mar 23, 2023 at 15:31
  • @user8400863: if you comment out the "add" method, the list Read_device_data_plot_data will be empty. What you did not show us is what happens with that list afterwards - and which may cause the performance issues, ot not - I don't have a crystal ball. Node also coding and debugging help is off-topic for this site.
    – Doc Brown
    Mar 23, 2023 at 19:43
  • Sorry I have updated my original post but was not sure should i post my code due to restrictions, but i have shown all the key steps i am doing Mar 24, 2023 at 18:54
  • @user8400863: and- what did your performance measurements tell you? Did you measure Read_Azure_IoT_Device_For_App_Graph_View, data_device.OrderByDescending, JsonConvert.SerializeObject(result); and StreamReader(req.Body).ReadToEndAsync();individually, as I told you? BTW, when you post code, please care for indentation and formatting.
    – Doc Brown
    Mar 24, 2023 at 19:38

Difficult to say without full investigation, but I would hazard a guess that the type of c# object you are casting your data to is not significant.

My number one thing to check would be the json serialisation. If you are repeating the field names for every object that's going to add a big chunk of extra data to download. You might be better off with some custom serialisation of the native return type.

Second would be streaming, are you collecting everything together into one big in memory object, then writing it all to one big string, then sending that big string over http? OR are you reading chunks of data from table storage and writing each chunk to a http stream as it comes in?


I have read that List objects can be resource-intensive compared to simple preset arrays

I wanted to address this, because it illustrates the difficulty understanding the differences in scales a computer operates at. I would refer to this table of latencies for an approximate overview of how long things take.

The main difference between an array and a list is that a list require an extra object to be allocated, and further allocations may be needed unless you set a correct initial capacity. I do not have the latency number for creating a small object, but in principle it involves incrementing a counter, setting some memory, and running the constructor. I would guess this is on a similar order as a main memory access, i.e. hundreds of nano seconds. Note that it might be much more if memory is not immediately available.

An extra hundred nano seconds may be significant in a tight loop, but it is not significant in human timescales. As a point of comparison we can refer to the table:

Main memory reference 100 ns

Round trip within same datacenter 500,000 ns or 500 us

Read 1 MB sequentially from disk 20,000,000 ns or 20,000 us or 20 ms

Send packet CA->Netherlands->CA 150,000,000 ns or 150,000 us or 150 ms

A clock cycle is usually about 0.5-0.2ns. Note that the difference between main memory access and a disk read is about 5 order of manitude! It is difficult to comprehend just how fast a single clock cycle in a modern computer is.

I have noticed significant performance issues compared to extracting data directly from Azure Table Storage.

The incredible thing is that we do not have to guess. There are tools that can tell you what the performance issues are. I'm not familiar with Azure, so I do not know what specific tools are available. But a simple stopwatch and some logging may give some insight, in lack of a real profiling tool.

  • I am experiencing an issue, as my data grows downloading data for the past 14 days or so could take about 40seconds or so Mar 23, 2023 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.