I'm looking for a way to store a large quantity of individual data values, with the following constraints.

• Assume that the types are bool, int32, double, decimal, string and blob (byte array).
• The type of each value will not be known at compile time (but the set of possible types is).
• The type of each value will be known at run time, so it doesn't need to know its own type.
• There are lots of them -- think millions at least.
• Values are collected into rows/bags of (say) 1-100 values and accessed by index (from a dictionary as it happens).
• Values are created and destroyed sometimes, read often and updated rarely.

Strategies I have so far are just these:

• Array of object, but would require value types to be boxed and that's a load on the GC that I would prefer to avoid.
• Serialise to a byte stream. String and blob would require an embedded length. Cost to read/update is high.

In C++ this could be an array of union of scalar and pointer types, but that strategy is unavailable (or at least highly unsafe) in .NET. This is a place where I really don't need a GC (object destruction is deterministic), so calling out to unmanaged code is a possibility.

The question is whether I've missed any viable strategies in the .NET world. Suggestions/comments welcome.

I'm looking for a way to store a large quantity of individual data values, with the following constraints.

• Assume that the types are bool, int32, double, decimal, string and blob (byte array).
• The type of each value will not be known at compile time (but the set of possible types is).
• The type of each value will be known at run time, so it doesn't need to know its own type.
• There are lots of them -- think millions at least.
• Values are collected into rows/bags of (say) 1-100 values and accessed by index (from a dictionary as it happens).
• Values are created and destroyed sometimes, read often and updated rarely.

Strategies I have so far are just these:

• Array of object, but would require value types to be boxed and that's a load on the GC that I would prefer to avoid.
• Serialise to a byte stream. String and blob would require an embedded length. Cost to read/update is high.

In C++ this could be an array of union of scalar and pointer types, but that strategy is unavailable (or at least highly unsafe) in .NET. This is a place where I really don't need a GC (object destruction is deterministic), so calling out to unmanaged code is a possibility.

The question is whether I've missed any viable strategies in the .NET world. Suggestions/comments welcome.

For my particular application it appears that creation/deletion happens in batches (rows/bags/sets) and individual values are not added or deleted. Think serialise/deserialise rather than random access. There are also set operations, so value-compare-equals is a common operation and boxing is bad for that.

I'm looking for a way to store a large quantity of individual data values, with the following constraints.

• Assume that the types are bool, int32, double, decimal, string and blob (byte array).
• The type of each value will not be known at compile time (but the set of possible types is).
• The type of each value will be known at run time, so it doesn't need to know its own type.
• There are lots of them -- think millions at least.
• Values are collected into rows/bags of (say) 1-100 values and accessed by index (from a dictionary as it happens).
• Values are created and destroyed sometimes, read often and updated rarely.

Strategies I have so far are just these:

• Array of object, but would require value types to be boxed and that's a load on the GC that I would prefer to avoid.
• Serialise to a byte stream. String and blob would require an embedded length. Cost to read/update is high.

In C++ this could be an array of union of scalar and pointer types, but that strategy is unavailable (or at least highly unsafe) in .NET. This is a place where I really don't need a GC (object destruction is deterministic), so calling out to unmanaged code is a possibility.

The question is whether I've missed any viable strategies in the .NET world. Suggestions/comments welcome.

I'm looking for a way to store a large quantity of individual data values, with the following constraints.

• Assume that the types are bool, int32, double, decimal, string and blob (byte array).
• The type of each value will not be known at compile time (but the set of possible types is).
• The type of each value will be known at run time, so it doesn't need to know its own type.
• There are lots of them -- think millions at least.
• Values are collected into rows/bags of (say) 1-100 values and accessed by index (from a dictionary as it happens).
• Values are created and destroyed sometimes, read often and updated rarely.

Strategies I have so far are just these:

• Array of object, but would require value types to be boxed and that's a load on the GC that I would prefer to avoid.
• Serialise to a byte stream. String and blob would require an embedded length. Cost to read/update is high.

In C++ this could be an array of union of scalar and pointer types, but that strategy is unavailable (or at least highly unsafe) in .NET. This is a place where I really don't need a GC (object destruction is deterministic), so calling out to unmanaged code is a possibility.

The question is whether I've missed any viable strategies in the .NET world. Suggestions/comments welcome.