Svick's answer is good but I thought I would add a few extra points.
First off, the paragraph is flawed. Pointer types do not inherit from object. Values that are at compile time known to be interface types or type parameter types will, at runtime, be either invalid references or bona-fide instances of something that inherits from object, but it has always struct me as odd to say that these types "inherit" from object; inheritance is the property that members of the ancestor are members of the descendent, but you don't normally think of "ToString" as being a member of IEnumerable. You think of it as being a member of the thing that implements IEnumerable.
The paragraph is also flawed because this is the only place "primitive type" appears in the spec, and it appears without definition. It is therefore both unnecessary and confusing and should be removed.
I've wanted this paragraph to be fixed for a while. Next time I see Mads I'll remind him.
To address your specific question: svick is of course correct but it is helpful to see a specific example. When you say:
and you create, say, an array:
ColorfulInt x = new ColorFulInt;
Then the storage for those 100 ints and 100 Colors goes in the array itself. If ColorfulInt were instead a class then the array would contain 100 references to ColorfulInt, each one of which would have to be individually allocated. Individually allocating those hundred elements is much less efficient in both time and space than simply allocating the storage right in the array itself.