To answer your question about extant research
But has anything been written or researched on recognizing the point where striving for code brevity stops being useful and becomes a barrier to comprehension?
Yes, there has been work in this area.
To get an understanding of this stuff, you have to find a way to compute a metric so that comparisons can be made on a quantitative basis (rather than just performing the comparison based on wit and intuition, as the other answers do). One potential metric that has been looked at is
Cyclomatic Complexity ÷ Source Lines of Code (SLOC)
In your code example, this ratio is very high, because everything has been compressed onto one line.
The SATC has found the most effective evaluation is a combination of size and [Cyclomatic] complexity. The modules with both a high complexity and a large size tend to have the lowest reliability. Modules with low size and high complexity are also a reliability risk because they tend to be very terse code, which is difficult to change or modify.
Here are a few references if you are interested:
McCabe, T. and A. Watson (1994), Software Complexity (CrossTalk: The Journal of Defense Software Engineering).
Watson, A. H., & McCabe, T. J. (1996). Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric (NIST Special Publication 500-235). Retrieved May 14, 2011, from McCabe Software web site: http://www.mccabe.com/pdf/mccabe-nist235r.pdf
Rosenberg, L., Hammer, T., Shaw, J. (1998). Software Metrics and Reliability (Proceedings of IEEE International Symposium on Software Reliability Engineering). Retrieved May 14, 2011, from Penn State University web site: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.4041&rep=rep1&type=pdf
My opinion and solution
Personally, I have never valued brevity, only readability. Sometimes brevity helps readibility, sometimes it does not. What is more important is that you are writing Really Obvious Code(ROC) instead of Write-Only Code (WOC).
Just for fun, here's how I would write it, and ask members of my team to write it:
if ((costIn <= 0) || (costOut <= 0)) return 0;
decimal changeAmount = costOut - costIn;
decimal changePercent = changeAmount / costOut * 100;
Note also the introduction of the working variables has the happy side effect of triggering fixed-point arithmetic instead of integer arithmetic, so the need for all those casts to
decimal is eliminated.