First, since we only care about the y-distance and will be drawing a horizontal line, we only need to think about the y-coordinates of the points and the y-coordinate that defines the line. The distance between a point and our line will be the absolute difference between the y-coordinate of the point and the y-coordinate that defines the line.
So, rephrasing the problem, we have a set of numbers, y_1 to y_n, and need a number, z, that minimizes the aggregate of the absolute differences between z and the points y_1 to y_n. Instead of minimizing the aggregate, we can just minimize the sum and get the correct result (aggregate = sum / number_of_points).
It turns out that it is the median that does this, not the mean.
For intuition, have points at y-coordinates 10, 10, 10 and 110. The median is 10, aggregate distance is (0+0+0+100)/4=25. The mean is 140/4=35, aggregate distance is (25+25+25+75)/4=37,5. In fact moving the line any distance, d, away from y-coordinate 10 towards 100 increased the distance to 3 points (with d) while only decreasing the distance to 1 point (with d) and hence increasing the aggregate.
(If we would take squares of distances then the mean would be the correct answer)