You will probably want to make several measurements here, because you'll want to understand how the system works both including and excluding the outliers. There are many possible solutions for exploring the data with and without the outliers.
Here are a few starting points for measurements I would be interested in:
You can try choosing a few meaningful percentiles, say 50, 90, and 99. For example, if you chose 90th percentile and that value ended up being 500ms, you would then be able to say: "90 percent of our latencies are below 500ms".
Taking a trimmed mean will exclude the outliers so they don't skew your average. For example, in an extreme case where 90 percent of your requests are under 500ms, but you have one request on record that took three days, a trimmed mean would keep this extreme outlier from affecting your average.
The median is another common measurement for drawing insights from your data set. Wikipedia describes its behavior well:
The basic advantage of the median in describing data compared to the mean (often simply described as the "average") is that it is not skewed so much by extremely large or small values, and so it may give a better idea of a "typical" value.
The median is generally equivalent to a very aggressive (50%) trimmed mean and the 50th percentile value.
Finally, as Robert Harvey suggested in his comment, you may want to create a histogram and visually examine the data. This can provide inspiration as to where to focus your measuring efforts.