If the problem is not the user / client's fault then your server should return a 5xx error. A 4xx error should only be returned if the >>request<< is incorrect in some way.
Depending on the situation, either 500 or 503 could be appropriate:
As far as a client is concerned the "external factors" are internal to your server / service. So 500 could be appropriate.
If the problem is likely to resolve itself (or be resolved) in a relatively short period of time, 503 could be appropriate.
Probably either of those is OK.
Another possibility is to return a non-standard 5xx code. Note that non-standard (i.e. not defined in the HTTP 1.1 spec) status codes are not wrong. There is ample precedent for doing this ... including the precedent of other RFCs defining extra codes; see the Wikipedia List of HTTP Status Codes for examples.
I noticed a comment suggesting this:
In that case I will go for 500, however your users might blame on you for the error.
Provided that you include details of the cause of the problem in the 500 response body (in an appropriate format!), that should not be a problem.
I'd output a proper page -- 200 code, informing them about the third-party call failure and will ask them to try it again later.
IMO, that is incorrect. A 200 code means that the request has succeeded ... which it patently hasn't. Assuming that this is a RESTful service, the client is going to use the status code to decide what to do next. It should not be a lie. If you want to say "try again later", you should send a 503 response, possibly with a Retry-After header.