There are basically two aspects of this. Smart pointers exist to define ownership of some resource. So the first question is, should your API be exposing the outside world directly to ownership issues?
That all depends on what your API is doing. If your API is allocating memory, it makes sense to wrap that in an explicit smart pointer. If your API is providing objects that internally use smart pointers to manage resources, that's a different matter. The outside world doesn't need to see the smart pointer itself.
The other issue is one of ABI. For DLL/SO-based libraries, putting any standard library type (or any type not in your library's control) into the API exposed by the library has substantial implications. Such a library can only be used by code using the exact same standard library implementation which was used to build the library. Standard library version changes can break things. Most standard library implementations will announce when an ABI break happens, but it still represents something not entirely under your control.
Coupled with this is the fact that smart pointer types tend to be header-only, so the various specific information about the type would be compiled into the DLL/SO as well as the program consuming the library. Slight differences in implementation can break things.
There's a school of thought that basically says that DLL/SO APIs should not expose any standard library type directly (with a few exceptions, like
initializer_list). Now this doesn't stop you from writing your own smart pointer types and using them in your APIs.