Unit testing is only useful if it's testing correct implementation of said function, not correct usage, because a unit test can't say that a function is not going to be called from another thread in the rest of your codebase, just as it can't say that functions won't be called with parameters that violate their preconditions (and technically what you're trying to test is basically a violation of a precondition in usage, which is something you can't effectively test against because the test can't restrict how other places in the codebase use such functions; you can test if violations of preconditions result in approriate errors/exceptions, however).
This is an assertion case to me in the implementation of the relevant functions themselves as some others have pointed out, or even more sophisticated is to make sure the functions are thread-safe (though this is not always practical when working with some APIs).
Also just a side note but "main thread" != "UI thread" in all cases. Many GUI APIs are not thread-safe (and making a thread-safe GUI kit is damned hard), but that doesn't mean you have to invoke them from the same thread as the one which has the entry point for your application. That might be useful even in implementing your assertions inside the relevant UI functions to distinguish "UI thread" from "main thread", like capturing the current thread ID when a window is created to compare against instead of from the application main entry point (that at least reduces the amount of assumptions/usage restrictions the implementation is applying to only what is truly relevant).
Thread safety was actually the "gotcha" tripping point in a former team of mine, and in our particular case I would have labelled it the most counter-productive "micro-optimization" of them all of a kind that incurred more maintenance costs than even handwritten assembly. We had rather comprehensive code coverage in our unit tests, along with rather sophisticated integration tests, only to encounter deadlocks and race conditions in the software that eluded our tests. And that was because the developers haphazardly multithreaded code without being aware of every single side effect that could possibly occur in the chain of functions calls that would result from their own, with a rather naive idea that they could fix such bugs in hindsight by just throwing locks around left and right, and perhaps even getting a false sense of confidence from their test coverage*.
I was skewed in the opposite direction as an old school type that distrusted multithreading, was a real latecomer to embracing it, and thought correctness beats performance to the point of rarely ever getting use out of all these cores we have now, until I discovered things like pure functions and immutable designs and persistent data structures which finally allowed me to fully utilize that hardware without a worry in the world about race conditions and deadlocks. I must admit that all the way up until 2010 or so, I hated multithreading with a passion except for a few parallel loops here and there in areas that are trivial to reason about thread-safety, and favored much more sequential code for the design of products given my grief with multithreading in former teams.
To me that way of multithreading first and fix bugs later is a terrible strategy to multithreading to the point of almost making me hate multithreading initially; you either make sure your designs are rock-solid thread-safe and that their implementations only use functions with similar guarantees (ex: pure functions), or you avoid multithreading. That might come across a bit dogmatic but it beats discovering (or worse, not discovering) difficult-to-reproduce issues in hindsight which elude tests. There's no point optimizing a rocket engine if that's going to result in making it prone to unexpectedly explode out of the blue halfway along its journey to out of space.
If you inevitably have to work with code which is not thread safe, then I don't see that as an issue to solve with unit/integration testing so much. Ideal would be to restrict access. If your GUI code is decoupled from business logic, then you might be able to enforce a design which restricts access to such calls from anything other than the thread/object which creates it*. That's far mode ideal to me is to make it impossible for other threads to call those functions than to try to make sure they don't.
- Yes, I realize that there's always ways around whatever design restrictions you enforce typically where the compiler can't protect you. I'm just speaking practically; if you can abstract "GUI Thread" object or whatever, then it might be the only one handed a parameter to the GUI objects/functions, and you might be able to restrict that object from having access to other threads. Of course it might be able to bypass and dig deep and work its way around such hoops to pass said GUI functions/objects to other threads to invoke, but at least there's a barrier there, and you can call anyone who does that an "idiot", and not be in the wrong, at least, for clearly bypassing and seeking loopholes for what the design obviously was attempting to restrict. :-D That's actually a very practical litmus test to me is like how confidently you can call someone an "idiot" in misusing a design, without the risk of you actually being the idiot for designing something prone to misuse.