... as it seems sort of insecure without a secret/password involved
It's as secure as social logins. How does moving to a different Google account than the one used to sign up on StackExchange makes SE insecure?
What happens if the user switches phone numbers? Can they access their account or need to go through some kind of customer service?
It depends. This is a pure business decision. Some providers won't allow you. Not even the mighty WhatsApp allow it. Hinge either. At the moment. For whatever reason.
It doesn't mean that this kind of "migration" is not possible. Technically is, but the design must support it and we know what drives design.
How would you prove the new owner was tied to the old number?
Good question. It's very hard to prove. We never have the reassurance that the person on the other side is the owner of anything (account, phone, phone number, ...). Anybody can be impersonated.
However, we can mitigate the uncertainty. One way is requesting information about the owner that only the owner and you know1. Another way is asking about the activity up to a certain date.
Could someone else access the original account if they later obtained the original number?
It depends. Have you removed the account? Does the account expire after a certain period of inactivity? Have you communicated the problem to the provider of the app?
No? then yes, it's possible but it depends on the probability that someone else will get your phone number, install and run the app. All of this before you remove the account or before it expires.
Do you need to keep track of device information to identify
when a new device is authenticating with a phone number?
Tracking device information and user activity is a good feature to have because it gives you something to ask the user about, as I mentioned previously. Banks track expenses, activity and locations so they can notify if any expense or activity is detected out of the ordinary. Gmail so does when we access the account from different devices.
The bank is likely to make a call to inform you about suspicious credit card activity and Gmail will send an email informing you about the recent activity carried from a different device and will ask you for verification.
1: Years ago, we used to have secret questions and answers to recover accounts.