What my concern is someone can figure out the API, and start hitting
with different combination of OTPs for mobile number and gain the
access to an account
This is a frequent question related to the security of web applications. Once our APIs are public, they are exposed to all sort of malice.
Besides the https, which should be mandatory, here some measures you should consider too.
Setting a max number of request per second and source (remote address).
Let's say X Req/s per IP address.
Thresholds are commonly implemented in the API Gateway or in the Authentication server. Many API Managers provide control of thresholds out of the box.
The point is, the number of possible combinations of phone/otp and their respective permutations will be usually greater than the threshold, what reduce the possibilities of hitting a valid tuple or at least, it make it harder.
We can set endpoints with different thresholds. Usually, endpoints related to the security will have lower values than those related to the business.
Hitting the threshold causes ban. The ban last as long as you want (1, 5, 10 min, ...). If you like blacklists this is the right place to add one.
We often think that we should provide as many info as possible to the user when errors happen. That's ok when we speak about business rules, but it's not when we speak about security.
If the login process fails, a simple Invalid credentials should be enough. Don't tell to whoever is on the other side of the wire the causes of the error.
Making your security opaque to the externals reduces the attack surface.
I would encourage you to don't reinvent the wheel, overall with security. JWT.
It's a plus if you can force the expiration at will from the backend.
Https connections are encrypted. However, the query strings can be traced in log files once the message has been decrypted. So, doesn't matter if the query string values were encrypted twice. I would suggest sending POST request for authentication processes. Do it for any other request that may transport sensitive data.
As prevention to MITM, checking if the server certificate matches the server's domain is a plus.
Security is serious business. Keep yourself up-to-date and well documented. Here a good place to start working. OWASP - Categories.
Here some interesting projects: