First of all, beware Brooks's Law: adding manpower to a late software project makes it later. This is just a rule of thumb but does tend to be true, so if you only throw more people at a project in order to speed it up, without being able to divide the work differently, you may actually cause the opposite to happen.
Basically, if the teams are to be able to work without interference and without much coordination between them, you have to be able to somehow split the work by scope. Otherwise, despite the theoretical splitting into two teams, they would in practice still be working as a single large team with all the issues that this brings. Trying to work on a single backlog and just assigning tasks to one team or another will either introduce an amount of communication overhead that will eat up all your increased capacity, or even worse, you avoid the communication, which will cause the teams to step on each others' toes and cause lots of unnecessary work, conflicts and reduced morale.
So, I'm afraid you need to find some line of division. Maybe it will not be as clearly cut as backend vs UI. Maybe one team can concentrate more on the algorithms and the other on everything else? Maybe there are some components within the application which could be developed in a more or less independent fashion? Maybe some of the work is related to performance tuning and is localized so that it does not affect the whole application and can be done independently? Perhaps there are some high-level mechanisms which make use of more low-level components? Then one team could program against the interface of the low-level components and the other team could implement them. Once the interfaces are agreed upon, such teams could work in a largely independent manner.
Unfortunately it's hard to suggest anything more detailed without knowing the domain of your application and its structure. For example, you suggest that backend vs UI is not an option but you don't explain why.
EDIT: If the work consists of customizations which are mostly independent from each other and which do not modify the base product (which could affect the other team's work), I think you could try working as two separate teams with separate backlogs by splitting the work based on the customer.
I assume there are several customers and each of them orders many customizations over time. Since it is known in advance who ordered which customizations, it should be easy to tell into which backlog to place each item.
Caveat: it's still really just one product, so you should make sure the teams do know what the other one is doing in order to avoid duplicating work and to exchange knowledge. Attending sprint demos should probably be enough, and make sure the two teams' leaders do talk to each other more frequently.