For example are there any specific tests I can run?
The design of an ACID database is significantly different from that of a BASE database. ACID gives you assurances that a transaction will be complete. Distributing an ACID database adds significant complexity. BASE and other non-ACID databases are simpler to build and test.
Concepts to test (choose according to ACID / BASE design):
- Is is possible to see part of a transaction change? (Depends on isolation level.)
- Do changes appear in order?
- Is the database view consistent with the effective isolation level? (Effective isolation level may not be the requested level.)
- Do committed transaction survive a restart?
- Do committed transactions survive a database crash?
- Can a transaction be rolled back?
- Is the appropriate data returned after a rollback?
- Do uncommitted transactions survive a database crash or restart? (Fail case)
- Can a record with an uncommitted change be updated by anoouther transaction?
- Are conflicting changes handled appropriately?
- Are results appropriate is changes have not converged?
- What happens if one instance of a distributed database instance shuts down (crash or requested)?
- What happens when a distributed database instance rejoins the cluster?
- What happens if a distributed database instance never rejoins the cluster?
Your expectations are for these concepts will drive the type of database you build.