1

I have two audio clips:

  1. Source of truth
  2. Recording of user

I want to compare the two, testing if they are similar enough, removing accents, etc. Any idea how I could do this on Android?

To add more detail, I want to record the user reciting some Arabic and then compare it to the correct pronunciation. The idea would be test their pronunciation and give them feedback on where they need to improve. I'm thinking of doing this offline (vs online) for faster response times to the user.

7
  • Migrate this to stack overflow.
    – Jandroid
    Jan 19, 2021 at 3:53
  • 1
    Are you asking for a workflow to leverage some key components, a specific signal related algorithm, or an implementation?
    – lennon310
    Jan 19, 2021 at 15:47
  • @lennon310, Either an algorithm I could implement or an existing implementation I could use. I've worked on AI before, but never audio. Reason I mention offline is I don't want to use something like a Google API that requires uploading the recordings which would lead to significant latency.
    – moinudin
    Jan 19, 2021 at 18:39
  • I guess you may get more helpful comments/feedbacks from dsp.stackexchange.com
    – lennon310
    Jan 19, 2021 at 18:50
  • Oh, I wasn't aware of that site. Is it considered OK to repost there?
    – moinudin
    Jan 19, 2021 at 19:04

1 Answer 1

1
+150

The idea is to compare the spectrogram of the voices. I'm not sure if there are proper libraries that you can use directly, but the basic steps would be

Filtering the audios

Applying a band-pass filter (e.g., Butterworth) to remove the unnecessary frequencies (ref on voice frequency range) while keeping the main waveform.

Determine the background

This is useful when you compare the two clips to avoid comparing on noises. You can pick the intervals between the voices to represent the background noise.

Generate spectrogram

Generate the spectrogram (example implementation) from both filtered audio clips. Since spectrogram is the Fourier transform of the audio, with the selected time that represents the background, you get the frequency correspondingly in the spectrogram, and these are the regions you don't need to compare.

Blob detection

Run blob detection on both spectrograms based on brightness (representing frequency). So these are the "useful" voice signals you want to compare, and you can fill out the rest element with zeroes. Note if two audios have different speed, an extra normalization step is needed (to calibrate the expansion on the blob "shape" on two audios), but this could be tricky to implement. So you may either need a ratio factor in the comparison next step, or make sure the user talks in the same speed with the sample.

Compare the filtered spectrogram

So this step would be a comparison between two 2D arrays with your predefined criteria. One typical metric you could use in this step is Cosine similarity. Any region in spectrogram that two clips have dramatic difference can be mapped back to the timestamp in original audio clips so you know which part the user needs improve.

A good algorithm is important, but I think how well and accurate it fits in your use case also depends on the SNR on the audio, how good the filter works, and normalization process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.