0

I have a set of data about songs. Each entry is a line of text which includes the artist name, song title, and some extra text. Some entries are only "extra text". My goal is to resolve as many entries as possible to songs on Spotify using their web API.

My strategy so far has been to search for the entry via the API - if there are no results, apply a transformation such as "remove all text between ( )" and search again. I have a list of heuristics and I've had reasonable success with this but as the code gets more and more convoluted I keep thinking there must be a more generic and consistent way. I don't know where to look - any suggestions for what to try, topics to study, buzzwords to google?

1

I'm not sure what language you are using, but I would create a some objects/functions that can perform the transformation of the song data. I would pass these objects/functions to a transformation consumer. The consumer takes the lyrics,artists,etc. and loops over the transformation functions, each time passing in the lyrics,artist,etc., and then querying the api for a match with the transformed data. If their is a match then your done, otherwise the next transformation occurs and the next query occurs. Keep it modular.

| improve this answer | |
1

Well, your basic algorithm sounds to be like this (in C#, but that does not really matter):

bool DoFullSearch(List<Transformation> transformationList, SearchData originalData)
{
    foreach(var transformation in transformationList)
    {
        var transformedData = transformation.Apply(originalData);
        bool success=DoSingleSearch(transformedData);
        if(success)
            return true;
    }
    return false;
}

(The first transformation in the list maybe the "noop" transformation, just returning the original data unchanged).

IMHO that's not very convoluted - it is pretty straightforward. Of course, the key here is to provide a Transformation class with several subclasses, each one for each possible transformation, each one overriding the Apply method in a different manor.

| improve this answer | |
  • Right, that's exactly what I'm doing. It's convoluted because the transformations themselves are a bunch of hacks for this particular data set, so I'm looking for a new approach entirely. Seems like someone must have already put a lot of effort into how to extract data from a messy dataset and I'm hoping to benefit from their struggles - sorry for the vagueness of my question but I feel like there must be a better way. – Rob Lourens Nov 6 '13 at 4:57
  • 1
    The difficulty with messy datasets is just that - that the data is messy. Sometimes you just have to "brute force" your way through every pattern that the dataset has. The thing is that with something like this you might be able to get away with using a program to do 95% of the work and then manually change the rest of the records. – Stephen Nov 6 '13 at 5:01
  • Divide and conquer. Just apply the transformations that seem to work, remove the 'solved' records from the dataset, then look at the remaining data to get fresh ideas about more possible tranformations. – Jan Doggen Nov 6 '13 at 7:13
  • @RobLourens: if you would provide an example for the what you think "convoluted" part, we might be able to give you a better answer. Currently we don't have much more information from you than "My code is becoming a mess - what should I do?" – Doc Brown Nov 6 '13 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.