2

I'm working on web application, which gets Wikipedia data from Wikidata (such as description, image, Wikipedia link) and shows them on the page.

I see 2 ways of making this:

  1. Upload requested data from Wikidata to my database and then get them from it. In this case my application will be working when Wikidata is not available (it happens sometime).

  2. Make Wikidata request every time to snow view. In this case I only need to keep item's code (for example Q1637368)) in database. In this case data is always uptodate. Otherwise I need sometimes to run some process for synchronization.

And not sure what is better for performance.

Question is: which way should I choose?

2
  • "And not sure what is better for performance." Depends on how fast and reliable Wikidata is, does it? Commented Nov 24, 2016 at 15:55
  • @Trilarion, depends also on database, difference that Wiki isn't under my control.
    – Alexan
    Commented Nov 24, 2016 at 17:17

1 Answer 1

3

Well right off the bat, we can't possibly tell you which will be more performant because we have no idea how fast your database querying for staged data is, versus how fast the Wikidata API will give you data.

In order to decide, you'll have to first decide if you app needs to work if Wikidata is offline. If so, then you'll HAVE to store the wikidata per method 1.

Lets assume you do decide that you need to show something even if Wikidata is down.

You'll need to set up your database to store user requests, along with the method that you're using to show the data (your copy or fresh Wikidata call) and the processing time from start of the request to end of request. You'll need to implement something like a random selection of which method to use (I'd go 50/50 with something like an even/odd choice on the current timestamp to decide which to use).

After having that in place for a while, you'll learn exactly what the speed difference is between the two.

Next, you could decide on a strategy like "Store my copy of the data for 24 hours before purging" (where if you don't have a < 24 hour old version of the data, you pull it fresh from Wikidata then store it in your database then return that to the user). Other strategies might include performance enhancers like having the copy of the data written to the database asynchronously in that scenario, so that your users aren't waiting on your database Write action before seeing their query results.

Essentially, do it both ways (depending on your requirements) and measure it!

2
  • 1
    so, I should choose based on performance measurements, right?
    – Alexan
    Commented Nov 23, 2016 at 19:34
  • 1
    It's hard to imagine your DB being slower, but worth some testing. A other advantage to caching recent results is you are much kinder to Wikidata's servers. +1 for the "refresh based on age" idea.
    – user949300
    Commented Nov 24, 2016 at 2:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.