4

I'm working on a product that requires a large amount of interactions with different external data providers. These data providers uses a number of standards (primarily xml-based) which are all governed and agreed upon. Some of these standards aren't that great and can be open for interpretation.

Now, once in a while, some data source decides to differ just a little bit from these standards, which means that my application shows an error when the user tries to connect to them - however - a lot of competing products seems to be less concerned about correctness, and simply goes in to quirks-mode to get something working. From my customers' perspective, my product sucks since it can't do X but the competing product can, and when I try to contact the data source to have them fix their issues, I either don't receive an answer, I get to talk with some non-technical person who has no clue about what I'm talking about, or the time-to-deploy on the fix is simply really long.

The way I see it, I got 2 options:

  1. Stand firm and risk losing customers, or
  2. Do ad-hoc quirks and hacks for semi-compatible sources (which would then need to be supported)

I've been looking for historical precedence for how others have handled these problems (for example, the Firefox vs. IE battle, where some sites simply didn't work in FF because the site contained errors or non-standard constructs), but without much luck about finding anything of substance.

Regarding option #1 I think it's clear that if that is the route to go, I somehow have to make the user understand that the problem is on the side of the source, without getting too technical, which can get tricky.

Regarding option #2 I also have to consider the huge upkeep cost it's gonna be we I have 50+ differently hacked and quirked sources, not to mention the overhead in the of the weird bug-reports that are bounds to come. Also, by embracing quirks I would myself be part of the problem of everyone not following the standards.

Does anyone have any experience with a situation like this? What are the major things to consider here? Have I overlooked something?

[Update] After reading the comments here I've decided to give some more context. I'm working for a small team (3 devs, ~10 people total) managing a product in a large coorporation (1k+ employees). Our product is a spatial suite (think google maps) with a lot of domain-specific workflows build in. My main issues are with WFS/WMS servers using the OGC standards (http://www.opengeospatial.org/standards/wfs and http://www.opengeospatial.org/standards/wms respectively). These standards are a maze of optional parameters, open-for-abuse, and open-for-interpretation definitions specifically regarding default values (well, that's my opinion anyway). Now, if you look past the standards, in the world of spatial data you also have a lot of other issues like wrong projections, self-intersecting geometry, and oh so much more :)

As I stated, we are a rather small team, but since our corporate name is quite large, our customers expect us to handle everything in quite a professional manner - well, some customers anyway.

  • 3
    I would consider buildiong Anti-Corruption layer using XSLT as means to correct incompatible XML input (been there done that) – gnat Feb 10 '16 at 15:29
  • 1
    Thanks for the update; but, that's not quite the context you need to make this decision. But, that technical context is the sort of information your manager and their manager (and possibly a few of the VP's) probably need to consider within the larger context of the business's model and goals and the product's marketability. – svidgen Feb 10 '16 at 17:17
  • One of the primary aspects of creating a successful implementation of any standard is to be strict in the messages you send, but be as forgiving as you can in what you receive. – Bart van Ingen Schenau Feb 10 '16 at 17:38
  • @BartvanIngenSchenau Unless, by being forgiving in what you receive, you're potentially making assumptions about critical messages! – svidgen Feb 10 '16 at 17:40
7

This depends entirely on the role of your company and its software. In some models, you simply need to estimate the cost of making your integration layer flexible and adaptable and weigh it against the estimated cost of losing customer who either will not or cannot comply with the particular standards you've chosen to respect.

And, then there are cases wherein your business model may revolve more around adapting to customer needs.

A few examples to consider:

Browser vendors never fully implement the web standards. And, to the extent that a browser does implement the latest standard, it's also forgiving of bad markup. No one's going to use a browser that doesn't render 90% of the web because the vendor decided to "stand their ground."

The SQL Server development team authors and implements the T-SQL standard in their database engine. It's the paying customer's expectation that the database manages their data efficiently, safely, and predictably. It would be harmful to the customer and therefore to Microsoft to "forgive" mistakes in the customer's T-SQL and/or make assumptions about their intent!

Clever, a school system integration service provides a single API for curriculum publishers (the paying customers). But, it's their business model to write hundreds (or thousands?) of integrations specific to their customers' customers, who may or may not even be aware of any standards.


With respect to the update: The role and importance of the data you're managing is still unclear to P.SE. The criticality of your data could be anywhere from routing ambulances to pizza deliveries. We don't know. And that's OK -- let's keep it that way.

You have lots of technical context to bring to your manager. And, if the complexity of the software or the ecosystem is significant, which it appears to be, it's a complexity your upper-management team needs to understand. They need to answer these questions for you from a business perspective.

If you dare to hesitate in approaching them, thoughtfully consider how thrilled your President or CEO will be to learn that business decisions are being made by you, or svidgen ... or even Kilian Foth!

  • I've decided to award this with an accepted answer since it seems contain the broadest viewpoints, also, there's a concrete examples here I can use going forward :) Thanks. – cwap Feb 10 '16 at 19:48
2

You're taking yourself entirely too seriously.

The customer wants data aggregated from certain sources. You've agreed to make it happen in exchange for money. Therefore, it's your obligation to deliver on that promise. The fact that some sources don't conform strictly to a protocol that you thought they did is of no relevance to the customer; indeed, they probably wouldn't even understand the difference, which is why they hired you: you're the expert.

Therefore, the thing to do is to extract what data you can from the sources in the way the customer most likely would like (in this case, it's easy: extract at least as much as your competitors do). The fact that this requires hacks, ugly code or breaking standards is irrelevant - you've already agreed to shield the customer from these complications by taking their money.

You're only justified in rejecting a task if quirks in the input would make the entire project so expensive that you can't turn a profit at the agreed price. Then it's time to renegotiate (although in the real world, you'd often have to swallow the cost if the customer is otherwise profitable). Otherwise, it's sell-your-idealistic-soul-for money time.

  • 1
    Possibly. But, how do we know that? He hasn't told us what product he's building ... Suppose each new, unique or non-standard integration increases the number of assumptions made about data that his customers depend on? ... Suppose he's some rogue SQL Server developer who figured he'd ask P.SE about standards compliance instead of his boss? Uber-unlikely, I know, but, how can we assume he's taking his company's software too seriously? And why is it more serious to lean one way versus the other? If his business isn't "serious" ... isn't it just about weighing the costs? – svidgen Feb 10 '16 at 16:01
  • I agree I left out a lot of context. I did that deliberately, in the hopes that somewhere there had been done some sort of pros n cons analysis I could stay objective to. I can tell you though that my own opinion is tilting towards Kilian's but that some developers on my team seem to differ. It really is a 'no single solution'-problem. – cwap Feb 10 '16 at 16:05
  • @cwap Sure. And honestly, I'm not sure this is even the right forum for the question. It would probably belong on Business.SE if such a thing existed. That aside, if you want to contextualize your question more, it might make an answer like this more on-target and meaningful to the folks who visit here ... – svidgen Feb 10 '16 at 16:11
  • Yeah, I wasn't sure either and after reading most of the 'what belongs on programmers.SE'-posts I still think it's somewhat of a grey zone. I was also thinking about some business SE, but honestly this question seems to fit better in a Product Owner SE. Anyways, I will update the OP with some context. – cwap Feb 10 '16 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.