There is an endless data stream of XML messages (and "heartbeats"), that I receive via a telnet connection and through a site-to-site VPN IPsec tunnel.

I'm still pondering. What is the best/most elegant solution to process the XML messages without losing any data, without redundance and with a (nearly) constant processing time? A never-ending process/script? Writing the stream in file(s) and processing it/them periodically step by step? Or something completely different?

The messages usually come every few seconds. Sometimes every second. Sometimes maybe every 10 seconds. It differs but not a lot. One XML message within the stream contains 45 rows. The messages should be stored afterwards.

Note: The concrete structure of the XML messages and the infrastructure of the participating systems are negligible in my opinion.

  • 1
    You might want to look at some XMPP implementations, because XMPP is basically the same: an endless XML stream. – Joachim Sauer Mar 21 '13 at 14:05
  • 1
    Sounds like the Observer/Listener pattern might be helpful. More information about your problem might help in suggesting a solution. How quickly/how often is the data coming in? How large are the xml messages? Do you need to store the message content somewhere? What do you do if there's an exception (the xml is incorrect)? – Asaf Mar 21 '13 at 14:08
  • @Asaf: I've edited my question to share you a few more information – derphil Mar 21 '13 at 15:10
  • @derphil Do you have any advice regarding telnet? Like, how to handle disconnections on the server or any surprise you've had that you weren't expecting? I'm about to do something similar and I've had some misconceptions (i.e. server dies and client still thinks it's connected, so I have to check for pings instead of for connection to reconnect). My issue right now wouldn't be parsing the XML, it would be more on the telnet side. – Rafael Merlin Mar 29 '19 at 22:36

Event stream based is your best bet! in C I will use Expat. In Java, I will use SAX parser.

  • 2
    Short answers like this are always better received if they are accompanied by a paragraph or two explaining why you are recommending this as a solution. – Robert Harvey Mar 21 '13 at 15:58
  • well the reason why is because both are stream based – Armando Mar 21 '13 at 21:49
  • 1
    That explains why you didn't pick something else that's not stream-based, but it doesn't really say why you are picking Expat specifically (what features or characteristics make it ideal), or why an Event-based stream is better in this scenario. – Robert Harvey Mar 21 '13 at 21:53
  • @Armando: Ok, I've took a look at SAX. So, just to ensure, that this is a good solution: Do you have any experience of using endless input streams (via telnet) with the the Java SAX parser? Or do you only suggest SAX because it's event based? – derphil Mar 22 '13 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.