1

I stepped away from a project for a few weeks. Before leaving the organization was discussing an architectural pattern for our microservices. An instance typically serves two roles in our organization.

  1. It receives RESTful requests and process them accordingly
  2. It works as a batch processor, reading in large volumes of data and processing them.

Going forward we will probably want to separate batch processing from REST handling. The question of how best to separate these concerns was being discussed. One proposal we had was to configure our build deployment process so that an instance could either be deployed in web mode or data loading mode. I liked this approach in particular because load balancing would be facilitated with this approach as we could ramp up or ramp down the number of instances based on the amount of work out there for a context-specific task.

However, upon my return I found that a different paradigm was settled on: they want to create two additional source code repositories for a total of 3 projects:

  1. A common project that contains all Service and Repository level logic, in addition to domain objects.
  2. A web project that references common to handle REST traffic
  3. A dataloader project that references common to handle data-intensive tasks

I'm really not a fan of this approach. For one thing, all of our integration tests are now in the web project and are really testing logic in #1 (we can do an overhaul of our testing to favor integration/unit testing at the common level, but that will take time and sweat). The debugging/development cycle is really inconvenient as errors discovered in 2 or 3 require going back to 1 to make changes, updating the maven dependency, and returning to testing. Working with these included dependencies is really a pain in general.

It's workable, but more time-consuming. It really has me thinking about the concept of the developer experience. I don't think it gets talked about but I think it is just as important as the user experience. Especially since this is a project that has an aggressive development cycle. Development on the branch where this is implemented is truly cumbersome and... annoying!

Management's reasons for favoring this approach don't seem to fit the inconvenience either. Some arguments I have heard is that the codebase will be too large without this separation. I am also hearing that they may want to deploy one version of the common library for data and a different one for web (I simply can't imagine a valid use case for this request).

I am wondering if anyone else has exposure to this architectural style which, to me, seems like an anti-pattern. Am I prejudiced against a valid approach? I am going to meet with management soon to recommend returning to a single code base but organizing it as a multi-module project (3 build files, deployable based on context).

  • usually there are workaround to improve the dev experience. what language are you using? – Ewan Jul 9 '18 at 16:07
  • I deliberately left that out because I did not necessarily want to entertain vendor-specific responses as this is more of a general architectural conversation (I hesitated to even mention maven). With that being said, it's a Java/Spring maven project :-D – IcedDante Jul 9 '18 at 17:30
  • How tightly integrated are they? I'm assuming you are publishing the build to a repository so that Maven can pick up the changes, or control the versioning. It complicates the build/test cycle a bit. – Berin Loritsch Jul 9 '18 at 17:52
  • Very tight, considering it was one application just a few days ago. Essentially you have two bodies sharing the same heart and brain. – IcedDante Jul 9 '18 at 18:04
1

I can't presume to know enough about what you are dealing with, specifically. However, in software development there are trade-offs, and those trade-offs have to be considered with the benefits and risks associated with the choice. So I would recommend going to your meeting with an open mind, so you can understand what the end goal is. I have a feeling that the change is still in process.

Can there be a benefit to splitting data processing and web processing?

Sure. This really wasn't your question, since you weren't opposed to the split, but the way it was done.

Where should the boundary lines be drawn?

This question, and the fact it used to be one application a few days ago (per your comment), leads me to believe that this hasn't really been answered yet. In other words you have a work in progress.

In general:

  • Unit and Integration Tests should be executed as closely to the code under test as possible. It's better to catch it before you push your changes than to watch multiple projects go red because of an avoidable mistake.
  • Keep things as simple as possible.
  • Simplify after that.

Since this is a work in progress, there will be pieces that are in the Common library now that should be moved elsewhere. It doesn't mean that this is necessarily the wrong approach, but that there is still more work to be done.

Why the separate repositories?

There can be legitimate reasons, and those reasons are mostly business related. Deciding when to create a separate repository vs. a single repository with 3 projects is never as cut and dry as you (or I) want. Correct me if I'm wrong, but this seems to be the core of your heartburn.

What you lose

By going to separate repositories you lose:

  • Cohesive versioning of the three projects (maven modules)
  • Having all the source code needed for a product in one place
  • Having all the tests needed for the product in one place
  • Understanding the core dependencies

What you gain

By going to separate repositories you gain:

  • Looser versioning so that one product only depends on one version of common.
  • Ability to release updates to batch processing without affecting the perfectly functioning web API (and vice-versa)
  • Discovering integration headaches if common is intended to be shared beyond just the two services.

I think maven/gradle do a better job of dependency resolution after you split projects into repositories than Nuget (C#). That's a technology related reason though (Direct project references help develop new features cohesively, but the generated nuget package might not have the right dependencies declared).

There will be headaches

While I personally would have started with one repository and 3 projects (i.e. the maven modules) like you suggested, that would be for the initial work.

Separating logic that was once in one project is always a tricky thing no matter how carefully you crafted it. In well crafted systems, 90% of the work is easy, but the remainder has to do with finding all the little ways your app broke because the code no longer complies with the assumptions under which it was developed.

There's two philosophies with making changes like this:

  • Rip the band-aid off and deal with the mess.
  • Slowly peel back the band-aid a little at a time.

Sounds like your team opted for the first approach.

  • I think the core of what bothers me is that we're distributing a jar file that calls a repository later to access the database in two different projects. After some back and forth, I think we greed that our data layer's primary concern is doing ETL. For that reason we're going to have our "core" project keep it's HTTP functionality, and also allow it to process messages from, say, a RabbitMQ or other messaging service. Or data layer can then primarily be concerned with reading data, transforming it, and generating the previously mentioned messages. – IcedDante Jul 9 '18 at 21:03
1

I'm wary of your objections. Usually because I'm the one adding projects, separating stuff out into components adding build servers and octopus deployment etc and getting moaned at for 'making things hard to debug!'

Lets take your objections at face value though and assume that, yes its a very weird setup that adds a whole bunch of annoying admin tasks to the developers daily routine for no benefit.

I'm pretty sure we have all been there to one extent or another. for example I have to raise annoying tickets atm for other teams to change firewalls or install updates etc which is a pain for a pretty minor organisational benefit.

Sometimes you are even forced to write unit tests!! RIDICULOUS!!! MY CODE IS PERFECT!!

Developer Experience is a thing. If things are easy then development goes fast, if you have to wade through annoying procedure things slow down. It gets boring.

But the cool thing about being able to code is you can automate stuff. (Or download automations other people have written and install them). So I would roll with it, make templates for your tickets, automate your annoying build processes, hire testers to write the stupid tests and answer stack overflow questions when you are compiling.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.