-2

I want to expose APIs in python for my workload automation software. I have a Job class as defined/outlined below. Instance of this job class represents a particular Job run. Job can have multiple checkpoints and each checkpoint can have multiple commands.

Job
 - JobName
 - [JobCheckpoint]
 - StartTime
 - EndTime
 - Status
 - ...

JobCheckpoint
 - JobCheckpointName
 - [JobCommand]
 - StartTime
 - EndTime
 - Status
 - ...

JobCommand
 - JobCommandName
 - [Command]
 - StartTime
 - EndTime
 - Status 
 - ...

I want user APIs for satisfying following use-cases:

  1. Ability to query Jobs given a criteria (e.g. return jobs that ran in a given duration, return jobs in failed state, return checkpoints/commands of a particular job and many more...).
  2. Ability to create and run jobs using Python API.
  3. Ability to take control actions like rerun already ran job, kill running job etc.

I am thinking of providing following method in my user API.

get_jobs(Filter) - This serves/solves use-case#1

Question 1:

I am not sure how best Filter class can be designed. The goal is to provide intuitive and powerful API's to users.

Question 2:

I am not sure how should I solve use-case #2 and #3 above. I have couple of solutions in my mind but I am not sure which one is best.

Solution#1. For creating Jobs, users can directly instantiate Object of class Job. I will provide methods in Job Model class itself for running and re-running Job. Users will create instance of Job class and call methods (run, rerun, kill etc) on Job object.

Solution#2. Along with get_job API, expose separate APIs for control actions like run and rerun in user interface.

get_job(Filter) 
run_job(Job)
rerun_job(Job)
...

What if in future I want to support many more control actions on Job object (for e.x. hold, resume etc.). If I go with solution#2, I will end up creating lots of methods in user API. If I go with solution#1, I am not sure if Job class can be called as model class.

I am not sure which of the approach mentioned above is better and why? Or there is some other way I should think of.

1

First, Python doesn't support generics or static return types. This is mostly due to the fact that you don't need them because Python is dynamically typed. You can't do List<Job>, or anything like that. Python's list can contain any type, including a Job object. If you want a particular list to only contain Job objects, then do that. There's no forcing mechanism, but if it's a list of Jobs, then don't add a bunch of ints or any other type to it and you'll be fine. This bothers a lot of Java/C++ purists (I used to be one of them), but it works just fine in practice.

Secondly, looks like you're falling victim to BDUF (Big Design Up Front). Overall, API design is subtle and requires a lot of design decisions. But who says you have to make them all before you start? Try writing a few unit tests and modules in a TDD style, and see if something doesn't crop up. This provides two advantages:

1) You will actually have an API that does something instead of an API on paper that does nothing

2) You might find that emergent design works better for your problem. Try using TDD. Example (using your proposed API):

def test_job_can_get_command_names():
    job_command = JobCommand('some_command')
    job_command_2 = JobCommand('some_other_command')
    job_checkpoint = JobCheckpoint(name='test', commands=[job_command, job_command_2])     
    job = Job(name='job1', job_checkpoint=job_checkpoint)
    cmd = job.job_checkpoint.job_commands[0].job_command_name
    cmd2 = job.job_checkpoint.job_commands[1].job_command_name
    assert cmd == 'some_command'
    assert cmd2 == 'some_other_command'

My word, that was painful and verbose. Do you really think anyone would want to use that? As a Python dev, I can tell you I wouldn't. If that's what the tutorial looked like for your API, I'd stop reading right then and there. In general, Python APIs are short and to the point. They rarely create a bunch of objects or anything like that. Using string instead of enums, and "non-stacking" objects are common. Having to create a JobCheckpoint and JobCommand object in order to create a Job are big no-nos. Python != Java. If I ever found myself writing a test like this I would immediately delete it and try again.

Keep it simple when you start (it should always be easy to access the job's underlying commands):

def test_job_can_get_commands():
    job = Job(name='job1', commands=['some_command', 'some_other_command'])
    cmd = job.commands[0]
    cmd2 = job.commands[1]
    # maybe overload __eq__ if you want an underlying JobCommand object, but it should be transparent to the user.
    assert cmd == 'some_command'
    assert cmd2 == 'some_other_command'

Forget all the crazy classes at first. Maybe you need them, and maybe you don't. Even if you do, they should likely be transparent to the user. Write some tests, make sure your API is actually usable. TDD will make sure your API is usable. If you find it hard to write small tests like this, guess what, your users will find it just as hard to use your API.

Keep it clean, and keep it simple. Build as you go: don't try to design it all up front. Try writing some simple filtering tests that tackle a simple use case, and go from there. Same for the other classes. One test at a time, one class at a time, and eventually, a design and API will emerge.

If you don't like the API that emerges (now that you can see the API flaws, because you have one to work with in the first place), the beauty of using a test suite and TDD is that you can just beat it into whatever API you want while maintaining correctness. This is because TDD puts you in a position where you can refactor until you get the usability and power that you need. If you make a bad factoring, you'll have a failing test, and can easily revert. TDD helps separate correctness and design so that you, as a developer, can tackle each concern individually.

HTH.

  • I agree to your point that "job = Job(name='job1', commands=['some_command', 'some_other_command'])" looks much cleaner. But I do have a concept of checkpoints. i.e. Job has checkpoints and checkpoints has commands. – Lokesh Agrawal Oct 15 '18 at 15:29
  • And there are tons of configurations at each of these levels. For example, there is a dependency at all these levels. Dependency in itself is a class. Users can specify something like do not run job if following dependencies are not met, or they can say do not run this checkpoint if following dependencies are not met, similarly for commands inside checkpoint. Though all of these are optional configurations. I am not sure how can I modify this "job = Job(name='job1', commands=['some_command', 'some_other_command'])" to handle this three level initialisation. – Lokesh Agrawal Oct 15 '18 at 15:30
  • @LokeshAgrawal If you have to use checkpoints, then use checkpoints. What I'm advocating is starting small/simple, and building as you go. It is hard to imagine you'll need check points out of the gate. Get a simple job runner going without checkpoints would be a logical first step in my eyes. It it doesn't work with them, it won't work with them. You can build it as complex or as simple as needed. Keep in mind it's best to hide things a user won't want to interact with (or the majority of users won't want to interact with). Design for your simple 95% use case and work out details as you go – Matt Messersmith Oct 15 '18 at 15:37
  • I completely agree to your point. But I am just curious to understand, if one really have such nesting and users already understand the concept of job, checkpoint and commands (coming from legacy perl based implementation of it) then how cleanly they can create a Job object in Python using Pythonic style. – Lokesh Agrawal Oct 15 '18 at 15:40
  • Use the simple constructor Job(name='name', commands=list_of_string_commands). Then you can have an instance method like set_checkpoint(command_num=0) or set_checkpoint(command=some_command_str), or you could alternatively have your constructor accept a list of string commands or a list of checkpoints. I don't know your main use case, but hiding as much class/object instantiation/creation is always a good thing in Python. Making it short, flexible, and to the point is always good in any language. Sane default checkpoints are good too (helps avoid instantiations except by power users) – Matt Messersmith Oct 15 '18 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.