0

EDIT: The project is about creating an algorithm for Procedural Content Generation for a videogame. The actual game is secondary to the algorithm. It must be something basic to test the Procedural Content Generation. I'm approaching this as a single developer.

Thanks to @ThomasOwens I realize this question is more about how to work combining CRISP (the datamining process model that will help me create the algorithm) with a project management methodology (preferably something agile such as XP).


I'm working on a school project involving an interactive genetic algorithm where the program must create some content in real time based on the user's preferences. Near the end of the project, in order to validate the algorithm and creating the necessary reports, it must be tested by actual users.

My tutor wants me to use CRISP-DM as the methodology for the project, and I'm wondering in which phase will the user tests go. Please don't advice me to change methodology. And also, my tutor isn't available this week, but I want to make some advances in the planning.

So my question is: Regarding the CRISP-DM process model, which phase and tasks relate to User Acceptance Testing?


You can skip the following paragraphs if you have the answer. I'll just try to explain my confusion a little better. The main problem here is my lack of knowledge on CRISP-DM and there's a liiiitle bit of a language barrier. To my udertanding the User Testing would be on the Evaluation phase, but I just want to be 100% sure, so I'm seeking expert advice.

Yes, my project doesn't involve reading data from a database, but it's a data mining problem because my algorithm will have to extract information from some data and analyze it and transform it into something else. This is how I've related the CRISP phases to my project so far:

  • Business Understanding will be all about defining a problem that would benefit on the user for finding it's solutions.
  • Data Understanding and Data Preparation will be merged on a single phase that involves figuring out how the data will be treated.
  • Now, on the Modeling phase, there's the Generate test design task, and this test is performed on the Assess model model task. But this tests don't involve User Acceptance Testing, only Unit Testing or Functionality Testing. I'm right here, right?
  • That leaves me with the Evaluation phase, in which it's task Evaluate results says:

This step assesses the degree to which the model meets the business objectives, and seeks to determine if there is some business reason why this model is deficient. Another option is to test the model(s) on test applications in the real application, if time and budget constraints permit.

Is this where it goes? Should I adapt this task to my User Tests? Or would it be better to add another task before this one that specifically refers to executing the user tests?

  • I hadn't heard of it until now, but CRISP-DM is not a methodology for executing a project, but a set of common steps to consider when performing data mining activities. What, exactly, are you trying to test? Are you trying to test your software system? Or are you testing that your models are correct and calibrated? – Thomas Owens Mar 29 '18 at 14:25
  • So I guess I don't understand your question. The Evaluation phase of CRISP-DM is explicitly for ensuring that you have high quality models and ensure they conduct the business objectives. I guess I'm confused by the notion of "user tests". Are users using your model directly? Or is your model being embedded into a larger software system as a component? This is why CRISP-DM isn't a project management methodology - it's built specifically for data mining parts of projects, not actually delivering a full software system to users. – Thomas Owens Mar 29 '18 at 15:24
  • Thanks, sorry for deleting the previous comment. I don't want to test the correctness and calibration of the models on this user tests. I want the users to evaluate the software, so I can evaluate the technique that creates the models. – Adocad Mar 29 '18 at 15:28
  • My question stands. CRISP-DM is focused only on the data mining portion of a project. The Evaluation phase is where you test your models. CRISP-DM really doesn't have the considerations built-in for where to perform user-based testing of a software system that uses the models. If the whole purpose of the project is simply to produce models, then Evaluation is where you would do whatever testing you want to do. If you're building a software system that uses your models that have reached the Deployment phase, you project management methodology will decide how you perform user testing. – Thomas Owens Mar 29 '18 at 15:32
  • Well, first of all, it's a video game concept. The purpose of the software is to create content based on user preferences. So the models in this case are just the content that gets generated on each player run. Does this make sense? – Adocad Mar 29 '18 at 15:33
0

On project management and being a solo developer.

Most development methodologies are built to support a team working on a project, not with an individual developer in mind. There's quite a bit of information on Software Engineering Stack Exchange about working as a solo developer. There's a common theme found when talking about working alone - many of the more defined development methodologies come with various practices. These practices can be useful, even if you are working alone.

From a software development perspective, there are plenty of processes and frameworks to draw upon. As a whole, the lean and agile family of methods gives us practices around a highly iterative and incremental approach. Extreme Programming embraces ideas such as test-driven development, continuous integration, small releases, a sustainable pace. Scrum provides regular reflection and improvement and very strict timeboxing on events and activities. Kanban encourages visualization of work and managing flow. Lean Software Development teaches us about various types of waste and finding ways to reduce them, including limiting work-in-progress (partially done work).

Since your project is also about data mining, you can also look for good practices from processes from that field. CRISP-DM is one such process model that you can look at to help you identify and organize part of the work that needs to be done. However, do be aware that it may be built around considerations for a team and not an individual working alone, so some aspects of it may not be appropriate.

On User Acceptance Testing in CRISP-DM.

CRISP-DM doesn't have considerations for user acceptance testing. CRISP-DM contains an Evaluation phase that is used for evaluating data mining models that are to be used. This is the equivalent of unit testing. After you evaluate (read "unit test") your data mining models, you would integrate them into a larger software system, perform integration and system testing, and then submit for user acceptance testing.

In XP, your user acceptance testing strategies would be part of your release and iteration planning. In Scrum, you want your potentially shippable increment at the end of every Sprint to meet the needs of the user and be ready for deployment if it's deemed to be appropriate. The guidance from these project management methods would inform when and how you did the activities associated with the Evaluation and Deployment phases of CRISP-DM.

This is why you need to combine CRISP-DM with other, more suitable project management methods.

On combining CRISP-DM with other project management techniques and methods.

One of the activities in any systems development lifecycle is planning. When you plan, whether it's a more up-front project planning or planning for a particular iteration, you need to understand what work you need to do. The phases and steps from CRISP-DM can be used as an input into any kind of planning activity. This will help you to make sure that the things that you need to do to be successful at a data mining project are considered.

CRISP-DM is silent on a lot of other things that go into projects - how iterative and incremental vs plan driven you are, how often you deploy your product, how you keep track of work that you need to do or are currently doing, how to reflect on your work and improve your methods, how to keep your work visible to stakeholders, test methodologies and approaches. You'll have to turn to other sources for suggestions and practices for doing these things.

  • So, on every XP release or iteration, I could check which phase I'm on regarding CRISP-DM, and follow it's guidelines in order to write my user stories or tasks? I can picture this on CRISP's Modeling phase, where the techniques and tools are selected and the model is built and tested (unit testing). But on the Data Learning and Data Preparation I don't see it as much. In here I'm just figuring out the data and datasets to use, as well as the techniques... I don't see how it would work on the Evaluation phase either... since every user story usually involves coding and customer validation – Adocad Mar 29 '18 at 17:59
  • @Adocad Maybe. CRISP-DM is also iterative. Depending on what you're doing and how long activity takes, you may map one iteration of CRISP-DM to one iteration of XP. Or maybe they will be disjoint and one iteration of XP will be a portion of the phases of CRISP-DM. If you are using XP (or any of the agile methods), keep in mind that every iteration is supposed to add value to the stakeholders. – Thomas Owens Mar 29 '18 at 18:02
  • Yes, but I don't really see CRISP following XP's methodology. XP is all about building the software and continuous tests. Maybe I could go with CRISP overall and use XP only for programming the software? that part could fit on CRISP's Modeling phase. After that, CRISP's Evaluation test would be done with no user testing... and finally CRISP's Implementation phase could be done involving the actual user tests. How does that sound? – Adocad Mar 29 '18 at 18:09
  • @ThomasOwens Could you please answer this question: datascience.stackexchange.com/questions/33265/… Thanks – ebrahimi Jun 17 '18 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.