Measuring the success of coding dojos

Question

I'm interested in starting coding dojos at the company I work at (during or after working hours). I think it will help to spread knowledge on how to write clean, concise and consistent code. I've been part of coding dojos in the past and it's really helped me be a better software engineer. I especially think that this could be useful for engineers who are switching to a new language e.g. Java to Golang.

What I'm not sure of is how to measure its success or failure? Is it worth taking 30-60 minutes of the company's time per week to allow their engineers to hone their skills?

I could send out a simple questionnaire after every session, and try to measure how much people have learned. Or I can try and measure the quality of code in PRs? That would be really interesting to do!

Has anyone done such a thing? If so what did you do?

Answer 1

If you work out how to measure whether someone is a good programmer or not then you have solved an age old problem and will be hailed as a genius by project managers the world over.

Instead measure whether people are enjoying the dojos. I suggest beer and pizza rather than in depth critiques of naming conventions.

Answer 2

As part social scientist by training, I'm partial to the questionnaire idea. If your organization is large enough and management is willing to support your efforts, it may be possible to do a double blind experiment to see if a particular dojo helped the participants in that particular subject matter (As Ewan mentioned, we currently lack an objective method to determine how good someone is at programming overall).

1. Determine the topic of your first dojo.
2. Create a brief survey/quiz on the key concepts. Ideally the questions will be verifiable enough that the answers can be easily determined to be correct or incorrect, but difficult enough that most participants will not get all the answers correct without the dojo.
3. Get as large of a group of volunteers/conscripts as possible to give yourself a lower margin of error.
4. Randomly assign half to the control group, which will not initially attend the dojo. Give this group the survey right away. Allow anonymous results to be submitted, all you need to keep track of is if the participant is in the control or not (Two different Google Forms etc. with the same content might be an easy way to set this up).
5. Give the first dojo lesson to the other half. Either at the end of the session or the next day, give them the survey.

This will give you the data to do a statistical analysis on the two groups. If the group that attended the dojo saw a statistically significant improvement over the control, we can conclude that the developers at minimum became more competent in the subject area in the short term.

As a bonus, if you would like to see if this information was retained longer term, create another quiz about the same topics, yet is different enough that remembering the answers to the first will not directly assist them. A few months later, give the new quiz to both groups. You can then compare the dojo group with the control to see if the results are still significant.

This would certainly be overkill to do for every dojo, but if it turns out that you can prove that one dojo has a positive impact on one area of programming, we can assume that others will as well.

Answer 3

Coding dojos should have "lightweight learning" for who is learning there. I believe PR in this context are not recommended once the intention of the dojo is not get fix or improve code. I had good experiences in rotative dojos once there was a laptop and the participants were intercalating between them, we used timer to give each participant 6 minutes to create a unit test in that case, given 30 minutes total + presentation and briefing of 15 minutes. A feedback sheet is a good ideia.

Answer 4

It's not how you measure success, but when that's important. Measurements made too early lead to premature judgment and false results. To know if something is working or not you must operate in larger time intervals or coordinate your measurements with others.

IOI (International Olympiad in Informatics) are organized every year. Just by looking at table (2019) I can say China is doing better than USA and Poland is doing better than Iran. But only with repetitive winning I can infer some underlying structure, a series "coding dojos" behind Poland being better than USA.

God knows I don't know what I'm talking about, but if it helps:

Do tests once a month and coding dojos once a week for a year and track the bug reports (there should be less bugs) and company's financial (there should be more money) returns and once you have enough data you make a judgment. If you conclude it does have a success on your business -- continue, if you conclude it does not -- stop.