Hypothesis Driven Development with example

My previous blog-post was an introduction to HDD (Hypothesis driven development). In this blog post I will take it a step further and give you an example how you can use it and how easy it can be and how it can change the way you work entirely to a new level.
You can read it [here].

Remember:
HDD is an experimental driven process, it’s main goal is to find out if a certain idea generates ROI and happy users. To get fast result you will also need fast support to evaluate your experiment, if you want to evaluate the idea in production I prefer a smooth decoupled architecture and a mature CI & CD organization. It’s not a must to, but it will really decrease development time and cycle time.
Sadly, I have no maturity model written in English yet. You can read one I did in Swedish on my LinkedIn posts [here]

Let’s start the story.

Team:  A cross-functional team with talent people regarding:
Design, UX, Coding, Testing and so on. And also including some market people and product owner.

At a meeting with the customer and team a team member (TM) got this cool idea regarding an e-commerce platform.

– What if the stock value changes in real time? Wouldn’t that be cool? Maybe it increases the conversion rate? The users might be stressed that this nice shoe will get out of stock?

A business person (BP) responds:
– What will it cost? Can you estimate?

– I don’t know, and I don’t. And I don’t know if my hypothesis even works. Can’t we skip this estimating and cost discussion and find out and then see if it’s worth implementing at all?  (TM)

– How? (BP)

– Let’s just do some A/B testing and fake a simulation of stock value changes and monitor if the user will buy the product more often or faster than today? If we get an 2% increased conversion rate it really worth implementing isn’t it?  (TM)

– Yes, you got a points there, this Sounds interesting. (BP)

Instead of creating User Stories the team now write Hypothesis stories.

Real-Time stock value changes
We Believe That
real-time stock values on the product page
Will Result in improved customer engagement and conversion
We Will Know We Have Succeeded When we see a 5% increased speed and about 2% conversion increase from the time the stock change in real-time till they press the add to cart button.

The meeting generate lots of hypothesis stories and the product owner can now prioritize them based on the ROI for the result outcome.

– A 2% increased conversion rate is really nice. Today we got like 5000$ each day, 2% will give us about 100$ more. In a month it’s around 3000$. If the experiment works and generate this value; You can spend around 30h of development time and we still get more money than not implement it at all. (BM)

– Yes you can say that, but let’s not convert it to time yet, let’s see if it’s worth implementing at all. Because if we spend around 10-30 hour for this feature and it will not increase conversion at all; we just wasted lots of money for developing a crappy feature. Let’s be smart and just spend time creating features that generate values in short and long runs instead. Assumptions is not a good thing. (TM)

– Ok, we can give it a try, because we have done so many bad assumptions we even can’t monitor at all. (BP)

The Product owner decided to activate the experiment and tells the team to do the fastest possible implementation to get information to see if this hypothesis will generate values at all.

The team gathered as usual at a stand-up and activated the work of the hypothesis.

– So what is the fastest possible solution to get rapid feedback regarding this? (TM)
– We can ask the customer?
– Nah, the customer might say, cool feature or – Do you want to stress us even more?
– Ok, let’s implement a test in our code and monitor the outcome?
– Sure, what’s the fastest way?
– This is part of the UI so let’s do it simple with some JavaScript. The only thing we need to understand is if it works. So there is no need to make it perfect. We do not need to add back-end functionality to get this information either. I can type a JavaScript that just randomly decrease the stock-values on some products. And then monitor with our application-insight code if they will press the button faster or not. It doesn’t matter if the stock value is a faked value for a while… If the value is 5 or 4 doesn’t matter that much.
– Nice, cool this will take us around 15 minutes or so? And thanks to our CI  & CD approach the feature will be out within 20-30 min or so and we can then start monitor it. If the result is good, then we implement it all the way with tests and quality in mind.
– Cool, this is nice, if the PO asked us to estimate it and add management work around it and do all the other stuff we do for a user story it would take longer cycle time than 30 minutes for just get started on this one. We aren’t wasting anything here, just getting knowledge and information if the idea is worth implementing or not. We will save around 15 minutes and also get the information if it’s even worth implementing. It’s much better than assume it does by implement it and hope it will generate values.

The team agreed.

Liza loved the idea regarding NoPSD so instead of spend time on working with PSD files or other tools for design she just opened the SASS file (css manipulation) and added three new classes with three different color. Green for normal stock values, Orange that indicate near out of stock and red that indicate very soon out of stock.

.stock-Normal
{
     color: #38893d;
}.stock-Warning

    color: #fe8f01;
}
.stock-Critical

    color: #be0000;
}

Thanks to nice SASS structures it just took her 2 minutes. When she pushed the code to the source control it automatically generated a new css file that got released into production, this because the team had nice CI & CD approach. (Why not release it? It will not hurt anything though nothing using the classes at the moment anyway.)

The front-end was built on AngularJs and the stock area was created as a web component and rest of the system is based on ASP.Net MVC and Web Api. Meanwhile Liza added the css John added a feature toggle in the code and took the html from the web component template and added it to the new feature for the hypothesis.

The feature toggle framework they use uses one configuration file to indicate if the feature is on and off.
They simple add it as an appSetting in web.config or appSettings.json (if using ASP .Net 1.0 Core).

“IRLStock” : “true”;

The team uses a framework that uses a C# class as a feature toggle so they easy can remove the class and get compilation error where the feature existed, so they do not forget to remove the code later on.

public class IRLStock {}

The code with razor syntax (ugly code? I know but much faster than create a new web-component to get the result from this experiment, it’s important to not waste unnecessary time for a more complicated solution, then you missed the whole point. This code will be deleted after the experiment anyway. For experiments it’s just waste to add sugar to it.):

@if(Feature<IRLStock).Is().Enabled)
{  
     < script >
… a script that just gets a random number between 2-6 seconds…
… this number just trigger a interval eg.
setInterval(changeStock, randomNumber);..
..
and set a new class to the element…
getElementById(“stockDemo”)…

< /script >

< div class=”stock-number” id=”stockDemo”>< /div >
< div class=”stock-text”> In Stock< /div >
}
else
{
<stock-indicator></stock-indicator>
 //Old code that works. If feature is turned off the ordinary code will run. An easy way to turn on off features, you can even turn on feature in dev mode but disable it in production mode…

}

John started the page to see if the script worked as intended. He didn’t create any tests since the code will be deleted as soon the experiment is done.

It worked. So he pushed the code, the CI & CD took care of everything and now the experiment was live.
It took him only 15 minutes.

And the result:

 

Two weeks later:
The PO asked the team how the experiment went. The team showed the PO the monitored data they have gathered. It didn’t show any direct speed increase only 0.5% faster than average. But the conversion rate was higher than before. 1.8%. So the hypothesis with the goal of 5% increased speed and 2% conversion rate did not work. But it still generates 1.8% conversion rate. The PO was not fully convinced if the feature was that good so he wanted to test the experiment one more week.

One week later:
Still the same result 1.8%.

– Ok the experimental code just gave us about 2400$ extra income. That’s nice, lets implement it. Is it possible to get it done within a week or two? (BP)

– Sure, we already have the infrastructure with SignalR since before so we just need to add a new REST API in our Asp .Net Web Api module for the product page. We already have code for retrieving Stock   values in our validation of the cart so thanks to our SOLID approach with SRP (Single responsibility principle) we already have a repository method to retrieve the actual stock value for an articleid.
The style we added can be reused, no need to change that; we just need to add the event handling in our Angular web-component for the stock indicator. It will not take that long either though we only need to act on the event triggered from SignalR and add the new stock value to our view model. Add some code that switch the class based on the stock value. I think with tests we will have this implemented up and running within few days or in a week. Especially thanks to our CI & CD approach and our architecture. (TM)

– Nice. Let’s create a user story and add it to the backlog (BP)

Summary
Sounds too easy right? Like a Utopia? I know, but it’s not. With good agile organization mindset, with well-educated team, with good CI & CD, DevOps and architecture it will be easier than before. As humans we mostly do things too complicated, and thanks to a lack of slack in organizations you don’t have time over for the team to be more innovated and creative in learning.We need time to invent things easier and better than last time. Continuous improvement is so important so don’t waste that benefit.

 

Who is Johan Normén?

Johan Normén is 37 years old, work as a speaker, mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden. He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. He has been nominated as the top 10 developers in Sweden in the Swedish version of Computer Sweden 2015

Twitter: @johannormen

 

 

Advertisements

Implement HDD (Hypothesis driven development – Part of Lean Startup) with #NoEstimate

I remember when I study Information Technology in college (1995), we published local newspapers for our school. I’m not that good writing articles so I mostly worked with the layout and ideas to make the paper look great. My talent was used for the cover and also to make the paper more user friendly. To do this I tested lots of ideas with the team and with survey,  interviews and we monitored what worked best for our readers. We also had another competitor who also launched a newspaper at our school.

I remember one time when I wanted to add a little tag on the upper left corner. A tag with black background and white uppercase words explaining what section you where on. Like SPORT, MEDIA, GOSSIPS, GAMES and so on. I wanted this because I thought it will make it easier for the readers to remember what section they where on and also an indicator so the reader can jump to the most interested section without reading the table of contents. We added this section topic tags on each page as a test.

One week later we asked the readers what they think about the new version. Most of them pointed out that it was really nice with the section topic tags but they wanted to us to add the page number up in that corner as well. Some other stuff was bad so we removed that in the next number and added the page number in the upper left corner as the user wanted. We used reusable template so it was an easy fix.

Two months later, our competitors also took this idea to their newspaper. At this point we got a better colored RGB-printer so we removed the section topic tag names and colored the background instead. Why read the topic when you can see the topic? Blue for sport, red for gossip, yellow for fashion and so on. It worked even better, the user thought some colors felt wrong for some sections, so we changed the color based on the user inputs and they where happy. Guess what our competitor did later on?

Some ideas were great so we kept them even if the competitors copy us. We still sold more newspapers then them. Because we tested new ideas and listened to our readers all the time to get rapid feedback. We engaged our readers. And it was important for us to fail fast if our ideas were bad. It’s our readers who pays us so it would be bad idea not listen to them. How often do you ask your users what pain them the most with your software and what change they expect? What functions can you delete to clean up the code from smells and other unnecessary things that are not used anymore? Do you monitor how the user use your software?

In this post I will give you some introduction and guidance of some practices regarding how you can make this possible with HDD and #NoEstimate.

HDD (Hypothesis driven development)
As development industry continues to mature, we now have an opportunity to leverage improved capabilities with Continuous Design and Delivery to maximize our potential to learn quickly what works and what does not for the users. To do this we need to engage the user or monitor the user’s actions. We need to learn from them not just teach them.

Are you familiar with User Stories and maybe even BDD (Behavior riven development?)

User story focused on capturing requirements for what we want to build and for whom and why. The main goal is to enable the user to receive a specific benefit from the system.

User story format:

As a…  <user>
I Want… <something>
So That… <I receive benefit>

User story are mostly common in the process when some market people or it-leaders have decide what shall be built. It’s not uncommon that the selection of user stories is based on the estimated time rather then the real user value. – Is X that big? Then let’s skip it, we need to give the user features fast and can’t spend time on the most expensive once.

(BDD) Behaviour Driven Development takes us to the next step. It aims to improve the user stories by supporting communication and collaboration between developers, tester and non-technical participants in a software project.

In Order To… <receive benefit>
As A… <role>
I Want… <goal/desire>

User Stories and BDD stories are both often created within the development process as a result of already prioritised features that will be built based on our definition of done and so on. But how value-driven is that? Do you really know if that feature will be used by the users? They might say “we need it!” if you ask them, but will they use it? Is it worth spending time to estimate something you are not sure if a user really need? Is it worth develop a user story with high quality and release it month later if the user wont even use it anyway? What about the feature Y you decided to skip because it was estimated to high? We need to view our work as an experiment to really find out. We need to ad a better process for innovation and monitoring what pains the user most to understand what benefit them the most. We need to learn from the users, not the value of an estimate or assumptions.

When viewing work as an experiment, the traditional story process is insufficient. As in college when we defined the steps we needed to achieve the desired outcome. We need to observe if our idea (hypothesis) is valid.

If we observe signals that indicate if our idea or hypothesis is correct, we can be more confident that we are on the right path and can alter the user story process to reflect this.

Keep in mind that a hypothesis that was once valid may later be disproven when markets change. Markets move, so you need to continually ask whether the business model is still valid or not. HDD is value-driven and the market is too.

A hypothesis story can be implemented with different technologies and tools, the main goal with a hypothesis is the explanation what action you need to implement to get the indication if it worth implementing.

We believe… <this capability>
Will result in… <this outcome>
We will know we have succeeded when… <we see a measurable signal>

Examples of Hypothesis-Driven Development user stories are;
We Believe That real-time stock values on the product page
Will Result In improved customer engagement and conversion
We Will Know We Have Succeeded When we see a 5% increased speed from the time the stock change in real-time till they press the add to cart button.

Why I mentioned #NoEstimate in this post topic is because you do not estimate when wotking with hypothesis, you just use the easiest and fastest tools to reach your goal, so you can take action if it’s worth implement it as a full qualified feature in your system or not.

Lean startup and UX for Lean Startup will explain more how. I will not cover this in this blogpost, it’s almost to big as is.
A hint: Instead of write user story, spend time estimating and then build all the technical stuff regarding real-time stock value changes, just add a simple timer in javascript that decrease the stock value on different products to see if the user reacts on it. If not, then you have saved lots of time not implement a fully working feature. I can promise you that it’s worth spending less then one hour to implement some test data rather then add a push or pull system (with eg. SignalR?) that monitor the stock values in the e-commerce platform you use.

To use value-driven-design and HDD you need to change the way your organization work and also the traditional mindset of yours.

Some methods you need to add to your organization are:
– Increase observations
– Formulate a hypothesis
– Design an experiment to test the hypothesis
– Evaluate if the experiment has succeeded
– Conduct the experiment
– Evaluate the results of the experiment
– Accept or reject the hypothesis
– If necessary, make and test a new hypothesis

To make this possible you also need the developers to be more mature in the way they craft. HDD requires that you write code that is easy to test, profile, benchmark, and change. They need to learn about A/B Testing and how to implement it to the code.
A more mature model for CI (Continuous integration) and CD (Continuous Delivery) are also a must for even bigger benefits. Other useful techniques can be feature toggling (can be used for A/B Testing) and also rapid development for releasing and remove measured and monitored prototypes, dark launches and even live ops.

“By combining Continuous Delivery and Hypothesis-Driven Development we can now define working software and validated learning as the primary measures of progress.”
– Thoughtworks

Traditional process vs HDD process

Traditional process
1. PO add epics, features or user stories in a backlog.
2. Team estimate the items
3. PO decide what items need to be implemented (often based on time and cost)
4. The team implement the feature
5. The team release the feature
6. The process starts over from 1-6 again.
(7). Team spend time on supporting and maintain released software.

Result: Something will be implemented and released, but it’s uncertain if its what the users really want.

HDD process
1. The organization make observations
a. Ask the support people what makes the users scream the most
b. Ask the team and acceptance testers what features are hardest to test
c. Interview real users and also create personas if hard to get interview, or use both
d. Record users using the software to get indication of the impression and feelings etc…
e. Look at trends, competitors
d. Monitor and measure the app (buttons clicks, sequence orders, timers etc..)
(Did you know that people mostly use only about 10%-20% of all features in one
application?)
2. Formulate the hypothesis.
3. Design what’s needed to experiment the hypothesis.
(see: Lean startup, Lean startup ux for more inspirations.)
4. Evaluate the result of the experiment
(If you need to implement a monitored A /B Testing and  prototype, remember you need   some time to collect that data. It’s better to wait for a result then spend lots of            unnecessary time to implement something that takes time and never gets used and spend that time on the most valuable features instead)
5. Accept or reject the hypothesis.
(If rejected, start over from 1 or continue from 3)
6. Make a User story of the accepted hypothesis.
(#NoEstimate: There is no need to estimate the user stories, at this point the users have already told you they will use the feature and need it. Save unnecessary waste instead and spend time to continuous improve your CI & CD process instead.)
7. Implement the user story and release with CI  & CD in mind.

Result :
You will mostly spent time on feature users really want that gives you the best value and ROI. You don’t waste time on features that never get used. And the whole HDD process is like an innovation process as well. So you get that as a bonus too.

If you liked this, please read: Hypothesis Driven Development with example

 

Who is Johan Normén?
Johan Normén is 37 years old, work as a speaker, mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden. He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. He has been nominated as the top 10 developers in Sweden in the Swedish version of Computer Sweden 2015

Twitter: @johannormen