Hypothesis Driven Development with example

My previous blog-post was an introduction to HDD (Hypothesis driven development). In this blog post I will take it a step further and give you an example how you can use it and how easy it can be and how it can change the way you work entirely to a new level.
You can read it [here].

Remember:
HDD is an experimental driven process, it’s main goal is to find out if a certain idea generates ROI and happy users. To get fast result you will also need fast support to evaluate your experiment, if you want to evaluate the idea in production I prefer a smooth decoupled architecture and a mature CI & CD organization. It’s not a must to, but it will really decrease development time and cycle time.
Sadly, I have no maturity model written in English yet. You can read one I did in Swedish on my LinkedIn posts [here]

Let’s start the story.

Team:  A cross-functional team with talent people regarding:
Design, UX, Coding, Testing and so on. And also including some market people and product owner.

At a meeting with the customer and team a team member (TM) got this cool idea regarding an e-commerce platform.

– What if the stock value changes in real time? Wouldn’t that be cool? Maybe it increases the conversion rate? The users might be stressed that this nice shoe will get out of stock?

A business person (BP) responds:
– What will it cost? Can you estimate?

– I don’t know, and I don’t. And I don’t know if my hypothesis even works. Can’t we skip this estimating and cost discussion and find out and then see if it’s worth implementing at all?  (TM)

– How? (BP)

– Let’s just do some A/B testing and fake a simulation of stock value changes and monitor if the user will buy the product more often or faster than today? If we get an 2% increased conversion rate it really worth implementing isn’t it?  (TM)

– Yes, you got a points there, this Sounds interesting. (BP)

Instead of creating User Stories the team now write Hypothesis stories.

Real-Time stock value changes
We Believe That
real-time stock values on the product page
Will Result in improved customer engagement and conversion
We Will Know We Have Succeeded When we see a 5% increased speed and about 2% conversion increase from the time the stock change in real-time till they press the add to cart button.

The meeting generate lots of hypothesis stories and the product owner can now prioritize them based on the ROI for the result outcome.

– A 2% increased conversion rate is really nice. Today we got like 5000$ each day, 2% will give us about 100$ more. In a month it’s around 3000$. If the experiment works and generate this value; You can spend around 30h of development time and we still get more money than not implement it at all. (BM)

– Yes you can say that, but let’s not convert it to time yet, let’s see if it’s worth implementing at all. Because if we spend around 10-30 hour for this feature and it will not increase conversion at all; we just wasted lots of money for developing a crappy feature. Let’s be smart and just spend time creating features that generate values in short and long runs instead. Assumptions is not a good thing. (TM)

– Ok, we can give it a try, because we have done so many bad assumptions we even can’t monitor at all. (BP)

The Product owner decided to activate the experiment and tells the team to do the fastest possible implementation to get information to see if this hypothesis will generate values at all.

The team gathered as usual at a stand-up and activated the work of the hypothesis.

– So what is the fastest possible solution to get rapid feedback regarding this? (TM)
– We can ask the customer?
– Nah, the customer might say, cool feature or – Do you want to stress us even more?
– Ok, let’s implement a test in our code and monitor the outcome?
– Sure, what’s the fastest way?
– This is part of the UI so let’s do it simple with some JavaScript. The only thing we need to understand is if it works. So there is no need to make it perfect. We do not need to add back-end functionality to get this information either. I can type a JavaScript that just randomly decrease the stock-values on some products. And then monitor with our application-insight code if they will press the button faster or not. It doesn’t matter if the stock value is a faked value for a while… If the value is 5 or 4 doesn’t matter that much.
– Nice, cool this will take us around 15 minutes or so? And thanks to our CI  & CD approach the feature will be out within 20-30 min or so and we can then start monitor it. If the result is good, then we implement it all the way with tests and quality in mind.
– Cool, this is nice, if the PO asked us to estimate it and add management work around it and do all the other stuff we do for a user story it would take longer cycle time than 30 minutes for just get started on this one. We aren’t wasting anything here, just getting knowledge and information if the idea is worth implementing or not. We will save around 15 minutes and also get the information if it’s even worth implementing. It’s much better than assume it does by implement it and hope it will generate values.

The team agreed.

Liza loved the idea regarding NoPSD so instead of spend time on working with PSD files or other tools for design she just opened the SASS file (css manipulation) and added three new classes with three different color. Green for normal stock values, Orange that indicate near out of stock and red that indicate very soon out of stock.

.stock-Normal
{
     color: #38893d;
}.stock-Warning

    color: #fe8f01;
}
.stock-Critical

    color: #be0000;
}

Thanks to nice SASS structures it just took her 2 minutes. When she pushed the code to the source control it automatically generated a new css file that got released into production, this because the team had nice CI & CD approach. (Why not release it? It will not hurt anything though nothing using the classes at the moment anyway.)

The front-end was built on AngularJs and the stock area was created as a web component and rest of the system is based on ASP.Net MVC and Web Api. Meanwhile Liza added the css John added a feature toggle in the code and took the html from the web component template and added it to the new feature for the hypothesis.

The feature toggle framework they use uses one configuration file to indicate if the feature is on and off.
They simple add it as an appSetting in web.config or appSettings.json (if using ASP .Net 1.0 Core).

“IRLStock” : “true”;

The team uses a framework that uses a C# class as a feature toggle so they easy can remove the class and get compilation error where the feature existed, so they do not forget to remove the code later on.

public class IRLStock {}

The code with razor syntax (ugly code? I know but much faster than create a new web-component to get the result from this experiment, it’s important to not waste unnecessary time for a more complicated solution, then you missed the whole point. This code will be deleted after the experiment anyway. For experiments it’s just waste to add sugar to it.):

@if(Feature<IRLStock).Is().Enabled)
{  
     < script >
… a script that just gets a random number between 2-6 seconds…
… this number just trigger a interval eg.
setInterval(changeStock, randomNumber);..
..
and set a new class to the element…
getElementById(“stockDemo”)…

< /script >

< div class=”stock-number” id=”stockDemo”>< /div >
< div class=”stock-text”> In Stock< /div >
}
else
{
<stock-indicator></stock-indicator>
 //Old code that works. If feature is turned off the ordinary code will run. An easy way to turn on off features, you can even turn on feature in dev mode but disable it in production mode…

}

John started the page to see if the script worked as intended. He didn’t create any tests since the code will be deleted as soon the experiment is done.

It worked. So he pushed the code, the CI & CD took care of everything and now the experiment was live.
It took him only 15 minutes.

And the result:

 

Two weeks later:
The PO asked the team how the experiment went. The team showed the PO the monitored data they have gathered. It didn’t show any direct speed increase only 0.5% faster than average. But the conversion rate was higher than before. 1.8%. So the hypothesis with the goal of 5% increased speed and 2% conversion rate did not work. But it still generates 1.8% conversion rate. The PO was not fully convinced if the feature was that good so he wanted to test the experiment one more week.

One week later:
Still the same result 1.8%.

– Ok the experimental code just gave us about 2400$ extra income. That’s nice, lets implement it. Is it possible to get it done within a week or two? (BP)

– Sure, we already have the infrastructure with SignalR since before so we just need to add a new REST API in our Asp .Net Web Api module for the product page. We already have code for retrieving Stock   values in our validation of the cart so thanks to our SOLID approach with SRP (Single responsibility principle) we already have a repository method to retrieve the actual stock value for an articleid.
The style we added can be reused, no need to change that; we just need to add the event handling in our Angular web-component for the stock indicator. It will not take that long either though we only need to act on the event triggered from SignalR and add the new stock value to our view model. Add some code that switch the class based on the stock value. I think with tests we will have this implemented up and running within few days or in a week. Especially thanks to our CI & CD approach and our architecture. (TM)

– Nice. Let’s create a user story and add it to the backlog (BP)

Summary
Sounds too easy right? Like a Utopia? I know, but it’s not. With good agile organization mindset, with well-educated team, with good CI & CD, DevOps and architecture it will be easier than before. As humans we mostly do things too complicated, and thanks to a lack of slack in organizations you don’t have time over for the team to be more innovated and creative in learning.We need time to invent things easier and better than last time. Continuous improvement is so important so don’t waste that benefit.

 

Who is Johan Normén?

Johan Normén is 37 years old, work as a speaker, mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden. He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. He has been nominated as the top 10 developers in Sweden in the Swedish version of Computer Sweden 2015

Twitter: @johannormen

 

 

Advertisements

How to make Lean UX with A/B testing fun in ASP .Net MVC? Part 1

A/B testing is jargon for a randomized experiment with two variants, A and B, which are the control and treatment in the controlled experiment. It is a form of statistical hypothesis testing with two variants leading to the technical term, two-sample hypothesis testing, used in the field of statistics.
Source: https://en.wikipedia.org/wiki/A/B_testing

download
And some other nice explanation:  https://vwo.com/ab-testing/

It’s really hard to change or add something and know if it’s a success or a failure before we have test it on real users. In Lean UX it’s important to do this test process as cost effective as possible by decrease unnecessary waste without skipping important work. The best feedback is when you let people use it and see if they like it or not. You can test wireframes, design sketches but that’s not the same as test It for real in real action. Many users will see this cool new thing as a sketch and think; that looks really cool! but is it really that cool when it’s implemented and live in action? Not all the time, maybe some times but we can’t be really sure before we can test it. In Lean UX A/B testing is one approach that can help you get early and rapid feedback if the idea is great or just a new pain for the user. The core idea of Lean UX is to aim small and hopefully shoot big. In other words, do as little as possible and test it before you spend too much time on your ideas.

Let’s see how ASP .Net MVC can make this by just add very little code to make A/B testing fun and easy. 🙂

I love to use the creativity to think; – how can I add something simple that works and also gives me many other possibilities? It’s some kind of evil obsession I have and that’s also what I love most about software design. In this case I will show you how you can add some code to make your page not only A/B test friendly you can use this idea for other things like themes, change layout bases on personalisation and you can even use it to give your user some luxury in e-commerce sites and so on.

In the software, there are thousands of ways to solve the A / B testing. What I found very useful that also decrease waste from more complex solution is to copy the view you want to test to a new location and add changes to that new file, and then let ASP .Net MVC show just that new view based on some business rules. I also want less code, and it shall be easy to remove my test-view without too much work. Therefore, I want to use the view folder and it’s files and not manipulate controllers if not needed. It’s more waste to add new controller if you can use the the you already have when you want to test another layout. My approach doesn’t stop you from A/B test controller as well if you want to add that feature in the future. Remember I’m fan of Agile and Lean approach and principles like KISS (Keep it simple stupid) and YAGNI (you aren’t gonna need it). That’s why I keep things as simple as possible and expand it if more advanced featured is needed later on. 

This is my goal:
Let’s say you want to test your home view, you can then easily add a new folder inside the view folder for example “B Testing” and add your home folder and its view in there. And then add some changes to it… DONE! Cool right? In less of a second you just made a A/B tested home view. Sell it for 5h of work 😉

ab_1a

To make this possible I want to let my view engine look for views in special folders but fallback to the original if there is no files in this special folder. ASP .Net MVC do this by default. Like this setup:

ab_1

I call the variable theme because I can use it for other purpose than just A/B testing too.

Lets create this
Go to your ASP .net MVC project and add a new class that implements the interface IViewEngine from System.Web.Mvc. And implement the view engine code.

ab_2

Then add the folder locations within a method. I call mine CreateThemeRazorViewEngine

ab_3

Then I create a RazorViewEngine and configured the Partials, View and Master locations to the new folder structure of mine. In this case ASP .Net MVC razor will act as normal but start looking for files in my theme folders if they exist.

I love the factory and provider pattern so I will use those. As factory I will use DI/IOC framework, I like Autofac but you can use whatever you like the idea is the same.

I start by creating one IViewThemeManager interface with one method GetTheme. This interface is the one we use on all the ViewManager providers that will handle all the business rules based on why you want ASP .Net MVC call for another view or partial.

ab_3a

And then I implementation my factory method to the ViewEngine with the IOC and some error handling.

ab_4

If there is no IViewThemeManager provider configured it will give me null and return the default Razor view engine. That’s why I use the _fallbackViewEngine. If there is an IViewThemeManager provider registered I ask for the Theme which in this case is the folder name I want the view engine to use.

After that we need to tell ASP .Net MVC what Engine it shall use so we need to add some code for the methods implemented from IViewEngine

ab_5

In this case I call my CreateViewEngine that will return either the default engine or mine.

Here is the full file:
ab_6

Now go to Global.asax and clear all View Engines and add this new one in Application start.

ab_7

We can now run the project and everything will work as normal. This because the resolver can’t find any IViewThemeManager provider yet and fall back to default.

ab_8

Now it’s time to add the A/B testing business rules. In this case I want to know if the “Learn More >>” button gets more clicks with another text. So I want some user to get “Learn more >>” and others “Want to learn more?”

First I copy the index file in my home folder to a new home folder under “B-Testing” and just change the text on the button nothing more.

ab_9

ab_1a

Then I create my IViewThemeManager provider. My ABTestingViewThemeManager.

ab_10

After that I will add some simple rule. In this case I just send them to my “B-Testing” view folder if the date time second contains number 2. (Just for the test, ok? Real A/B testing might need more complex logic and maybe even som settings, depends on how important the function is, remember don’t make it to complex if not needed. KISS 🙂 )

ab_11

After that I configure my interface and provider with autofac.

builder.RegisterType<ABTestingViewThemeManager>()
.As<IViewThemeManager>()
.InstancePerRequest();

And when I run the page and if the DateTime.Now.Second contains number 2  it will use my B-Testing view instead.

ab_13 ab_12

I also added the view rules for partials so you can just test a partial as well if you want to, you don’t need a full view, you can use the default one that just calls for your partials.

Just copy your A/B partial to “B-Testing/<your controller folder>” and this code will take care of that too.

And if you use partials to render templates for example angular you can even test unique directives and web components if you want to. Just add the angular template in your partial and copy just that partial and its folder structure to the “B-Testing” folder or what name you prefer.

You can also add a provider that will give some user a whole new view, layout or webcomponents layout if they are lets say “pro users”. Just add a pro user folder and create a new IViewThemeManager provider that check if the user is of the role pro or something like that.

Happy play around.

Who is Johan Normén?
Johan Normén is 37 years old, work as a speaker, mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden. He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. He has been nominated as the top 10 developers in Sweden in the Swedish version of Computer Sweden 2015

Twitter: @johannormen