Feature toggling and A/B testing in production with ASP .Net Core

Do you remember the old days when we used asp 3.0 or earlier version and it was common to fix things in production? Ad-hoc was king, kind of, if you did not forget to add the changes to the source control.
The funniest thing with innovation is that we mostly take to steps back to jump three steps ahead. Feature toggling, A/B testing, DevOps, CI and CD and Hypothesis development make this test in production popular once again, but this time in a more controlled manner. Instead of change code in production and test it,  we simple want to do the same thing today but with automation, so we do not need to add or change the code in production to get the same result. We simple add the changes to our master code push it and it will get released into production for testing. In this blogpost I will give you some examples how you can manage this in a structured and simplest possible way.

First of all, what we want to do is test our ideas our hypothesis in production, we will also make it possible to turn feature on and off in production. Here is an earlier blogpost if you wonder why we want this.

and here https://johannormen.com/2016/04/07/how-pain-driven-developmentdesign-pdd-can-be-used-as-an-extension-to-hypothesis-driven-development-design-hdd/

To make this happen we need some discipline, first of all we must get rid of our pride to do the best possible solution from start, we don’t want to waste too much time (and money) to just get the information if a feature is worth implement as full scale. So we need a controlled AD-hoc approach. This can be solved with a feature toggle framework. I made one here, but feel free to use others.  https://github.com/jnormen/NS2.FeatureToggler

This frame work will help you to encapsulate the AD-Hoc code so it will be ease to remove and also easy to read.

You simple create a class that implements IFeature like this.
public <nameOfTheFeatureIWantToTurnOnAndOff> : IFeature

Eg: PressMeButtonText

And then use an if statement like
if ( Feature<PressMeButtonText>.Is().Enabled )

To handle nice CI and CD we also need a god structure in our solution. If you remember clean code that’s all about just clean code, it’s also important to have clean structure and architecture. In my case I love to use one project not many and also a folder called module, in this I put all the code for different domain boundaries. Kind of the same way we separate Microservices and so on.

A module can be everything regarding products, carts, invoices, users and so on. To indicate for the developers there is a test of a feature ongoing, I simply add a featuretoggle folder in my module. If there is no such folder then there is no feature on that module for test in production.
Here is where the discipline comes in. You really need to remove stuff you do not use to help others and yourself to understand what’s going on with the code, architecture etc. The same rule as for remarked code.

So this is how I made my folder and project structure:
1

Every code that is just for the FeatureToggler exist in this folder, the IFeature class, scripts, styles if needed. I use gulp for my build pipeline so I just let all the sass, less or js files to be included in my bundle if needed as well. Therefore, I also add the modules in my gulpfile. Eg:
1a

I also made a more advanced gulp-pipline to move views from my module to the views folder structure. I know it seems insane but I want all code to exist in my own module not laying around everywhere in the project. So if I want to update a view I do it in my module views and it will be copied to the views folder for ASP .Net Mvc to handle it. In this case I can delete the whole module folder and get rid of all the codes with just a click.

My feature toggle framework uses appsettings.json to turn off and on features. I also made it possible to add a method to my feature class if I want to do more advanced code to indicate if the feature is on or off. Like some A/B Testing routines, if a person is in a special role and so on.

This can be done by just implement a IsEnabled() function that return a Boolean on the class that implements IFeature. (This is a dynamic method so it’s not required by the IFeature implementation.)

public bool IsEnabled()
{
   … your special code to indicate if the feature is on or off…
}

To use the Configuration in ASP .Net MVC .Core just send in the Configuration to the static SetConfiguration method in your startup.cs
config2

To set a feature on and off this way simple add the class name and the value in appsettings.json like this
config

Now let say that I want to test if a person press my button if the text is something else then the ordinary button and color is red. (I know really bad example, but it’s just to show you how you can handle your code).

Go to the view you where you want to test your feature, add a razor if statement and all the code needed just for the test. It doesn’t matter if it’s inline code, remember it’s just a test, nothing more and nothing less.
2

I added my test style and my test script and my test button. Within my new feature. My old code is still there in the else criteria. Now I’m ready to push this little code into my master branch and let Visual Studio Team Services (VSTS) trigger the CI and build configuration and add it to production. A tip, you can use Docker or Windows Containers here as well for better testing benefits and so on. I do not use feature branches just my master. Therefore, it’s also important to have tests in your CI & CD pipeline. And here are why containers are so useful as well, easy to rollback and then fix if any problem with this newly added code.
For more information regarding feature toggle and source control
http://martinfowler.com/bliki/FeatureToggle.html

If you want to add the test on many different view I recommend to do some shared module and create an ad-hoc web components instead.

module
…..shared
……..featureToggling
………..scripts
…………..pressMeButton.Js

The build pipeline with gulp will include it as well so no manual work at all, beside creating the folders and the web component file. (Angular 1, 2, Aurelia, React, Ko or whatever you like.)
3.Ng

And in production we can see something like this. In this case the feature is off and we got the standard button.
off

And here it’s on.
on
Remember it’s also important to log data so you can evaluate what buttons get most clicks. So you can decide if this feature is worth implement in full scale.

Remove the feature
I did this feature framework help you with the cleanup. You can add if statements in the code without any framework, but then it can be harder to know where to clean up in your code. In this case, just delete the feature toggle folder in your module folder. And build your project.
4delete

You will then get compilation errors and it will help you clean up just that feature.

6delete

So just delete the if statement. Or add the new code in its place.

7delete

8delete

Easy right? The structure is self-explained, the code is mostly clean, the test is a was ad-hos solution just to save time to get data if it’s forth implement at all and so on.

 

Who is Johan Normén?

Johan Normén works as a speaker, mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden. He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. He has been nominated as the top 10 developers in Sweden in the Swedish version of Computer Sweden 2015 and member of IDG Expert Network Team

Twitter: @johannormen

 

 

 

A mature maturity model for continuous delivery including culture and DevOps

Do you love the idea of a model that rules them all? I bet you have, we all have 🙂

“Devops has huge business benefits: Statistics show that organizations that practice devops outperform the SMP500 over a 3 year period, high-performing IT organizations have 50 percent higher market cap growth, and so on.”

I have ben fan of models in my whole life (kind of, since I remember anyway). The idea with a model it that it’s not a process. It just helps you understand things. While a process is how you shall navigate. I’m not fan of processes. I love to create them though, for others. It might sound rude but I’m a creative person that love to be on the top of the wave. Processes can’t make me think outside of the box or live outside it so I just creates them for others that don’t have time or interest to be on the top of the wave all the time. My daily activity in my head loves the ideas of innovation, efficiency – how can we make it easier and still with high quality? How can we do it in such a way that we do not need to do a lot of bad work when we want to change something? These are the questions I always ask myself every day in all areas. I can not stand the feeling to stand still and be smug. I am a thinker who loves changes. I’m curious person.

The last 10 years or so I have always been interested in different working models/processes. Therefore ALM (Application Lifetime Management) became an interesting area for me. Agile processes, gamification and so on.

We have this constant goal and vision to work towards something better. It shall be easy to make adjustments to the code and the architecture. It shall be easy to delete code that is not used anymore and, last but not least it should be fun and innovative approach to work with software development and deliver a smile on the customer’s lips every time. Faster deliveries and keep up with the competition has never been more important than now. Before, it was easier to become complacent and just sell on. Today we cant work this way if we want to keep up.

drastolen

Today we talk about agile delivery, continuous delivery. We’re talking about spending less time on code problems with smart and effective continuous integrations. We talk about the importance to measure value via monitor behaviour etc. We’re talking also about saving time in testing, operations and so on in order to keep up. To help and make managers and buyers less busy, several companies worked to develop various maturity models. With the main goal to be able to run faster delivery and early take part of the digitalisation and its digital transformation taking place today.

I wrote myself a Swedish post on my LinkedIn about five maturity levels for the  transformation regarding better continues delivery benefit.

Level 1 – A few develop are the organisations super heroes (Ad hoc solutions)
Level 2 – Time-Boxed releases
Level 3 – Regular deliveries
Level 4 – Release Command
Level 5 – Hypothesis-driven development

I recently wanted to find a model that explains all this in a more technical level. To help companies to identify where you are in the area of DevOps, test, code, culture, processes and so on. I have read four different maturity models from four different companies. But I didn’t like them. Most of them feel like they were created based on other companies’ problems and not based on a general level. A colleague of mine gave me a link to a model that actually is a great addition to the five levels I previously presented.

The model comes from the company Praqma. Not only that it is presented nicely, it is also spot on based on my thought and experiences in that field. The expert step is a great addition to #NoEstimate, Lean Startups with Hypothesis-Driven-development/Design and Pain-Driven-Development/design as the level 5 above.

cd-maturity

You can read more about it here
http://code-maturity.praqma.com/

If you are interested how you can work or how a system need to be designed and what transformation you need to do to reach the expert level, don’t hesitate to contact me or read my blog and forthcoming posts.

Who is Johan Normén?

Johan Normén works as a speaker, mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden. He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. He has been nominated as the top 10 developers in Sweden in the Swedish version of Computer Sweden 2015 and member of IDG Expert Network Team

Twitter: @johannormen

The top 3 most important skills for being a great team-working agile developer

1. The Skill of Collaboration
Why agile work with teams in agile frameworks is because of two factors. Cooperation skills and collaboration skills. Cooperation is the part when you work as a whole with the customer, product owner (if scrum), agile coach (scrum maser if scrum) , organisation and the team. Cooperation skill is one of the most important requirements for continues delivery and success. Without it you are lost. On top of cooperation you got collaboration. Collaboration is the part where there is room for innovations. When you as a team brainstorm, discuss how to go forward together, innovate new features or maybe effective ways to solve hard problems. Collaboration is working with others to do a task and to achieve shared goals. Agile works because cooperation and collaboration is a must. In other methods like waterfall this is often split into all the different phases and different roles that poorly talk to each other. Every phase do its own work and deliver it to the persons in the next phase. It’s not effective at all and cooperation and collaboration will get lost.

You also need to welcome changes of the requirements. Waterfall mostly don’t. And we know that customers never know what they really want in detail so we really need to have a model that welcome changes. To welcome changes we need to cooperate and collaborate as a whole team. Use Collaborative Design if needed, use Agile UX, Lean UX or whatever you need for the user experience model. The UX people must talk to the designers and the developers to understand if the idea is too expensive or even possible to do. Testers might also have inputs regarding the UX. The developers might have learned some new technical frameworks that make the UX even better. Technical information that UX people often do not know about if no one present it to them. And it’s also important for the developers to understand the UX ideas. It’s also important for you as a developer to be curios and ask why to understand. Never assume anything (we will come to that later.) It’s also important as a team to be able to explain the goal to each other so we are all on the same page all the time in the whole process.

To get the most out of collaboration shared workspaces is important. The more distance to each other the harder it will become to collaborate with less misunderstanding. So as a team it’s important to have the possibility to talk to each other, and understand that talking with each other is a must. Your mission is to help each other despite your role. If you are not willing to spare time for a discussion you will probably harm yourself in the long run. The team as a whole will probably get problems. The team’s problems is also your problem. You are not alone in an effective team. The best tool for collaboration is people standing with a whiteboard. Different people understand information in different ways. To draw, talk, use metaphors mostly is enough for the teammates to understand the issues. If you are unsure if the persons you talk to understand what you just told them, let them explain for the team the same thing you just did to see so everyone is on the same page. Second best collaboration tools is video or skype calls, the worst media for collaboration is chat-programs and e-mail conversations [Cockburn – Agile software development].  So as a team try to demand for an open space where you can draw, talk without disturbing other teams any time you need to. If that’s not possible the collaboration part can be tricky and not as effective as needed. As a team member it’s important to understand that communication and meetings are not bad things. Meetings are held to solve problems and conflicts, communication in general to understand the goal and help each other reach this goal. Work close to each other so you can ask a question, let others review your work and you others and so on. Be open to share your ideas and thoughts and listen to others, work with the same principles and guidelines.

If you put people in different rooms they will surly produce something but without collaboration it will probably be something no one wants. The assumptions will increase and the blame as well. To blame people in your team indicates bad collaboration and only destroys it. So it’s important to handle conflicts in the collaboration part as well. Conflicts will be there when working so tight to each other, it’s unavoidable. So don’t be afraid of conflicts. But handle them as soon they arise.

Story:
I was one of the developers in a project for a big company that own buildings. The company I worked for was very fond of roles. UX, Designers, information Architects, Developers, Testers. All grouped together in different rooms based on their roles. Testers in one room, Designers in one etc. Every role had x-total time to deliver their part. The UX person got 100 hours. After 100 hours the UX person went to another project and the Designer gets the material and had 200 hours to add look and feel of the wireframes. The designer only got 70% of the work needed from the UX person. Because the UX person was assigned to other projects as soon the time was out the designer was forced to do the best he could. When the designer’s time was up the UX material and design sketches was delivered to the developers. The designer only managed to give the developers design for 55% of the projects main requirements. The UX never talked to the designer and the developers, the designer never talk to the UX person or the developers. When the developers needed to understand all the missing parts the project manager gave us this answer: “The UX and Designer are now in other projects” when we told the project manager that this is an issue, he just told us to assume what to do. We are the experts, how hard can it be? I think you can figure out the rest…

You are the expert [You tube]

2. The skill to gain respect/trust
Respect is a powerful skill, but also a really hard one. Respect others and you will also be respected. We all have different skills and talents, if we respect that we also get stronger together as a team. A respected team will get better relationships with the customers and the organization. Try to understand the strength and weakness within your teammates. Don’t blame a weakness, it’s your responsibility with the team to compliment them. Get rid of “me” and increase the “we”. We are doing this together, we support each other to solve the problems together, we can do it, we will do it… It’s not one person’s code that’s failing it’s ours (even if it’s a certain person who wrote it). If something is broken, fix it. If you blame the creator it will only split the team with conflicts and trust issues. One day you will be the one that break something and find gratitude if someone help you fix it.

Show gratitude to each other, compliment the achievements of each other. Even small parts count. If you say you will fix something, do it. Don’t commit to something you can’t handle. Offer assistance if someone got a problem and respect each other abilities. A highly respected team will get less conflicts and will cooperate and collaborate better to meet the same goal. By trust you get respect. Trust and respect are both two ways communication. You must trust others and respect others to gain both. Some organizations think that developers are the solution to everything and forget that the organization as a whole is. With bad leaders you get bad teams. It’s important to let teams coach leaders and leaders to coach teams. In agile frameworks like scrum you have retrospectives as a golden opportunity to bring this up if it’s a problem. If you got many conflicts in your team, then you know that you have trust issues. Bring them up early before they end up in resentments.

Be open minded, let go of control and make you vulnerable. It’s important to make mistake and it’s also important to be able to. Making mistake is not a weakness it’s you motivation for establish strength and knowledge. See things from other perspectives is a skill. That’s why you also must let go of fear and control and make yourself vulnerable. Let others know when you did a mistake rather than activate defensive mode. Take a moment and reflect how you act in front of others that just made a mistake. You don’t blame them, right? You often give them your hand and help them do better, right? We are all the same, other people have the same feelings, don’t judge yours. Be transparent with your team. The same goes with collaboration with customers. Be transparent, try to see their side before you criticize them. If there is some kind of conflicts with the costumer it’s mostly because you don’t see the vision and goal the same way. Open up your mind for new ideas and give yourself the opportunity to change how you think and how you view the world. Doing so increases the options to enter new ways to solve problems and increases your confidence with others and you will gain more respect and trust too.
Coach your teammates, let them know that mistakes are not bad, give them positive feedback, don’t blame them for their mistakes. And most of all listen to what they have to say.
That doesn’t say you cannot have strong opinions, but you need weakly held.

Story:
The first time I was a full time Team Leader I did the biggest mistake ever. I thought I could handle everything. The requirements with the customer and the team, the architecture of the system, the design of the system and to lead the team. I got trust issues, the respect was very low, the collaboration was bad, and everything was bad. I could not understand why. I was there supporting the team with every little question that was delivered to me. I told them what to do all the time. I worked overtime for my team and all I got was angry comments, no respect at all. When I told something they ignored me and went to someone else. Later on I found out that my good will was just bad. I wanted so badly to help my team to succeed so I took all the responsibilities from everyone. That’s a nightmare for a team. You can never create a self-going teams if you take all the responsibilities. It’s important to hand it over to the team, let them solve the problems together. I took all the trust from them not letting them decide by them self. I was not open minded, I never saw the forest for the trees. This is also important for the team members, don’t take too much responsibilities let other have some. Even if you are or think you are better than them. It’s so easy to be the one taking most of the responsibilities, the one the team must ask if they want to go forward. In the end you are the one your teammates will blame if anything is broken. Let all take responsibility and let them learn from each other’s and their own mistakes, just be the mentor and the mentee.

3. The skill to challenge Assumptions
Assumptions are our worst enemy. It often give us bad business. We often make assumptions when we don’t fully understand a situation or a need. The reaction is natural for making up our own story by filling in any missing information. We do this because we try to make sense of uncertain requirements or customers need. The problem with this is that most of the time our story is incorrect and most often causes all kinds of complications. Assumptions is like the red devil on our shoulder while the good angle is on the other side and try to tell us something but we won’t listen. We are afraid of showing weakness and think we do well by our assumptions. Bad assumptions just don’t give you bad decisions or features it infect the whole team.

Challenge assumptions by getting more information about something if you are not certain. In some mysterious way questions seems to be a weakness but it’s not. To not ask is. So when you don’t know it’s a clear sign you need to get all the facts. Coach your teammates to be open with questions. Don’t blame or reject questions, they are often you and your teammates guide to success.

There is no stupid questions only stupid answers. The fact is, we don’t know what the truth is unless we ask. If you find a bad requirement go ask for more details. In agile frameworks like scrum you got the product owner as your service to help you understand the requirements. Never blame the product owner for bad requirements, the product owner also need help to get the best out of the team.

You also need to be curious to challenge assumptions. If someone talk about things you don’t know anything about, just ask or go and google it for a while, you will be amazed what you will find out. Don’t stop with “I don’t know”. If you can’t find any answers at all, remember it’s no weakness to ask for help on forums or in other teams. Ask for help is strength. It also increase respect and awareness. With high curiosity you will also grow up and continue to learn and practice new topics. Knowledge is your tools for success.

Let your teammates know that words like “how? Why? etc. are the words of curiosity and also very important to understand to move forward as a team. If you feel that someone in your team are not curios at all and always want you to fix the problems for them, try to motivate them by ask them questions back rather than just give them an answer.

Story:
Some years ago I took over a project after a team leader who quit his job for another one. The project was one month late. I asked the team how long they had come. The answer was that 70% was done. When I asked the customer if they were satisfied so far. I got the answer NO. We haven’t seen anything yet. I found it odd and asked the team do demonstrate the system for the customer. What we found out was that it was only 30% done. The rest was just assumptions by the team. I asked how this was possible. The team told me the team leader and product owner was too busy in more important projects so they were never around. When I asked the customer they told me they wanted continues deliveries and continuous feedback loops but got none so they thought we know what we were doing because of the silence. I think you can imagine how that project ended. I bet most of you have been in this kind of situation before.

Summary:
Delete the “Me and you” and add the “We”. Never assume anything and ask when uncertain and give feedbacks and be open minded for all kinds of collaboration, even if it’s about solving a conflict within the team, with the customer or just find out how to solve a problem.
Who is Johan Normén?
Johan Normén is 37 years old, work as a speaker,mentor, team leader, agile coach, and senior .net developer at Softhouse in Gothenburg Sweden.
He has over 18 years business experienced and worked in many different projects and roles. Was one of the creators of Swenug (Sweden .Net User Group) with over 3000 members all over the country. He started the computer era as game designer at the age of 12 with his Amiga and team. Has been nominated two time as the top 100 developers in Sweden in the Swedish version of Computer Sweden.

@johannormen

Special thanks to Linus Norén for the review of this post.

#NOESTIMATE, SUCCESSFUL PROJECTS IN AN AGILE WORLD. PART 2.

The big difference between agile methods and other methods is that you focus more on value and results before time and deadline. The word focus does not mean that time is an unnecessary factor, it is but agile is more about using it effectively in an agile manner.

Many of us that are really passionate about the agile methodology, have actually succeeded with good results in our projects. And our customers have been extremely satisfied and in return we got good relations.

Our highest priority as agile coachers and team is to satisfy the customer through early and continuous delivery of valuable software.
We welcome changing requirements, even late in the development phase. The agile process yields change to the customer’s competitiveness and give us as team leaders or developer the creativity to do good.
We want to deliver working software usually with the timeframe of few week sprints or within few months. Together with the team we want to create good design and code to get out as much of the customer’s requirements as possible, based on requirements with the highest ROI (Return Of Investment) value. And deliver it all within the client’s budget and desired time frames.
We know that business people and developers must work together daily throughout the project. We also like continuous meetings to synchronise with customer continuously. We are simply motivated to deliver high value to the customer together with the customer. And we remove unnecessary management works to visual instead. Therefore, we must constantly work to reduce factors in our own process that does not give teams and customers a good ROI. That is what Agile is all about. As agile team we also know that our time costs money and we need to use it effectively and profitable. We use product owners and agile coachers to remind us.

Heres where #NoEstimate comes in as one of many factors to decrease waste in our projects. #NoEstimate does not mean we should stop estimating, we shall. #NoEstimate focus is to make it right to bring out the best value for both the customer and the team. In Part 1, I talked about the Chaos report that more or less says that about 78% of our estimates are wrong. And therefore classic estimates are waste of time in our process and can instead be used for other things. Like delivering value to the customer, right?  So why not use it to deliver more value to the customer instead? We can never trust our estimates, they are truly lying for us.

Let me give you some more examples.

You estimate with planning poker or something similar to get as close estimates as possible right?
There is a problem with planning poker. Say that a team set an user story with the number 13. What does that mean? 13h? (2.1 days?). A space of time against Another User Story? 13 as in 13 days? The higher the number is, the harder it is to get the correct estimate. Higher numbers are higher risks. The higher it is the more complex it is. If we want to trust the Chaos report, then 13 user story point has 78% chance of being wrong. It even has a chance (if it is a medium sized company) to go 220% over  the time.
That mean that a 13 user story point can result in about 30 instead. If 13 is a estimate number for days it’s 78% chance that such a point would take 30 days (a month, it is more than a full sprint if you have four weeks sprints.). Imagine then if our project has 10 x 13-day points then it is 78% chance that our 130 days user stories instead will take about 290 days.
(Remember there is also statistics that say that the more experienced you are as a team the better you will hit the right time. But how many elite teams do we have in today’s society? We must constantly replenish with new people so we will never be able to have 100% elite team. That is how our reality looks and the reality we have to adapt.)

Another problem is also the negotiations on the requirements during the project. The customer wants a new requirement but would not extend the deadline. So we negotiate away less important requirement for this new requirement. But how? We do this by estimates the new requirements with Planning Poker and get the number 8. But what does it mean? Normally replace another 8, right?. But is an 8 equal another 8? What if the 8 we will replace actually happened to be one of the 25% chances you have to spot the right estimate and the new 8 happen to be one of the 78% that we gets wrong. There is therefore a risk that the new 8 takes about 18 days and the one you removed actually took 8 days to finish. So it’s 10 days overtime instead. It can also be the other way around, but less likely.
Why do we expose ourselves to this risks when we know about this? Why spend unnecessary time to estimates like this instead of use it to focus on delivery on time instead? In next part (part 3), I will describe some #NoEstimate ideas that will provide safer time estimation and much more measurable velocity quality and ROI for the team and customer.

Stay tuned…