Estimating Features: options, context and God

“We need to support registration in the application. Here are some details, how many days to develop?” — Where should you start from? Should you simply throw a number? Maybe read it first and use your gut feeling, your experience? Is there a trap here? Although there are no recipes for estimating successfully™, I want to elaborate on the options in front of us, before we dig into the details.

Guesstimate everything months in advance by 20 seconds of thought

If you are able to read a Feature specification and give estimation based on “whatever works for you”, you win! For small features this should probably be as simple as pie, but what about the big ones? Still easy?

“We have someone who can do it with some sort of dark magic ball !”

I call him (or her) God. If you have God in your company, you win!

(1) Estimate by detailed breakdown

Giving proper estimation without feeling bad about it 10 seconds later is not cheap. I don’t care how much experience you have. So, are you ready to invest? You should not only consider coding effort, but also organization time – testing, deployment, monitoring, bug fixing etc. After you have this in mind, did you consider risks? What about 3rd party integration? What about inexperienced developer who might work on this feature?

One way to provide detailed estimation is by actually doing the Design (of the solution) and bring one to the table; This means doing a Design Review with your peers, break the feature into tasks, estimate hours needed per task and give very solid numbers. You can do it by yourself, you can do it with others; the price is not 20 seconds, but this is highly useful for solid estimation.

Wait, Context is king!

Before you start throwing numbers into the air or diving into an expensive design to come up with estimation, what is expected of you?

1. Purpose – how your estimation is going to be used? To determine roadmap? To understand feasibility? For prioritization between 2 Features? Each might require different “estimation plan”

2. Size – is it good enough to say “it’s a big feature” or maybe “it’s around 20-40 days” or maybe “it’s 24 hours”?

Back to your options list

Well, you could estimate by detailed breakdown, if you really need low granularity. Sometimes, rough estimations might be enough. What now?

(2) Estimate by personal hunch

You can obviously estimate based on your personal hunch, taking into consideration organization time, risks and what not. In many cases, assuming you’ve got enough “gut feeling” built by your sheer experience, this should suffice to build roadmap or understand feasibility. If this is the expectation, it’s probably the cheapest way to provide numbers.

(3) Estimate by collective hunch

You can improve (or at least feel more confident) granularity by adding one or more experienced teammates into the mix, double-checking your hunches and forming a collective hunch.

(4) Estimate by team hunch

Again, more people involved might mean better estimation, due to larger feedback. There is the obvious risk of “2 people, 3 opinions”, but at least you’ll have some other, non imaginary, voices of sense.

What does it all mean?

Each method offers some advantages and suffers from some inner faults. Understand the context and pick wisely. Accept the fact that things change and features might be thrown away – do not waste time on estimation more than is really required. More about features estimation to follow…

Planning your Sprint

With a given time window and people, how much can you pull off?

Many managers are requested to offer a plan. Knowing how much you can actually deliver is one of the toughest question managers need to face with. It’s too elusive to measure it correctly, and there are many moving parts. You need to please your customers and your teammates, creating a plan to deliver external and internal value, picking wisely what is really feasible.

I tried to come up with a few tips to make this process a bit more structured, making it less elusive, less daunting and less “we’ll see as we go”:

1. Pick by External Priority – what will make your customers happy? Make sure you have the right priority from product / business teams and give it high consideration in your planning. At the end of the day, you are here to provide value to your customers.

2. Pick by Internal Priority – making sure you are building confidence as part of your Sprint. You want to excel over time, you must devote the time to allow it.

3. Pick by Features Size – try to see how many features of each size you can deliver in a Sprint. For example, after 2-3 Sprints, you might notice that you’re usually able to do X big features, Y medium features and Z small features. Usually there is a connection between feature size and QA effort or deployment effort, thus making this size estimation very useful.

4. Pick by Velocity – if you know you’re able to complete X Story Points (or Ideal Days) every sprint, based on your last 2-3 Sprints, use this information to pick Features to fit this number give or take. I always believe that you should aim for ~120%, to allow positive pressure which encourages self-improvement ideas, but don’t aim for failure or naïve success. There is nothing to gain there.

5. Pick by Personal Growth – you might pick some features to allow inner growth in your team. For example, you might notice a need to strengthen one of your teammate’s leadership skills by allowing her/him to lead bigger features. It doesn’t mean you should act naively; organization needs is more important than personal whims, but the organization will never actively push toward personal growth as a goal. It’s up to you to make time for it.

Visibility over Progress

After talking quite a bit about how to break a Feature into tasks and avoiding leftovers, I would like to address the natural habit of what I call “blindly moving forward”. How often do you find yourself after 3 days of coding, still unclear of how many days are left or whether or not you’re closer to the end? How often you tend to forget what exactly you originally meant by “the end”? How often you find yourself feeling too invested to stop and figure out what the hack is going on?

We are, naturally, very outcome oriented. “I need this Feature done until tomorrow” or “I need to see your plans for next Sprint until end of day” keep pushing us during our daily effort. We are driven to supply outcomes, losing the ability to track the big picture because of vast, ever growing demand for results. Just like multi-threading coding, it’s not enough to throw some languages, tools or IDE with cool debugging options to the mix; we need to change the way we think and act, to supply visibility rather than mere progress.

When breaking a Feature, ask yourself these questions:

  1. Do you fully understand the Feature business value?
  2. Do you have Risks mapping (what can go wrong and what to do on such case)?
  3. Do you have the right people in-sync? (QA, Operations or other relevant team members)
  4. Do you break the Features by layers or by verticals? Are you OK with that?
  5. Do you have low granularity of tasks (up to 3-4 hours)? Enough to make you feel good about “no surprises”? Enough to allow others to join you?
  6. Do you know when you’ll be code-ready? Can you commit to a date?

As a Team leader, you should ask for visibility from your teammates. They should earn your confidence by offering a complete plan, knowing what the expected outcome is and communicate their status during the development time. If you ask me, I will always prefer better understanding of current status over naïve “I managed to progress today”.

As a developer, you should aspire for complete visibility to make sure you’re moving in the right path. This will allow others to give you honest feedback, join you if you’re a bit behind schedule and practice leadership in a way that is building confidence in your organization: people will feel that you know what you’re doing and will trust your good judgment and commitments. They will feel more confident when you’re behind the wheel.

If you feel that this is an overhead, you probably in a misconception of the time it will take you versus the time it will save you. Answering the above shouldn’t take 5 days; it should take couple of hours. This will ease your mind regarding “did I miss something?”, saving you hours of juggling between building confidence with your manager(“Can you update your progress?”, “Where are you standing?”) and actually building the software.

Adjusting big Features to Sprints

Many times when picking Features as execution unit, it’s hard to fit a big Feature into Sprint. This always cause the feeling of Features is too big as an execution units, leading some organization to prefer User Stories over Features, wrongfully neglecting Features over time. User Stories as execution unit are not the optimal way to go as they don’t provide enough value by themselves – you need to have “enough” Story Points to release a valuable version to your customers. They are also too abstract to understand the big picture, missing the overall why behind them. I believe that Feature broken into Tasks is a better approach and better execution unit. Obviously, there are multiple approaches to deal with big Features. You should pick the right tool according to requirements and people involved.

Breaking Big Feature, some options:

1. Extract design from execution: You can complete the design of a big Feature in Sprint 1 and implement it on Sprint 2. This way, instead of having a very big design effort and implementation effort on the same Sprint, you can use the first Sprint to understand effort and risks and use the second sprint to implement and deliver the goods, after understanding ROI better.

2. Extract framework from client: User Stories, just like Scrum in a way, introduce too many “Do’s and Don’ts”. For example, it’s considered a “bad practice” to have Technical User Stories. I believe that if the framework has specific needs (raised by the Feature) and it’s easy to break it out from the Feature itself (user behavior), it’s perfectly okay. It’s actually another step in the right way to build confidence as part of your Sprint!

3. Break big value to multiple smaller values (*caution*): Features, by nature, bring a value to your customers. It is possible though, to break the value into multiple Features, each carries only part of the value, without damaging the overall behavior. Instead of creating a “Search Engine” feature, it’s perfectly fine to define “People Search” and “Articles Search” as different features, each bringing different value. It’s even better to define sub Features inside “People Search” to enrich user experience over time, according to feedback by your customers. Caution: you need to do it carefully, sometimes breaking a Feature into multiple Features is silly as each doesn’t bring enough value by itself, until all of the pieces come together. Still, this is a powerful question to use: “Can we deliver real value to our customers by breaking this into multiple parts?”.

Most of the time, your Features should be small enough to fit a Sprint. It’s that simple – if the features are usually estimated in 2 weeks, don’t have a 1 week sprint. That being said, you should have the right (or at least some) tools to handle big Features. If it wasn’t obvious so far, don’t be afraid to “twist and mix Agile”. You owe it to yourself to understand why you’re doing things and adjust it if needed, even if it’s written in some book that it’s a “bad practice”. It might be bad for them, but it might be perfectly fine for you.

Beautiful Code?

I remember enjoying Jeff Atwood’s (aka Coding Horror) post about “code isn’t beautiful”. It took me a while explaining to myself why people are chasing after “beautiful code” so much. Why people are so passionate finding or forming beautiful code? Should great painters talk about beautiful colors or beautiful paintbrushes? I always thought they appreciate the entire painting, created and driven from emotions, dreams, ideas, styles and yes – colors & paintbrushes. They appreciate “the all” more then the exact technicalities. Why are we trying to figure out the best paintbrush to use or the best color? Why not trying to figure out the “best all”, the best experience, the best value, the best process?

I would define Beautiful Software as a surface to allow three magic abilities to emerge:

  1. It should be easy to add new Features.
  2. It should be easy to change existing Features.
  3. It should be easy for new teammate to become productive almost immediately.

If you’re able to have these abilities in your painting, your software, then you’re probably doing something beautiful; you probably figured out the correct paintbrushes, the right colors and most important – how to use them wisely to introduce true beauty.

I am not mentioning pseudo-code, language specific or whether or not testing is worthy. There are many ways to paint beautifully, it is up to you to figure out what makes software beautiful to you and figure out the right tools and process to achieve it. Don’t hang to tight to the “perfect algorithm” or “perfect Design Pattern”. There is nothing neither perfect nor beautiful about color or paintbrush by itself.

How to break Feature into Tasks: no leftovers!

Once you decide how to break a feature into tasks, it’s crucial to make sure you don’t leave any leftovers as part of the breakdown. Feature breakdown must include each and every task to make it a complete unit you can deploy as part of a version. Completeness means that not only a Feature is deployable and it can be monitored, it’s also with great external quality (no major bugs left behind) and great internal quality (design-review, QA test-cases review, code-review, UI review).

You want to price a Feature by the gross organization time; this is the only way to understand how much a Feature really costs. Knowing that, you can correctly price future roadmap without needing “Bugs Elimination Sprint” or “Big Refactoring Sprint” to introduce quality after things were delivered to your customers.

No leftovers:

1. Design review – may it be over emails or formal meeting, the design should be solid and well thought

2. QA test-cases review – making sure QA will test the right things or add more test cases for relevant edge-cases.

3. Testing time – writing tests (unit + integration) should be priced as part of the Feature, not after, not next sprint.

4. Code review – for knowledge sharing and internal quality validation.

5. UI review – making sure the UI look & feel just like Product team wanted it to be.

6. Bug fixing + validation buffer – fixing major bugs should be priced as well! if you’re not sure how much time to spend, start with 5-10% of the total estimated effort, and learn from your experience (by Feature size, complexity etc)

You can add more to this list, if the team/organization needs it. The idea is to make sure that once a Feature is complete, you’re feeling great about it and got confidence it won’t be fired back to the kitchen, to cook from scratch. You should have the same confidence about changing an existing feature; it should be easy, voodoo-less and highly testable to do so, with low chances of breaking other flows in the system.

Technical Features (in your) backlog

One of your responsibilities as a developer or team leader is to make sure you’ll be able to shine over time by constantly building confidence. This is why I strongly believe in maintaining a list of technical Features as part of your backlog. These are Features you believe will make the application more robust, provide better monitoring or make the team more productive (& happier) on their day-to-day work.

Have it to pass the message “we leave nothing as technical debt”

When someone in your team says “we really need to refactor this! it way too buggy and we cannot write tests for it”, don’t make it a technical debt that is written as a comment in the code or as a note on someone’s inbox. Make it a Feature, make it part of your backlog, pass the message that even though we cannot fix everything now, we will give it the right attention.

Have it to build confidence

Pick a few technical Features at each Sprint and make them a reality. This will make sure your team will be able to continue to produce fast and with great quality. Plan how much effort, from the entire Sprint, you would like to use to reduce waste – 1%? 5%? 10%? More? It’s really up to you. You need to balance it with thoughts of producing value to your customers while producing value to your team.

Have it for rough times

Having technical backlog is very useful when you need to re-plan the next few versions due to priority shift by the product teams, while your teammates are waiting for work. You can reduce pressure by letting them kill waste while you make certainty in the plans.

How to break Feature into Tasks

There are many ways to take a Feature and break it into workable unit of Tasks. Measuring “breakdown quality” is being ignored most of the time, as it’s very elusive to define “good breakdown”. The ability to break something into smaller pieces is one of the biggest signs of an experienced craftsman, so you should definitely challenge yourself by asking: am I known for predicting work effort well? detecting edge-cases early? Communicate progress to my boss clearly? being able to work with others on the same thing smoothly?

Introducing a Feature

Before we can play with breaking Feature into tasks, we need to have an example of a Feature in front of us. Let’s pretend that our desired Feature is “Signup and Sign-in” where to goal is to allow users to register and sign-in to the system. Next, here are the flows:
1. User can enter the signup page, if she’s not currently signed-in (else – redirect to homepage).
2. User can enter his details and signup.
3. User details will be validated, by [rules]. If failed – user will be asked to fix and re-send.
4. User will need to confirm her signup via email, before signing-in.
5. User can sign-in only after confirming her signup, if pending – show message.
6. User can sign-in in case she’s not signed-in already, using her credentials.
7. User can sign-out, only if she’s currently signed-in.
8. Page header needs to show relevant links, according to whether or not the user is currently signed-in.
9. If user fails to login multiple times (configurable), deny login for X minutes

[a few mockups here, a few technical/business notes there regarding URL structure, SEO to have in mind, analytics requirements, etc]

Breaking by Layers, the common way

A very common way is to break Feature by “application layers”:
1. Adjust database schema.
2. Create Data Access code to allow persistence of user (details, credentials encrypted etc) and retrieval of information.
3. Create Business Layer logic to validate credentials, confirm signup, perform sign-in.
4. Add signup and sign-in pages, alter header according to user state.
5. Add relevant validation on client side via javascript.

You can imagine that each task here will be a priced as 10-15 hours or even more, according to the implementer and the exact details. If the implementer founds an edge-case, it might mean that the every one of the layers here will be affected by it, making the estimation even less reliable.

Breaking by Verticals

A different approach will be using the flows above to form the tasks:
* create readConfiguration helper method
* create validateUserDetails method for server side
* create generateConfirmationNumber(userDetails) method
* create(/use) email sender provider
* create validateUserDetails for client-side
* create signup(userDetails) – schema to store user details, Data Access and Business to validate information and activate generateConfirmationNumber for sending confirmation email.
* create signup page (using validateUserDetails, signup method)
* create signin(credentials) method – schema to store number of login attempts, Data Access and Business to validate information, blocking user due to multiple attempts.
* create confirmUser method – Data Access and Bussiness to validate confirmation
* create signup confirmation page.
* create signin page
* create isLoggedIn method
* change header links according to status (via isLoggedIn)
* redirect user on entering signup page when currently signed-in (via isLoggedIn)

It’s not a 1:1 match between Task and flow, sometimes one flow will require multiple Tasks, but the flows drive the breakdown. No doubt, all of the tasks here are much smaller, each will take around 1-4 hours and very self-explanatory.

Why should you prefer Verticals breaking?

Confidence in achieving progress: if you strive for small tasks (up to 3-4 hours), it means that you’ll need to understand exactly what is needed to be done before committing to it. Every time you’ll finish a task, you’ll feel better that you’re moving on the right path. Because tasks are small, you will enjoy this feeling a few times during a day, which is important. You need to run away from “I wrote some code today, but I’m not sure if I’m on track”, poor planning will make you feel unproductive for long time.

Provide complete context to allow sharing workload: what happens if you need someone else to assist you? If you work by layers, it’s hard to understand “what is done”. The context is the entire Feature, which is too big to share easily. By working on vertical tasks, the context is much smaller as flows are smaller unit of execution. If you’re stuck, you can raise the flag and ask someone to assist you by taking a few vertical tasks and make them happen. They’ll have no trouble understanding what needs to be done.

Confidence building: make it part of your Sprint

One of the responsibilities of a great execution team is to make sure they will deliver in great speed and great quality, over time. “It’s a marathon, not a sprint” might be a confusing statement if you’re actually using Sprints in your development process ;)

What do I mean by “confidence building” and how does it affect you? Well, I believe that you need to earn others confidence over time, making sure your customers, internal and external, are happy with your results. For that you’ll need to make sure you understand what’s expected from you, to reach deadlines on time, to raise the Red Flag early and offer alternatives, to deliver product with great quality and to produce estimation that prove themselves as meaningful. Your job is to keep the execution machine at full speed and building the confidence as you go.

My recommendation is to make sure your Sprints will contain some internal maintainability time so you could stay on track rather than “have a good Sprint once in a while”. Here are a few thoughts:

  1. Bugs elimination – making sure the backlog is not overwhelming. Pick wisely, based on ROI given by product and technical teams.
  2. Provide rough estimation on the Features in the roadmap, review previous estimations and see if you were close.
  3. Push small POC for risky features to come.
  4. Create technical design for big/risky features you plan to address next sprint.
  5. Eliminating technical waste – small refactoring to enhance team productivity.

You can either treat them as features, or simply buffer some availability of the team to handle this.

No matter what, do not burnout the trust people have in you; your boss hired you because s/he trusted you to do well, it doesn’t mean you don’t have to work hard and continue to earn her/his confidence in you. It is a marathon, after all.

User Stories do not replace Features

User Stories became very trendy when Scrum became popular due to its smaller abstraction level; this made it possible to break big Feature and adjust its pieces into a single Sprint. I think this was taken too far, where User Stories sometimes completely replace Features. Not only I believe that Sprint should not be a release unit, I also believe that User Stories are not the correct work unit in a release or a Sprint:

1. Losing the why –User Stories by themselves are incomplete: they are missing the *why* – why do we think it is wise to develop this all experience? User Story describes only a small portion of a full experience. If we want to have real discussion on the value of a new capability, we must understand the full picture. The value always lies in the forest, not in the trees.

2. User Stories are usually not valuable to the customers by themselves – If you break a Feature into 20 User Stories, how many completed User Stories are enough to release the Feature? You cannot give your customer the Story of “A user can enter his password, the system will validate it by [rules]” without actually letting him register to the system. Your customer might enjoy tracking the Feature as you develop it this way, but you probably cannot release a version with 2 User Stories out of the 20 needed.

3. Maintaining another abstraction is expensive – what happens if a Feature changes? If you’re holding your User Stories separately, you’ll need to update them. Sometimes, as mistakes being made and it’s hard to sync papers, you’ll update one and not the other. So your business team, as an example, will work with the Feature paper but your QA team with the User Stories. Abstraction is smart only if it is beneficial over time.

I believe that User Stories are too pricey to be used as planning and execution units; they abstract a lot of the significant value Feature has to offer and introduce fragility in the process.

What User Stories are good for?

I would use User Stories only as a lingo between developers and business/marketing teams. They are small enough to be discussed without thorough background and big enough to explain behavior and expected outcome. I would not use User Stories to plan my sprints or my releases; I would aim to work in units I can deliver to my customers and discuss value of complete experiences.

How to adjust a big Feature into a Sprint?

More on it to follow…