After bashing prediction systems, claiming that measuring performance of your teammates to predict release dates or personal behavior is inherently wrong, you might deduce that I meant “measuring Actual Hours is pointless”.

On the contrary my friend, I believe that tracking Actual Hours is imperative for better planning and cutting off waste.

Translate internal units (Story Points / Ideal Days) to external units (Calendar)

How can you translate a 3 Story Points or a 3 Ideal Days to “it will be ready next Tuesday?”
Using Actual Hours, you could actually see a correlation between Story Points size (internal to the organization) to total amount of Actual Hours Range (external as you can put it on calendar). For example, you might notice after a few sprints that 3 Story Points (based on latest 5 features), are usually taking 23-28 Actual Hours to complete. Assuming you pick a number, say 6, as the amount of Actual Hours in 1 day a developer can do, such features will translate to around 4-5 working days. Story Points can be translated quickly, based on empirical knowledge, to a range of working days. This will make release dates predication much easier, assisting your Project Manager and your Marketing team to communicate it in their roadmap or create plans accordingly.

Actual Hours help to quickly balance effort in a Sprint

You’re on the first day of the Sprint and you’ve got seven Features (/ User Stories) in “Not Started”. You also made sure that people availability (Joe got 50 hours this sprint) is updated as some have vacations and some got exams during the sprint. Even more, you already decided who is the owner of each Feature. How can you balance the tasks effort, in hours, between your teammates? how can you decide if some features will be developed by multiple developers and still see that your plan is feasible? 
With Actual Hours history, from the example above, you can see that 3 Story Points translate to 23-28 hours. You can immediately add to all features worth 3 Story Points one task called TBD with 25 hours (~avg) and assign it to the owner. You move to each one of your features, checking their Story Points -> Actual Hours range, adding the task. Next, you slowly break this big TBD task, in each feature, into multiple TBD tasks, each with different owner and estimation, until you see that the effort is balanced between all of your teammates.

2 flows for example:

a. 3 Story Points (range: 23-28 hours) -> 1 task worth 25h for Joe -> 1 task worth 10 for Joe, 1 task worth 5 for Jim and 1 task worth 10 for Annie.

b. 2 Story Points (range: 10-18 hours) -> 1 task worth 14h for Jim -> 1 task worth 10 for Jim and 1 task worth 4 for Annie.

c. [ … until all work effort are balanced by availability and optimal ownership … ]
d. Make sure that your breakdown makes sense in terms of people availability. Make changes if needed.

Summary

Actual Hours are imperative as they (1) provide a way to translate internal work units to external work units and (2) allowing fast effort breakdown. The key is to set expectations right with your teammates, explaining why their performance measurements are so crucial and how they will be used. They need to understand so they will be able to honest, to have an incentive to help you. Don’t break their trust.