It's one thing to be a fast driver, and to be able to race with others who are close to your skill level, but you have to look at it this way: In any race series, there's always someone who's the fastest, and there are always drivers who are slower, and that fastest driver will ALWAYS encounter those slower drivers -- especially the slowest drivers -- when lapping them, and then that fastest driver has to be able to cope with that situation. 

There are fast drivers who are sometimes the very fastest, but they won't become winning drivers if they can't cope with  those lapped cars, and those lapped cars in the field will always be people who aren't able to perform at the same level as that fast driver, and if they can't perform at that level, then they are not going to be used to that fast driver going past the limits. 

The fast driver learns to cope with those situations and goes on to become a winning driver, or he ends up just being constantly messed-up by others and forever remains a fast driver who's not pulling in the wins. 

The key is that you've got to be ready for wherever people decide to brake -- in a Time Trial or testing session, you're driving the track; in a race, you're driving the track with other cars on the track at the same time, presenting moving (and movable) objects. 

You can't occupy the same space at the same time, and some turns are NOT typically passing zones/opportunities, so your pace through these turns and your brake points should be dictated by the car you're following. 

Nobody brakes early or late -- they just brake at different points than you do, and you have to be prepared to deal with that variable. Their "early" braking points may yield faster lap times than your "ideal" braking points, and if that's the case, then that "slow" car you plow into will really be the car that's faster than you because he knows a better line around the track. In that case, even if they're faster than you, they're still going to be going slower than you are when you plow into them, but the bottom line is that it's still YOUR fault for hitting THEM, not their fault for braking what you consider to be "early" based on your typical lap. 

When you're following another car, it's YOUR responsibility to adjust your driving style, your driving line, and your braking points to make sure that you don't hit them. 

Using this approach, you should never find yourself in a situation where you're going to be forced to do a quick swerve because you should be watching the car in front of you, the area around you to spot braking points, and the track in front of the car in front of you to make sure there are no potential obstacles coming up that you both may have to avoid (such as ANOTHER car off the track).