Melt butter in a heavy bottomed 2 quart saucepan over medium-low heart, stirring as needed.
Stir in the brown sugar and cook for 2 minutes, stirring constantly.
Gently pour in the milk and stirring constantly, bring mixture to a boil.
Remove from heat and let cool for 30 to 40 minutes, or until the mixture is lukewarm or at room temperature.
Stir in the vanilla extract.
Stir in powdered sugar, one to one-half cup at a time, stirring until completely combined. Add powdered sugar just until the mixture is thick and has the consistency of fudge. (It wall harden as it chills.)
Stir in coarsely chopped pecans.
Spoon into the prepared loaf pan and top with additional pecans if desired.
Cover and cool until firm.
Cut into squares and store in the refrigerator in an airtight container.