Preheat the oven to 350.
With an electric mixer cream together the butter, sugar, and vanilla.
In a separate bowl combine flour, cocoa powder, baking powder, baking soda and salt.
Gradually add in the dry ingredients and milk and mix until well incorporated. This is a fairly dry dough but if it is too crumbly add in more milk a teaspoon at a time.
Using a dough scoop or spoon scoop out dough and roll it into balls. Place the balls about 2 inches apart on a cookie sheet and flatten them to about ¼ of an inch with the bottom of a glass (You can dip it in sugar if it sticks) Or for cookies with smoother edges you can flatten them by hand and then place them on the cookie sheet.
Bake for 11-13 minutes (until the tops appear dry) allow to cool for 3 minutes on the baking sheet and then transfer to a wire rack to cool completely.
After the cookies have cooled it is time to frost them.
Add the butter, cocoa powder, and milk to a saucepan and heat over medium heat. Stir and heat until the butter is melted, mix in the powdered sugar, and continue stirring until well incorporated and smooth.
One cookie at a time spoon frosting on and quickly top with marshmallow bits, the frosting sets quickly so working this way allows the marshmallows to stick. If the frosting thickens too much while frosting the cookies just re-heat it until it softens again.
Store the cookies in an airtight container.