To make the crust, mix butter, cocoa, cocoa nibs, sugar, salt, and vanilla in a medium bowl. Add flour and mix until blended. Press the dough evenly over the bottom and up the sides of a 10 or 11-inch tart pan with a removable bottom. Generously prick the bottom and sides of the crust with the tines of a fork. Bake in the lower third of an oven preheated to 350° F until the crust feels firm to the touch, about 20 to 25 minutes. Cool completely.
For the filling, combine milk and marshmallows in the top of a double boiler or a bowl that fits snugly over a sauce pan. Heat and stir over gently simmering water until marshmallows melt. Remove from heat to cool, stirring every 5 minutes or so, until lukewarm. Combine liqueurs and add to the marshmallows, blending well.
Whip cream until stiff peaks form. Fold marshmallow mixture into the whipped cream until well combined. Transfer the mixture to the cooled crust and smooth the top with a spatula. Freeze until firm, several hours to overnight.
Serve frozen or slightly thawed topped with lightly sweetened whipped cream and cocoa nibs, chocolate shavings, or sprinkles, if desired.