Swap out the lemon cake mix for white cake mix, if desired.
Heat oven to 375 degrees F. Line 2 cupcake pans with paper baking cups. Make the cake as directed on the box. Cool completely on a cooling rack, about 30 minutes.
Meanwhile, make lemon buttercream by beating the butter in a large bowl with an electric mixer on medium speed until smooth. Add the powdered sugar, 4 tablespoons of the lemon juice, and 2 tablespoons of the milk. Beat until smooth, adding more lemon juice and milk as necessary to reach a smooth, spreadable consistency.
Spoon the buttercream into a decorating bag fitted with large open star tip. Pipe the frosting onto the cupcakes. Top each cupcake with coarse decorating sugar and a PEEPS® chick. Cut about 4 inches off bottom of each straw; discard. Poke a straw into each cupcake.