Preheat over to 350 degrees F. Line a 10"x8" cake pan with parchment paper. Combine 1 1/2 cups of the crumbled graham crackers, flour, baking powder and salt in a medium bowl to set aside.
In a separate bowl, beat the softened butter until creamy. Add sugar, vanilla, egg yolks, milk, and flour mixture and mix until thoroughly combined.
Beat egg whites with a hand mixer in a separate bowl until stiff peaks form and fold into the cake batter mixture a 1/3 cup at a time.
Pour batter into prepared cake pan and bake until golden, 20-25 minutes. Set aside to cool.