How to replace all files with actual File Template in android studio

Found it out, I think! So what you do is you go to the more options by pressing right arrow and click "Run Inspection on..." 

Then, click "whole project" and run the inspection. You'll get a table that looks like this:

Turns out you can just spam the link circled in red, and it'll replace all the headers! Not sure if this is the way to do it, but this worked for me.