Ten. Gzip Compression: Permit Gzip compression on your server to reduce the size of files transferred involving the server as well as consumer’s browser. txt file is then parsed and will instruct the robotic concerning which webpages will not be to be crawled. As being a internet search engine crawler https://www.youtube.com/watch?v=4q42p8bnJy8