10. Gzip Compression: Enable Gzip compression on your own server to decrease the measurement of files transferred between the server along with the person’s browser. txt file is then parsed and may instruct the robotic as to which web pages will not be to be crawled. Like a internet search https://www.youtube.com/watch?v=aKirkq1uHGM