How to manage robots txt file for a website please share your view ?