robots.txt Generator

It is a text file placed at the site root (yourdomain.com/robots.txt) that tells crawlers which parts they may or may not traverse. It is used to avoid wasting crawl budget on low-value sections (dashboards, filters, internal search results) and to point to your sitemap. It is guidance for well-behaved bots, not a security mechanism.

Not reliably. Disallow stops the bot crawling the page, but if other sites link to it, Google may still index the URL, showing it without a description. To keep a page out of the results, use the meta robots noindex tag or an X-Robots-Tag header, and let the crawler read it. Blocking it in robots.txt is exactly what prevents it seeing the noindex.

User-agent names the robot a block of rules targets (Googlebot, Bingbot, or * for all). Disallow marks the paths that robot should not crawl, and Allow permits exceptions inside a blocked path. For example, you can block a whole folder with Disallow and leave one specific file inside it reachable with Allow.

Yes, it is recommended. Adding a Sitemap line with the full URL of your sitemap helps search engines discover it even if you have not submitted it through their webmaster tools. You can declare several sitemaps, and the line is independent of the User-agent blocks.

It is case by case. Crawl-delay asks the robot to wait a few seconds between requests, useful for modest servers. Bing and some others honour it, but Googlebot ignores it: Google's crawl rate is adjusted from Search Console, not from robots.txt. Setting it does no harm, but do not count on Google obeying it.