feat: add initial robots.txt file to manage web crawler access

Author: kingRayhan

public/robots.txt

@@ -0,0 +1,21 @@
+# robots.txt for techdiary.dev
+
+# 1. Global rules for all bots
+User-agent: *
+Disallow: /dashboard
+Disallow: /backdoor
+
+# Allow important resources
+Allow: /css/
+Allow: /js/
+Allow: /images/
+Allow: /static/
+
+# 2. Optional: Add crawl delay for non-Google bots to reduce server load
+# Useful for Bingbot, Yandex, etc.; Googlebot ignores this.
+User-agent: Bingbot
+Crawl-delay: 10
+
+# 3. Sitemap location(s)
+Sitemap: https://www.techdiary.dev/sitemaps/articles/sitemap.xml
+Sitemap: https://www.techdiary.dev/sitemaps/profiles/sitemap.xml