Tag

robots.txt

1 views collected around this technical thread.

Sohu Tech Products
Sohu Tech Products
Oct 6, 2021 · Frontend Development

Front‑End SEO Technical Optimization Guide

This article presents a comprehensive front‑end SEO checklist, covering passive and active optimization techniques such as site structure, meta tags, semantic links, speed improvements, external traffic acquisition, sitemaps, robots.txt, and search‑engine‑specific configurations to help developers enhance website visibility and ranking.

FrontendMeta Tagsrobots.txt
0 likes · 13 min read
Front‑End SEO Technical Optimization Guide
Python Programming Learning Circle
Python Programming Learning Circle
Jan 2, 2020 · Backend Development

How to Crawl Responsibly: Avoid Legal Risks and Server Overload

This guide outlines responsible web‑crawling practices, covering robots.txt compliance, legal pitfalls such as unauthorized personal data and copyrighted content, recommended request intervals, and relevant Chinese data‑security regulations, helping developers avoid server overloads and potential lawsuits.

Backend DevelopmentScrapydata ethics
0 likes · 4 min read
How to Crawl Responsibly: Avoid Legal Risks and Server Overload