We do not agree with the unknown. And to increase the tension here, you, the writer, must show us his growing fear. Show us his wide eyes, show the eye nerves beneath his pure optical lens, his shallow breaths and careful steps. Maybe even ask him to get a gun, because we know that gun won't help him in the end. Find and fix any broken links. Broken links are not good for search engine crawlers or your website visitors, so finding and fixing them as quickly as possible is always a good idea. You should also consider regularly checking your website for broken links to ensure you can remove them when they appear.0. You can make the API call in front-end instead of doing it server-side. For example, when the page loads, make an AJAX request to your server which will call the API and return the data. Presumably the bots and crawlers are just parsing the source code and not executing the JS, so they won't execute the AJAX request and you will. Of course, the web page also has dynamics usually written in PHP or ASP, for example, the user login interface is a dynamic web page, but if a spider web collapses, the spider will feel less. Put these URLs in a queue Browse the queue, read the URLs in the queue one by one, for each URL, crawl the corresponding web page, then repeat the crawling process above. Check if the stop condition is met. If the stop condition is not set, the crawler will continue crawling until it cannot obtain a new URL. It performs milliseconds to the robot between each request. Fast and easy Spatie crawler provides options features to set maximum crawl depth, response size, add delay between queries, limit crawl links, limit types of content to analyze, etc., which simplifies the web scraping process. Although this is an introductory article, you.