Search Engine Spider UAs
Collection of common search engine spider/crawler User-Agent strings.
Tool Guide
Tool Introduction
The Online Directory of Common Search Engine Spider/Crawler User-Agents is a quick-reference dictionary of web crawler identifiers frequently used by webmasters, SEO specialists, and server operations engineers. It fully includes the latest official standard User-Agent strings of Googlebot, Baiduspider, Bingbot, YandexBot, and mainstream social sharing crawlers, and supports one-click search and instant copy.
Core Features
- Full coverage of mainstream search engine crawlers: Includes spider/crawler UA header information for various scenarios such as web pages, images, mobile, video, and ad retrieval, with versions kept up to date in real time.
- Instant, high-precision smart search: Type a specific spider brand into the search box (such as "Baidu" or "Google"), and the corresponding list of UA strings is instantly filtered locally.
- Click-to-copy feature: Using an extremely simple interaction mechanism, just click any UA string and the system copies it perfectly to the system clipboard with one click—making Nginx debugging extremely convenient.
How to Use
- Open the Common Spider UA Directory page, where you can scroll up and down to browse the full list of included search engine crawler User-Agents.
- In the search box, type the name of the crawler you need to configure or troubleshoot (such as "Baidu"), and the system filters the output in milliseconds.
- Tap the UA string, and "Copied successfully" will appear above/to the right, allowing you to paste it directly into your Nginx configuration file to prevent scraping and rate abuse.
Related Tools
Emoji Reference
Complete emoji reference with online lookup and copy functionality.
Special Symbols
Collection of special symbols including emoji, math symbols, arrows and special text.
HTML Color Code Table
Common HTML color code table for web development with HEX/RGB values and preview.
ASCII Table
Complete ASCII code table with control characters, printable characters and extended ASCII codes.
MIME Type Reference
A comprehensive MIME type (Multipurpose Internet Mail Extensions) reference. Look up the standard Content-Type for any file extension (.json, .pdf, .mp4) along with its transfer format.
HTML Entity Reference
A quick-reference table of common HTML character entities. Look up and copy the named, decimal and hexadecimal codes for special characters (non-breaking space, copyright sign, Greek letters) to avoid markup conflicts.