Bots and agents

Kernel identifies its bots and agents with Web Bot Auth. Each identity serves its own key directory from its own authority, so site owners can allow or block each one independently by purpose — for example, allow search indexing while blocking user-directed agents (or vice versa).

Kernel is listed in Cloudflare's bots and agents directory and Vercel's public directory.

Kernel Agent

User-directed browser automation. Kernel Agent visits pages on behalf of an end user's request; it is not an automatic crawler. Requests are signed with Web Bot Auth rather than a dedicated crawler user-agent token.

Kernel Search

Crawls pages to build search indexes and retrieval databases. Kernel Search identifies itself with the KernelSearchBot user-agent token and follows robots.txt directives for that token, including crawl-delay preferences.

Verifying Kernel traffic

Each identity publishes its public key set (JWKS) at its key directory. To verify a request:

1. Read the Signature-Agent header to determine which Kernel identity signed the request.
2. Fetch the public key set from that identity's key directory and cache it per the Cache-Control header.
3. Verify the Signature and Signature-Input headers per RFC 9421.

Most major bot-detection services, CDNs, and WAFs verify Web Bot Auth automatically. See Web Bot Auth for how Kernel signs requests.

Contact

For questions about Kernel bot or agent traffic, contact support@kernel.sh.