InternOwlBot, our career-site crawler

Who we are

InternOwl is a free internship search for students. Every listing links to the employer's own posting, where students apply.

How to recognize the bot

It identifies itself as InternOwlBot/1.0 (+https://theinternowl.com/crawler; [email protected]). It never pretends to be a browser or another crawler.

What it does

• Reads your robots.txt first, and follows it: it fetches only allowed paths and honors Crawl-delay.
• Uses the sitemaps your robots.txt lists, and job pages' schema.org JobPosting markup — the information you publish for search engines.
• Fetches at most one page per second per site by default (slower if you ask), with a daily page limit per site, and only pages that are new or changed.
• Never signs in, creates accounts, submits forms or solves CAPTCHAs. If a site shows a challenge or answers 401 or 403, it stops and does not retry around it.

What it collects

Facts about internship postings (title, employer, location, dates, employment type), a short excerpt, and the link to your posting, unchanged. Students are always sent to your page to read the full posting and apply.

How to opt out

• Add to your robots.txt: User-agent: InternOwlBot then Disallow: /. The bot stops at its next visit.
• Or email [email protected] with your domain: we stop crawling it and hide its listings the same day. Employers can also use the For employers page (Claim your listings → Remove our listings).

Contact

Questions about the crawler, a request to slow it down, or a problem it caused: [email protected].