Competitor research: what to keep in-house, what to brief

What to take away

A Manchester software team is planning autumn content around three rival product blogs. The finance director wants the research bought in; the marketing lead has two analysts and a customer panel.

Buying the work suits teams who brief a new agency each quarter, because coding and templates stay comparable.
Keeping it in-house pays when you already hold analysts and customer access.
Either route needs a written scopecompetitor set, pages, questions, evidence format.
Automated collection must respect the crawl rules in a site's robots.txt file.
Keep raw evidence and diary a refresh date, or the study dates within two quarters.

Decide what the research must answer

Write the question before the brief. "What is our competitor doing?" is not one. "Which competitor pages rank for the buying terms our sales team hears, and what proof do they show?" is.

Define the research question

Write the question before the brief
Set competitor listfour to six domains
Name pages in scopepricing, product
Include comparison, case study, help centre
Name the audience the research serves

If you cannot name the audience the research serves, settle that first. The audience research guide for England sets out the baseline questions worth answering before a brief goes out.

Split the work between team and agency

In-house strengths are customer access and internal data. Your sales calls and support tickets are evidence no external researcher can buy. Agency strengths are speed, repeatable templates and pattern recognition across clients.

In-house vs agency split

In-house

Strengths: Customer access, internal data
Evidence: Sales calls, support tickets
Keep in-house: Named customers, contracts, roadmap
Hand over: —

Agency

Strengths: Speed, repeatable templates
Evidence: Public competitor pages
Keep in-house: —
Hand over: Page collection, keyword mapping, tables

A sensible split: the agency collects and codes public competitor pages, your team supplies customer language from calls and surveys.

Task	Better in-house	Better briefed out
Research question and competitor set	Yes: needs sales and product input	No
Customer language from calls and tickets	Yes: private data stays inside	No
Collecting and saving public pages	Possible with Screaming Frog	Yes: repeatable templates
Coding pages against categories	Possible for a small set	Yes: second coder and consistency checks
Keyword and ranking mapping	Possible with Search Console	Yes: paid tool access across clients
Interpreting results for the roadmap	Yes: needs unpublished plans	No

Hand tools commonly compared include Google Search Console (free), Ahrefs (paid plans from around $29 per month), Semrush (paid plans from around $140 per month), Similarweb (paid), and Screaming Frog (free up to 500 URLs, paid from around £199 per year).

On the agency side, the content agencies listed in the Story Growth directory are the place to build a shortlist; ask each for a sample coded spreadsheet from a past study before you compare quotes. A typical study covers four to six domains and 50 to 150 pages.

What to keep in-house

Anything touching named customers, contract terms or an unpublished roadmap. The handover cost usually exceeds the saving.

What to hand over

Public page collection, keyword mapping and structured comparison tables. These are mechanical, repeatable and easy to check against source URLS.

Set the rules before collection starts

Public pages, published prices and open job adverts are fair game; private dashboards and leaked documents are not.

Claims you publish off the back of this research fall under consumer protection and competition enforcement, which the Competition and Markets Authority oversees. Comparisons with named rivals are also governed by the Business Protection from Misleading Marketing Regulations 2008 and by the CAP Code rules on comparative advertising, which the Advertising Standards Authority enforces. If a comparison page implies a rival's product fails a test, you need the test.

Any crawler an agency runs should follow the site's robots.txt file, the Robots Exclusion Protocol, and its disallow rules in particular. Ask the agency to log which user agent it crawls as and to keep its request rate low enough not to load the site.

Judge the evidence quality

Ask for a source URL and capture date against every claim. Two analysts reading the same page should reach the same coded result, so test that on ten pages before the full run.

If the agency only reads landing pages, you learn nothing about retention or support content; add help centre and documentation pages to the scope.

Decide who the results describe before anyone is briefed, using the same audience definition as the research question above.

Weigh the skills you are actually buying

Competitor research increasingly includes AI-assisted summarising and clustering. The CIM AI marketing courses cover the training route for marketers who need to run and question that analysis rather than accept it.

Ask who reviews the output.

Plan the handover and refresh

Insist on the working file, not just the finished report. A spreadsheet with columns for competitor, URL, date, claim and evidence takes an hour to inherit. A slide deck takes a week to reverse-engineer.

Then set a refresh cadence: quarterly for pricing and comparison pages, twice yearly for editorial themes. Diary the date when you sign the contract.

Glossary

Competitor set: the four to six domains compared in one study.
Coded page: a competitor page tagged against agreed categories so results count.
Capture date: the day a page was saved, showing how current a finding is.
Crawl directive: a rule in robots.txt telling automated agents what they may fetch.
Refresh cadence: the scheduled interval for repeating the study.

Common questions

Is competitor research cheaper in-house?

Only if your analysts have spare capacity and customer access. Work out the in-house cost as analyst days multiplied by a loaded day rate (salary plus employer costs and overheads), add tool subscriptions, and compare that total with an agency quote for the same scope.

Can an agency crawl competitor sites for US?

Yes, within the site's robots.txt rules and your own legal advice. Ask for the directive check in writing before work starts.

How often should the study repeat?

Quarterly for pricing and comparison content, twice yearly for editorial themes.

What should the brief contain?

Competitor set, pages in scope, questions to answer, evidence format, capture dates and the refresh date.