Web data collection is one of the few areas where the tooling has genuinely outrun the governance. Bright Data (6.9) makes collection at scale straightforward, and n8n (8.2) makes scheduling and routing it straightforward. Neither will tell you whether you are allowed to.
So this guide starts where the project should start, which is not with the pipeline.
Answer these before you build anything
Three questions, and they are not rhetorical:
- What does the target site's terms of service say, and are you bound by them?
- Does the data include personal information — and if so, what is your lawful basis for processing it under the regimes that apply to you?
- What happens to your business if this source blocks you or changes structure next month?
Why the third question matters as much as the first two
Teams treat blocking as a technical problem to route around. It is better understood as a dependency risk. A pipeline feeding a product feature from a source that has not agreed to supply you is a feature with an owner who can switch it off.
If the data is load-bearing, get it under an agreement or a paid API. If it is genuinely exploratory, scraping is a reasonable tool. The mistake is quietly letting the exploratory thing become load-bearing.
What Bright Data is for
Bright Data scores 6.9 on our 361 score. It provides proxy infrastructure at large scale, scraping APIs, and pre-built datasets. The pre-built datasets are the most overlooked part of the offering and often the correct answer.
Before building a collector, check whether the dataset already exists. Buying data you were about to scrape is usually cheaper once you count engineering time, and it moves the compliance question to a vendor who has already answered it.
What n8n is for here
n8n is the orchestration layer: schedule the collection, handle pagination, route results into storage, alert when a run fails, and trigger downstream processing.
Keep the workflow thin. The scraping logic and the parsing belong in a service or in Bright Data's own tooling; the workflow should decide when things run and where results go, not carry the extraction rules. Parsing logic inside a visual workflow becomes unmaintainable faster than almost anything else.
The design details that save you later
- Store the raw response, not just your parsed output. When the parser turns out to be wrong, you want to re-parse rather than re-collect.
- Record collection time with every record. Data without a timestamp cannot be reasoned about later.
- Make failures loud. A pipeline that silently returns zero rows will quietly poison every downstream number for weeks.
- Rate-limit deliberately, below what you can technically get away with. Aggressive collection is how a tolerated activity becomes a blocked one.
The honest cost
Collection is the cheap part. The expensive part is maintenance: sites change structure, parsers break, and someone has to notice and fix it. Budget ongoing time, not just setup time.
This is why our score for Bright Data sits at 6.9 despite serious infrastructure — the platform is capable, and the surrounding work is heavier than the marketing for this category ever admits.