Data scraping and reporting tool

A system that collects data from external sources on a schedule, normalises it, and reports on it through dashboards and exports. Check each source's terms and robots rules before collecting.

You can change every choice once the builder opens.

Core features for the MVP

Configurable sources with their own schedule and parser
Scheduled collection jobs with retry and backoff
Normalisation into a common record shape, with the raw payload kept
Deduplication and change detection between runs
Collection health monitoring, alerting when a source breaks
Dashboards with filtering over collected records
Scheduled exports delivered by email as CSV or PDF

Left for later

Alert rules that fire when a value crosses a threshold
Historical trend charts and comparisons
Public API over the collected data
Browser-based collection for sources that need JavaScript

Recommended stack

Section	Recommended
Language	TypeScriptPython
Frontend framework	Next.js
UI and styling	Tailwind CSSshadcn/ui
Backend	Next.js server
API style	REST
Database engine	PostgreSQL
Database provider	Neon
ORM and data access	Drizzle
Caching	Upstash Redis
Authentication	Clerk
Authorization and roles	Role-based access control
Security	Input validationRate limitingSecurity headers and CSPSecrets managementAudit logs
File storage	AWS S3
Background jobs and cron	Inngest
Email and notifications	Resend
Hosting and deployment	Vercel
CI/CD	GitHub Actions
Monitoring and analytics	SentryBetter Stack
Testing	VitestPlaywright
Extras	CSV and Excel import/exportReports and PDF exportBackupsDocumentationAccessibility (WCAG)

Data model draft

Source

One external place data is collected from.

id
name
url
parserKey
scheduleCron
isEnabled
lastRunAt
lastStatus

Has many CollectionRuns
Has many Records

CollectionRun

One execution against one source, successful or not.

id
sourceId
startedAt
finishedAt
status
recordsFound
recordsChanged
errorMessage

Belongs to one Source

Record

One normalised item, with its raw payload retained.

id
sourceId
externalId
normalisedData
rawPayload
contentHash
firstSeenAt
lastSeenAt

Belongs to one Source
Has many RecordVersions

RecordVersion

A snapshot kept when a record's content changes between runs.

id
recordId
normalisedData
contentHash
capturedAt

Belongs to one Record

ReportSchedule

A recurring export delivered to recipients.

id
name
query
format
cron
recipients
lastSentAt