Judge Guide

GeckoRegen

GeckoRegen is the validation gate that makes Bright Data production-safe. BD’s Self-Healing API returns a new scraper — it does not verify that the new scraper actually works. We re-run and schema-check before accepting.

90 seconds — say this

  1. Problem: Regulator sites change. Scrapers die silently. Missed rules = fines.
  2. We are not a scraper. Bright Data scrapes. We govern: detect → heal → validate → remember.
  3. Point at Runtime: MONITOR → DETECT → HEAL → VALIDATE → MEMORY. That strip is the real Python path lighting up.
  4. Live proof: FCA = 751 real records. Simulate Break hits BD Test Shop (public URL). ESMA/BaFin/MAS are labelled fallback.
  5. Differentiator: We do not trust BD’s heal. Re-run, population ≥ 90%, then accept.

Open dashboard

Architecture — how a click becomes data

server.pyAPI + SSEFlask
monitor.pyOrchestratorThread pool
bd_client.pyBright DataStudio + Heal
detector.pyHealthJSONL diff
healer.pyGateValidate heal
memory.pySQLitedb.py

Request path: Browser → POST /api/scan or /api/break → monitor.monitor_regulator → bd_client.trigger_scraper (/dca/trigger) → poll /dca/dataset → detector.detect_failures → if broken, healer.heal_pipeline (refactor_template) → validate → memory.record_* → SSE /api/events updates the dashboard.

We do not use Unlocker, SERP, or Scraping Browser. Product pick: Scraper Studio + Self-Healing API.

Code flow (one scan)

  1. web/app.js — Simulate Break → POST /api/break then POST /api/scan for BD Test Shop
  2. server.py — cooldown lock, background thread, SSE scan_started
  3. monitor.run_scraper — bd_client.trigger_scraper + get_dataset
  4. detector.save_snapshot — append data/snapshots/{id}.jsonl
  5. detector.detect_failures — null / type / regex vs schema
  6. db.update_health — healthy / degraded / broken
  7. memory.record_change — every record into changes
  8. If broken: healer.generate_heal_prompt → BD Self-Healing → _validate_records (pop ≥ 90%, types, count). Max 3 attempts.
  9. adapter.py — optional: raise scan frequency if the regulator is noisy
  10. SSE scan_completed — Runtime strip walks Monitor → Detect → (Heal → Validate) → Memory

What is live vs fallback

Live Bright Data collectors: FCA (751 records), FINMA, BD Test Shop (https://ecommerce-shop-brd.vercel.app — public, BD can reach it).

Fallback (pre-recorded): ESMA, BaFin, MAS (active: 0). FINMA also gets fallback rows/health if it has never been scraped, so the card is never empty. Rows are tagged snapshot_id = fallback:synthetic. MAS is BD-blocked.

Simulate Break swaps local sandbox/index.html (view at /sandbox) so you can see the redesign. The following BD scan hits the public Test Shop URL — not localhost. Cloud scrapers cannot fetch your machine.

How to Read the Health Grid

Each regulator is a glass node. The amber strip and status label tell you scraper health:

Healthy — fields populated
Degraded — partial failure
Broken — healing required

On break the node pulses error-red. On heal it pulses amber. Updates arrive live over SSE.

What Happens When a Scraper Breaks

  1. Detector diffs current output against the last-known-good JSONL snapshot
  2. Healer builds a contextual prompt and sends it to Bright Data’s Self-Healing API
  3. BD regenerates scraper code. GeckoRegen polls until the job is ready
  4. Validator re-runs the scraper: population ≥ 90%, format consistent, record count stable
  5. Pass → health turns healthy. Fail → sharper prompt, max 3 attempts
  6. Memory stores what broke, which prompt worked, and how long it took

Step-by-Step Demo

1
Dashboard loads — health grid shows regulators. Summary metrics fill the header glass.
2
Click Simulate Break — POST /api/break swaps sandbox HTML (preview /sandbox), then scans BD Test Shop on its public URL.
3
Scraper breaks — the node strip turns error-red. Missing fields show in meta.
4
GeckoRegen detects — Anomaly Stream logs the snapshot diff and severity.
5
GeckoRegen heals — contextual prompt to BD Self-Healing API. Node shows HEALING.
6
GeckoRegen validates — re-run + schema gate. Accept or reject the heal.
7
Monitoring resumes — node returns to amber-healthy. Memory records the event.
8
Self-Healing Timeline — success or failure, duration, attempt count.

Impact correlation (implemented)

Memory does not embed documents. It scores historical changes with SequenceMatcher on title (50%), category (30%), and summary (20%). Exposed as GET /api/similar?title=...&category=....

Worked example from this dataset: ESMA fallback row “Guidelines on crypto-asset service providers and data retention” (category: guidance) is the same family as FCA live items on crypto guidance and data-retention themes. The API returns the nearest titles with a similarity score and a note (same category, similar title).

That is the GDPR-style claim, scoped to what the code actually does: “this looks like a guidance/retention change you have already seen,” not a legal opinion engine.

Pre-recorded heal (FCA)

If credits or time block a live heal, the timeline can show a prior FCA run: field summary null → contextual prompt → validation population 94.11% on 751 records → accepted in 1 attempt, 47s. Marked in validation_details as pre-recorded prior BD heal run.

Modules

Run Demo