divar-crawler/implementation-plan.md

Implementation Plan - Intelligent Divar Ads Crawler

This implementation plan documents the architecture, infrastructure, and component structure of the Intelligent Divar Ads Crawler with AI Flagging and Telegram Notification. It is strictly based on the reference specifications: product-scenario.md and project-architecture.md.

User Review Required

[!IMPORTANT] The crawler parses public search/category pages from Divar. In this MVP phase, standard HTML/API structures of Divar are handled cleanly with robust fallbacks. If Divar implements aggressive rate limiting or Cloudflare checks in future versions, proxy lists or captcha solvers can be added.

[!NOTE] For LLM integration, OpenRouter API or OpenAI API models (gpt-4o-mini, openrouter/free) are supported using Structured Outputs (JSON Schema mode via Pydantic) to guarantee schema compliance.

---

Completed Architecture & Component Changes

Docker & Infrastructure Configuration

Multi-container orchestration for Linux machines and servers using Docker Compose.

docker-compose.yml

• Defines services: db (PostgreSQL 16 Alpine), redis (Redis 7 Alpine), backend (Django DRF), celery (Celery worker with Gevent), and frontend (React + Vite).
• Sets CELERY_TASK_ALWAYS_EAGER=False for containerized asynchronous worker execution via Redis.

Dockerfile.backend

• Configures Python 3.12 slim environment, installs build dependencies, cleans cache, and strips carriage returns (sed -i 's/\r$//') from entrypoint.sh for cross-platform Linux build compatibility.

.dockerignore

• Prevents transferring .git, .venv, .node, frontend/node_modules, frontend/dist, and SQLite databases to the Docker daemon.

.env.example

• Exposes templates for all necessary environment variables: DJANGO_SECRET_KEY, POSTGRES_*, REDIS_URL, OPENAI_API_KEY, OPENAI_MODEL, TELEGRAM_BOT_TOKEN, and ALLOWED_HOSTS.

---

Backend Service (Django + DRF + Celery)

requirements.txt

• Defines dependencies: django, djangorestframework, django-cors-headers, celery, redis, psycopg2, django-celery-beat, requests, openai, pydantic, python-dotenv, gevent.

backend/config/settings.py

• Configures Django applications, PostgreSQL connection (with SQLite fallback), Redis caching/broker settings, Celery setup, and DRF permissions.

backend/config/celery.py

• Initializes Celery app and auto-discovers tasks across backend modules.

backend/config/urls.py

• Main URL router dispatching API endpoints (/api/crawlers/, /api/ads/, /api/health/).

backend/crawler/models.py

• Implements CrawlTask and CrawlRun models with validation, constraints, and indexes.

backend/crawler/tasks.py

• Implements run_crawl_pipeline task: fetches HTML/JSON from Divar public search pages, extracts ads, passes new ads for AI evaluation, and logs execution details.

backend/crawler/views.py

• Implements REST API endpoints for Crawl CRUD, dynamic manual triggers (/trigger/), and run history logs.

backend/ads/models.py

• Implements Ad, AdEvaluation, and NotificationLog models with unique Divar token constraints.

backend/ads/tasks.py

• Implements evaluate_ad_with_ai task (calling LLM with JSON Schema) and send_telegram_notification task.

backend/ads/views.py

• Implements REST API endpoints for viewing and filtering ads (/api/ads/).

---

Frontend Service (React + Vite + TailwindCSS)

frontend/package.json

• React 18, Vite, TailwindCSS, Axios, Lucide React (for icons).

frontend/Dockerfile

• Node 22 slim image serving Vite development dashboard with --host 0.0.0.0.

frontend/vite.config.js

• Proxy setting for /api pointing to VITE_BACKEND_URL in Docker (http://backend:8000) or http://127.0.0.1:8000 in local dev.

frontend/src/App.jsx

• Complete single-page dashboard featuring Crawler Management, Ads Feed with AI Reasoning details, Execution Logs, Modal Editor, and System Health Monitor.

---

Verification Plan

Automated Verification

• Unit Test Suite: python backend/manage.py test backend -> 16/16 tests passing cleanly.
• Healthcheck Endpoint: GET /api/health/ -> Status 200 OK returning database and redis connectivity state.

Manual Verification

1. Linux & Server Docker Compose: Run docker compose up --build -d on Linux environment and verify containers divar_postgres, divar_redis, divar_backend, divar_celery, and divar_frontend start successfully.
2. Crawl Task CRUD & Manual Execution: Create a Crawl task in dashboard, click "Trigger", and verify logs update with scraped ad counts, AI evaluation flags, and Telegram alerts.