# Implementation Plan - Intelligent Divar Ads Crawler

This implementation plan documents the architecture, infrastructure, and component structure of the **Intelligent Divar Ads Crawler** with AI Flagging and Telegram Notification. It is strictly based on the reference specifications: [product-scenario.md](file:///c:/projects/divar-crawler/product-scenario.md) and [project-architecture.md](file:///c:/projects/divar-crawler/project-architecture.md).

## User Review Required

> [!IMPORTANT]
> The crawler parses public search/category pages from Divar. In this MVP phase, standard HTML/API structures of Divar are handled cleanly with robust fallbacks. If Divar implements aggressive rate limiting or Cloudflare checks in future versions, proxy lists or captcha solvers can be added.

> [!NOTE]
> For LLM integration, OpenRouter API or OpenAI API models (`gpt-4o-mini`, `openrouter/free`) are supported using Structured Outputs (JSON Schema mode via Pydantic) to guarantee schema compliance.

---

## Completed Architecture & Component Changes

### Docker & Infrastructure Configuration

Multi-container orchestration for Linux machines and servers using Docker Compose.

#### [docker-compose.yml](file:///c:/projects/divar-crawler/docker-compose.yml)
- Defines services: `db` (PostgreSQL 16 Alpine), `redis` (Redis 7 Alpine), `backend` (Django DRF), `celery` (Celery worker with Gevent), and `frontend` (React + Vite).
- Sets `CELERY_TASK_ALWAYS_EAGER=False` for containerized asynchronous worker execution via Redis.

#### [Dockerfile.backend](file:///c:/projects/divar-crawler/Dockerfile.backend)
- Configures Python 3.12 slim environment, installs build dependencies, cleans cache, and strips carriage returns (`sed -i 's/\r$//'`) from `entrypoint.sh` for cross-platform Linux build compatibility.

#### [.dockerignore](file:///c:/projects/divar-crawler/.dockerignore)
- Prevents transferring `.git`, `.venv`, `.node`, `frontend/node_modules`, `frontend/dist`, and SQLite databases to the Docker daemon.

#### [.env.example](file:///c:/projects/divar-crawler/.env.example)
- Exposes templates for all necessary environment variables: `DJANGO_SECRET_KEY`, `POSTGRES_*`, `REDIS_URL`, `OPENAI_API_KEY`, `OPENAI_MODEL`, `TELEGRAM_BOT_TOKEN`, and `ALLOWED_HOSTS`.

---

### Backend Service (Django + DRF + Celery)

#### [requirements.txt](file:///c:/projects/divar-crawler/requirements.txt)
- Defines dependencies: `django`, `djangorestframework`, `django-cors-headers`, `celery`, `redis`, `psycopg2`, `django-celery-beat`, `requests`, `openai`, `pydantic`, `python-dotenv`, `gevent`.

#### [backend/config/settings.py](file:///c:/projects/divar-crawler/backend/config/settings.py)
- Configures Django applications, PostgreSQL connection (with SQLite fallback), Redis caching/broker settings, Celery setup, and DRF permissions.

#### [backend/config/celery.py](file:///c:/projects/divar-crawler/backend/config/celery.py)
- Initializes Celery app and auto-discovers tasks across backend modules.

#### [backend/config/urls.py](file:///c:/projects/divar-crawler/backend/config/urls.py)
- Main URL router dispatching API endpoints (`/api/crawlers/`, `/api/ads/`, `/api/health/`).

#### [backend/crawler/models.py](file:///c:/projects/divar-crawler/backend/crawler/models.py)
- Implements `CrawlTask` and `CrawlRun` models with validation, constraints, and indexes.

#### [backend/crawler/tasks.py](file:///c:/projects/divar-crawler/backend/crawler/tasks.py)
- Implements `run_crawl_pipeline` task: fetches HTML/JSON from Divar public search pages, extracts ads, passes new ads for AI evaluation, and logs execution details.

#### [backend/crawler/views.py](file:///c:/projects/divar-crawler/backend/crawler/views.py)
- Implements REST API endpoints for Crawl CRUD, dynamic manual triggers (`/trigger/`), and run history logs.

#### [backend/ads/models.py](file:///c:/projects/divar-crawler/backend/ads/models.py)
- Implements `Ad`, `AdEvaluation`, and `NotificationLog` models with unique Divar token constraints.

#### [backend/ads/tasks.py](file:///c:/projects/divar-crawler/backend/ads/tasks.py)
- Implements `evaluate_ad_with_ai` task (calling LLM with JSON Schema) and `send_telegram_notification` task.

#### [backend/ads/views.py](file:///c:/projects/divar-crawler/backend/ads/views.py)
- Implements REST API endpoints for viewing and filtering ads (`/api/ads/`).

---

### Frontend Service (React + Vite + TailwindCSS)

#### [frontend/package.json](file:///c:/projects/divar-crawler/frontend/package.json)
- React 18, Vite, TailwindCSS, Axios, Lucide React (for icons).

#### [frontend/Dockerfile](file:///c:/projects/divar-crawler/frontend/Dockerfile)
- Node 22 slim image serving Vite development dashboard with `--host 0.0.0.0`.

#### [frontend/vite.config.js](file:///c:/projects/divar-crawler/frontend/vite.config.js)
- Proxy setting for `/api` pointing to `VITE_BACKEND_URL` in Docker (`http://backend:8000`) or `http://127.0.0.1:8000` in local dev.

#### [frontend/src/App.jsx](file:///c:/projects/divar-crawler/frontend/src/App.jsx)
- Complete single-page dashboard featuring Crawler Management, Ads Feed with AI Reasoning details, Execution Logs, Modal Editor, and System Health Monitor.

---

## Verification Plan

### Automated Verification
- Unit Test Suite:
  `python backend/manage.py test backend` -> 16/16 tests passing cleanly.
- Healthcheck Endpoint:
  `GET /api/health/` -> Status 200 OK returning database and redis connectivity state.

### Manual Verification
1. **Linux & Server Docker Compose**:
   Run `docker compose up --build -d` on Linux environment and verify containers `divar_postgres`, `divar_redis`, `divar_backend`, `divar_celery`, and `divar_frontend` start successfully.
2. **Crawl Task CRUD & Manual Execution**:
   Create a Crawl task in dashboard, click "Trigger", and verify logs update with scraped ad counts, AI evaluation flags, and Telegram alerts.