# Implementation Plan - Intelligent Divar Ads Crawler

This implementation plan outlines the phase-by-phase development of the **Intelligent Divar Ads Crawler** with AI Flagging and Telegram Notification. It is strictly based on the reference specifications: [product-scenario.md](file:///c:/projects/divar-crawler/product-scenario.md) and [project-architecture.md](file:///c:/projects/divar-crawler/project-architecture.md).

## User Review Required

> [!IMPORTANT]
> The crawler will parse public search/category pages from Divar. In this MVP phase, we assume standard HTML/API structures of Divar. If Divar implements aggressive rate limiting or Cloudflare checks, we may need to introduce proxy lists, user-agent rotation, or captcha solvers in a later phase.

> [!NOTE]
> For the LLM integration, we plan to default to `gpt-4o-mini` (or an equivalent cost-effective model like `gemini-1.5-flash` or `claude-3-haiku` depending on the `.env` configuration). We will use Structured Outputs (JSON Schema mode) to guarantee schema compliance.

## Open Questions

> [!NOTE]
> None at the moment. The technical specifications and requirements are fully defined in the reference documents. If any specific preference arises during implementation (e.g., custom bot formats or specific UI themes), it will be addressed.

---

## Proposed Changes

### Docker & Infrastructure Configuration

We will establish the docker containerization structure to orchestrate all services: PostgreSQL, Redis, Django Backend, Celery Worker, Celery Beat, and React Frontend.

#### [NEW] [docker-compose.yml](file:///c:/projects/divar-crawler/docker-compose.yml)
- Defines the multi-container setup containing services: `postgres`, `redis`, `backend`, `frontend`, `celery_worker`, and `celery_beat`.
- Connects them via custom networks, environment variables, healthchecks, and volumes.

#### [NEW] [.env.example](file:///c:/projects/divar-crawler/.env.example)
- Exposes templates for all necessary environment variables: `DJANGO_SECRET_KEY`, `POSTGRES_*`, `REDIS_URL`, `LLM_API_*`, `TELEGRAM_BOT_TOKEN`, and `ALLOWED_HOSTS`.

---

### Backend Service

The backend will be a Django application using Django REST Framework (DRF), Celery for background processing, and Django-Celery-Beat for dynamic scheduler tasks.

#### [NEW] [requirements.txt](file:///c:/projects/divar-crawler/backend/requirements.txt)
- Defines dependencies: `django`, `djangorestframework`, `django-cors-headers`, `celery`, `redis`, `psycopg2-binary`, `django-celery-beat`, `requests`, `openai`, `pydantic`.

#### [NEW] [Dockerfile](file:///c:/projects/divar-crawler/backend/Dockerfile)
- Configures Python environment, installs requirements, and runs migration/development server.

#### [NEW] [core/settings.py](file:///c:/projects/divar-crawler/backend/core/settings.py)
- Configures Django applications, PostgreSQL connection, Redis caching/broker settings, Celery setup, and DRF authentication.

#### [NEW] [core/celery.py](file:///c:/projects/divar-crawler/backend/core/celery.py)
- Initializes Celery and connects it to Django settings.

#### [NEW] [core/urls.py](file:///c:/projects/divar-crawler/backend/core/urls.py)
- Root URL dispatcher that redirects API traffic to appropriate modules (`/api/v1/crawlers/` and `/api/v1/ads/`).

#### [NEW] [crawlers/models.py](file:///c:/projects/divar-crawler/backend/crawlers/models.py)
- Implements `CrawlTask` and `CrawlRun` models with validation, constraints, and indexes.

#### [NEW] [crawlers/tasks.py](file:///c:/projects/divar-crawler/backend/crawlers/tasks.py)
- Implements `run_crawl_pipeline` task: fetches HTML/JSON from Divar, extracts ads, passes new ads for evaluation, handles exceptions, and updates `CrawlRun`.

#### [NEW] [crawlers/serializers.py](file:///c:/projects/divar-crawler/backend/crawlers/serializers.py)
- Implements serializers for `CrawlTask` and `CrawlRun` models.

#### [NEW] [crawlers/views.py](file:///c:/projects/divar-crawler/backend/crawlers/views.py)
- Implements API endpoints for Crawl CRUD, dynamic trigger (`/trigger/`), and execution logs.

#### [NEW] [crawlers/urls.py](file:///c:/projects/divar-crawler/backend/crawlers/urls.py)
- Registers URL patterns for crawler endpoints.

#### [NEW] [ads/models.py](file:///c:/projects/divar-crawler/backend/ads/models.py)
- Implements `Ad`, `AdEvaluation`, and `NotificationLog` models with unique constraints and indexes.

#### [NEW] [ads/tasks.py](file:///c:/projects/divar-crawler/backend/ads/tasks.py)
- Implements `evaluate_ad_with_ai` task (calling LLM with JSON Schema) and `send_telegram_notification` task.

#### [NEW] [ads/serializers.py](file:///c:/projects/divar-crawler/backend/ads/serializers.py)
- Implements serializers for `Ad`, `AdEvaluation`, and `NotificationLog` models.

#### [NEW] [ads/views.py](file:///c:/projects/divar-crawler/backend/ads/views.py)
- Implements API endpoints for viewing and filtering ads (`/api/v1/ads/`).

#### [NEW] [ads/urls.py](file:///c:/projects/divar-crawler/backend/ads/urls.py)
- Registers URL patterns for ads endpoints.

---

### Frontend Service

The dashboard will be built using React with Vite. It interacts with the backend APIs to manage crawlers and view ads.

#### [NEW] [package.json](file:///c:/projects/divar-crawler/frontend/package.json)
- React, React-DOM, TailwindCSS/Vanilla CSS setup, Axios, Lucide React (for icons), and React Router DOM.

#### [NEW] [Dockerfile](file:///c:/projects/divar-crawler/frontend/Dockerfile)
- Multi-stage build for development/production.

#### [NEW] [vite.config.js](file:///c:/projects/divar-crawler/frontend/vite.config.js)
- Proxy setting for `/api` to avoid CORS issues in local development.

#### [NEW] [index.html](file:///c:/projects/divar-crawler/frontend/index.html)
- Main HTML landing page containing container root element.

#### [NEW] [src/index.css](file:///c:/projects/divar-crawler/frontend/src/index.css)
- Custom premium CSS styles and theme settings.

#### [NEW] [src/App.jsx](file:///c:/projects/divar-crawler/frontend/src/App.jsx)
- Setup routing and page layouts (Crawler Management Page, Ads Feed, Execution Log).

#### [NEW] [src/main.jsx](file:///c:/projects/divar-crawler/frontend/src/main.jsx)
- Entry point of the React application.

---

## Verification Plan

### Automated Verification
After building the docker environment, we will verify the services are active by running:
- `docker compose ps` - check that all 6 services are running.
- Run database migrations: `docker compose exec backend python manage.py migrate`
- Create superuser: `docker compose exec backend python manage.py createsuperuser`
- Check API health endpoint: `curl http://localhost:8000/api/v1/health/`

### Manual Verification
1. **Crawl Task CRUD:** Log into the admin/dashboard, create a Crawl task with a specific search link (e.g. Tehran buy-apartment query), interval = 5 minutes, specific LLM prompt, and a valid Telegram Channel.
2. **Dynamic Trigger:** Click "Run Now" in the dashboard and monitor the logs (`docker compose logs -f celery_worker`).
3. **Check PostgreSQL:** Verify that crawled ads are stored in the database.
4. **AI Evaluation Check:** Verify that ads matching/not matching the prompt are correctly flagged `is_flagged = true` or `false` in `AdEvaluation` table.
5. **Telegram Notification:** Verify that only flagged ads are sent to the Telegram channel.
6. **Timeframe Checks:** Update the crawl task window to test that executions outside the hour range are skipped.