divar-crawler/implementation-plan.md

Implementation Plan - Intelligent Divar Ads Crawler

This implementation plan outlines the phase-by-phase development of the Intelligent Divar Ads Crawler with AI Flagging and Telegram Notification. It is strictly based on the reference specifications: product-scenario.md and project-architecture.md.

User Review Required

[!IMPORTANT] The crawler will parse public search/category pages from Divar. In this MVP phase, we assume standard HTML/API structures of Divar. If Divar implements aggressive rate limiting or Cloudflare checks, we may need to introduce proxy lists, user-agent rotation, or captcha solvers in a later phase.

[!NOTE] For the LLM integration, we plan to default to gpt-4o-mini (or an equivalent cost-effective model like gemini-1.5-flash or claude-3-haiku depending on the .env configuration). We will use Structured Outputs (JSON Schema mode) to guarantee schema compliance.

Open Questions

[!NOTE] None at the moment. The technical specifications and requirements are fully defined in the reference documents. If any specific preference arises during implementation (e.g., custom bot formats or specific UI themes), it will be addressed.

---

Proposed Changes

Docker & Infrastructure Configuration

We will establish the docker containerization structure to orchestrate all services: PostgreSQL, Redis, Django Backend, Celery Worker, Celery Beat, and React Frontend.

[NEW] docker-compose.yml

• Defines the multi-container setup containing services: postgres, redis, backend, frontend, celery_worker, and celery_beat.
• Connects them via custom networks, environment variables, healthchecks, and volumes.

[NEW] .env.example

• Exposes templates for all necessary environment variables: DJANGO_SECRET_KEY, POSTGRES_*, REDIS_URL, LLM_API_*, TELEGRAM_BOT_TOKEN, and ALLOWED_HOSTS.

---

Backend Service

The backend will be a Django application using Django REST Framework (DRF), Celery for background processing, and Django-Celery-Beat for dynamic scheduler tasks.

[NEW] requirements.txt

• Defines dependencies: django, djangorestframework, django-cors-headers, celery, redis, psycopg2-binary, django-celery-beat, requests, openai, pydantic.

[NEW] Dockerfile

• Configures Python environment, installs requirements, and runs migration/development server.

[NEW] core/settings.py

• Configures Django applications, PostgreSQL connection, Redis caching/broker settings, Celery setup, and DRF authentication.

[NEW] core/celery.py

• Initializes Celery and connects it to Django settings.

[NEW] core/urls.py

• Root URL dispatcher that redirects API traffic to appropriate modules (/api/v1/crawlers/ and /api/v1/ads/).

[NEW] crawlers/models.py

• Implements CrawlTask and CrawlRun models with validation, constraints, and indexes.

[NEW] crawlers/tasks.py

• Implements run_crawl_pipeline task: fetches HTML/JSON from Divar, extracts ads, passes new ads for evaluation, handles exceptions, and updates CrawlRun.

[NEW] crawlers/serializers.py

• Implements serializers for CrawlTask and CrawlRun models.

[NEW] crawlers/views.py

• Implements API endpoints for Crawl CRUD, dynamic trigger (/trigger/), and execution logs.

[NEW] crawlers/urls.py

• Registers URL patterns for crawler endpoints.

[NEW] ads/models.py

• Implements Ad, AdEvaluation, and NotificationLog models with unique constraints and indexes.

[NEW] ads/tasks.py

• Implements evaluate_ad_with_ai task (calling LLM with JSON Schema) and send_telegram_notification task.

[NEW] ads/serializers.py

• Implements serializers for Ad, AdEvaluation, and NotificationLog models.

[NEW] ads/views.py

• Implements API endpoints for viewing and filtering ads (/api/v1/ads/).

[NEW] ads/urls.py

• Registers URL patterns for ads endpoints.

---

Frontend Service

The dashboard will be built using React with Vite. It interacts with the backend APIs to manage crawlers and view ads.

[NEW] package.json

• React, React-DOM, TailwindCSS/Vanilla CSS setup, Axios, Lucide React (for icons), and React Router DOM.

[NEW] Dockerfile

• Multi-stage build for development/production.

[NEW] vite.config.js

• Proxy setting for /api to avoid CORS issues in local development.

[NEW] index.html

• Main HTML landing page containing container root element.

[NEW] src/index.css

• Custom premium CSS styles and theme settings.

[NEW] src/App.jsx

• Setup routing and page layouts (Crawler Management Page, Ads Feed, Execution Log).

[NEW] src/main.jsx

• Entry point of the React application.

---

Verification Plan

Automated Verification

After building the docker environment, we will verify the services are active by running:

• docker compose ps - check that all 6 services are running.
• Run database migrations: docker compose exec backend python manage.py migrate
• Create superuser: docker compose exec backend python manage.py createsuperuser
• Check API health endpoint: curl http://localhost:8000/api/v1/health/

Manual Verification

1. Crawl Task CRUD: Log into the admin/dashboard, create a Crawl task with a specific search link (e.g. Tehran buy-apartment query), interval = 5 minutes, specific LLM prompt, and a valid Telegram Channel.
2. Dynamic Trigger: Click "Run Now" in the dashboard and monitor the logs (docker compose logs -f celery_worker).
3. Check PostgreSQL: Verify that crawled ads are stored in the database.
4. AI Evaluation Check: Verify that ads matching/not matching the prompt are correctly flagged is_flagged = true or false in AdEvaluation table.
5. Telegram Notification: Verify that only flagged ads are sent to the Telegram channel.
6. Timeframe Checks: Update the crawl task window to test that executions outside the hour range are skipped.