divar-crawler/checklist.md

Project Checklist - Intelligent Divar Ads Crawler

This checklist tracks the implementation progress of the Intelligent Divar Ads Crawler with AI flagging and Telegram alerts.

1. Project Setup & Infrastructure

• Create Python virtual environment and set up requirements.txt
• Configure Docker environment (docker-compose.yml, Dockerfile.backend, frontend/Dockerfile, .dockerignore)
• Setup environment variables template (.env.example)
• Sanitize Docker shell script execution for Linux/Server build environments (entrypoint.sh CRLF/LF handling)

2. Backend Base & Models

• Implement core abstract base models (BaseModel, TimeStampedModel in core/models.py)
• Implement Crawler app models (CrawlTask and CrawlRun in crawler/models.py)
• Implement Ads app models (Ad, AdEvaluation, and NotificationLog in ads/models.py)
• Generate database migrations and apply them

3. API Views, Serializers, and Routing (DRF)

• Create API health check endpoint (/api/health/)
• Create CrawlTask and CrawlRun serializers and views
• Implement manual trigger action (/api/crawlers/<id>/trigger/) and runs history (/api/crawlers/<id>/runs/)
• Create Ad, AdEvaluation, and NotificationLog serializers and views
• Implement ads filtering by crawl_task and is_flagged (/api/ads/)
• Write backend unit tests for all implemented API endpoints and ensure they pass

4. Background Tasks & Scheduler (Celery)

• Configure Celery app and connect to Django settings (config/celery.py)
• Setup Celery dynamic database scheduler (django-celery-beat)
• Implement crawler pipeline task (run_crawl_pipeline in crawler/tasks.py)
• Add scraping/extraction logic for Divar public listings

5. AI Flagging & Telegram Notification

• Implement AI analysis with LLM Structured Outputs in ads/tasks.py
• Implement Telegram Bot notification sender in ads/tasks.py
• Connect full pipeline (Scrape -> AI Flag -> Telegram Alert)

6. Frontend Dashboard (React + Vite)

• Initialize React project in frontend/
• Create layout and custom CSS styling
• Build Crawler Management page (list, create, edit, trigger, log view)
• Build Ads Feed page (display filtered ads with AI evaluation details)
• Connect React frontend to Django REST APIs

7. Verification & Production Readiness

• Run full system manual QA (Edge cases, errors, network failure)
• Optimize logging, performance, line endings, and rate limiting
• Verify unit test suite execution (16/16 tests passing)