# Project Checklist - Intelligent Divar Ads Crawler

This checklist tracks the implementation progress of the Intelligent Divar Ads Crawler with AI flagging and Telegram alerts.

## 1. Project Setup & Infrastructure
- [x] Create Python virtual environment and set up `requirements.txt`
- [ ] Configure Docker environment (`docker-compose.yml` & `Dockerfiles`)
- [x] Setup environment variables template (`.env.example`)

## 2. Backend Base & Models
- [x] Implement core abstract base models (`BaseModel`, `TimeStampedModel` in `core/models.py`)
- [x] Implement Crawler app models (`CrawlTask` and `CrawlRun` in `crawler/models.py`)
- [x] Implement Ads app models (`Ad`, `AdEvaluation`, and `NotificationLog` in `ads/models.py`)
- [x] Generate database migrations and apply them

## 3. API Views, Serializers, and Routing (DRF)
- [x] Create API health check endpoint (`/api/health/`)
- [x] Create `CrawlTask` and `CrawlRun` serializers and views
- [x] Implement manual trigger action (`/api/crawlers/<id>/trigger/`) and runs history (`/api/crawlers/<id>/runs/`)
- [x] Create `Ad`, `AdEvaluation`, and `NotificationLog` serializers and views
- [x] Implement ads filtering by `crawl_task` and `is_flagged` (`/api/ads/`)
- [x] Write backend unit tests for all implemented API endpoints and ensure they pass

## 4. Background Tasks & Scheduler (Celery)
- [ ] Configure Celery app and connect to Django settings (`core/celery.py`)
- [ ] Setup Celery dynamic database scheduler (`django-celery-beat`)
- [ ] Implement crawler pipeline task (`run_crawl_pipeline` in `crawler/tasks.py`)
- [ ] Add scraping/extraction logic for Divar public listings

## 5. AI Flagging & Telegram Notification
- [ ] Implement AI analysis with LLM Structured Outputs in `ads/tasks.py`
- [ ] Implement Telegram Bot notification sender in `ads/tasks.py`
- [ ] Connect full pipeline (Scrape -> AI Flag -> Telegram Alert)

## 6. Frontend Dashboard (React + Vite)
- [x] Initialize React project in `frontend/`
- [x] Create layout and custom CSS styling
- [x] Build Crawler Management page (list, create, edit, trigger, log view)
- [x] Build Ads Feed page (display filtered ads with AI evaluation details)
- [x] Connect React frontend to Django REST APIs

## 7. Verification & Production Readiness
- [ ] Run full system manual QA (Edge cases, errors, network failure)
- [ ] Optimize logging, performance, and rate limiting