|
|
|
@ -0,0 +1,44 @@ |
|
|
|
# Project Checklist - Intelligent Divar Ads Crawler |
|
|
|
|
|
|
|
This checklist tracks the implementation progress of the Intelligent Divar Ads Crawler with AI flagging and Telegram alerts. |
|
|
|
|
|
|
|
## 1. Project Setup & Infrastructure |
|
|
|
- [x] Create Python virtual environment and set up `requirements.txt` |
|
|
|
- [ ] Configure Docker environment (`docker-compose.yml` & `Dockerfiles`) |
|
|
|
- [ ] Setup environment variables template (`.env.example`) |
|
|
|
|
|
|
|
## 2. Backend Base & Models |
|
|
|
- [x] Implement core abstract base models (`BaseModel`, `TimeStampedModel` in `core/models.py`) |
|
|
|
- [x] Implement Crawler app models (`CrawlTask` and `CrawlRun` in `crawler/models.py`) |
|
|
|
- [x] Implement Ads app models (`Ad`, `AdEvaluation`, and `NotificationLog` in `ads/models.py`) |
|
|
|
- [x] Generate database migrations and apply them |
|
|
|
|
|
|
|
## 3. API Views, Serializers, and Routing (DRF) |
|
|
|
- [x] Create API health check endpoint (`/api/health/`) |
|
|
|
- [x] Create `CrawlTask` and `CrawlRun` serializers and views |
|
|
|
- [x] Implement manual trigger action (`/api/crawlers/<id>/trigger/`) and runs history (`/api/crawlers/<id>/runs/`) |
|
|
|
- [x] Create `Ad`, `AdEvaluation`, and `NotificationLog` serializers and views |
|
|
|
- [x] Implement ads filtering by `crawl_task` and `is_flagged` (`/api/ads/`) |
|
|
|
- [x] Write backend unit tests for all implemented API endpoints and ensure they pass |
|
|
|
|
|
|
|
## 4. Background Tasks & Scheduler (Celery) |
|
|
|
- [ ] Configure Celery app and connect to Django settings (`core/celery.py`) |
|
|
|
- [ ] Setup Celery dynamic database scheduler (`django-celery-beat`) |
|
|
|
- [ ] Implement crawler pipeline task (`run_crawl_pipeline` in `crawler/tasks.py`) |
|
|
|
- [ ] Add scraping/extraction logic for Divar public listings |
|
|
|
|
|
|
|
## 5. AI Flagging & Telegram Notification |
|
|
|
- [ ] Implement AI analysis with LLM Structured Outputs in `ads/tasks.py` |
|
|
|
- [ ] Implement Telegram Bot notification sender in `ads/tasks.py` |
|
|
|
- [ ] Connect full pipeline (Scrape -> AI Flag -> Telegram Alert) |
|
|
|
|
|
|
|
## 6. Frontend Dashboard (React + Vite) |
|
|
|
- [ ] Initialize React project in `frontend/` |
|
|
|
- [ ] Create layout and custom CSS styling |
|
|
|
- [ ] Build Crawler Management page (list, create, edit, trigger, log view) |
|
|
|
- [ ] Build Ads Feed page (display filtered ads with AI evaluation details) |
|
|
|
- [ ] Connect React frontend to Django REST APIs |
|
|
|
|
|
|
|
## 7. Verification & Production Readiness |
|
|
|
- [ ] Run full system manual QA (Edge cases, errors, network failure) |
|
|
|
- [ ] Optimize logging, performance, and rate limiting |
PouyaKhajavi
2 days ago