PouyaKhajavi
/
divar-crawler
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
12 Commits
2 Branches
0 Tags
278 KiB
 
 
 
 
 
 
Tree: 02204c706b
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '02204c706b'
${ noResults }
divar-crawler/checklist.md

2.3 KiB
Raw Blame History

Project Checklist - Intelligent Divar Ads Crawler

This checklist tracks the implementation progress of the Intelligent Divar Ads Crawler with AI flagging and Telegram alerts.

1. Project Setup & Infrastructure

  • Create Python virtual environment and set up requirements.txt
  • Configure Docker environment (docker-compose.yml & Dockerfiles)
  • Setup environment variables template (.env.example)

2. Backend Base & Models

  • Implement core abstract base models (BaseModel, TimeStampedModel in core/models.py)
  • Implement Crawler app models (CrawlTask and CrawlRun in crawler/models.py)
  • Implement Ads app models (Ad, AdEvaluation, and NotificationLog in ads/models.py)
  • Generate database migrations and apply them

3. API Views, Serializers, and Routing (DRF)

  • Create API health check endpoint (/api/health/)
  • Create CrawlTask and CrawlRun serializers and views
  • Implement manual trigger action (/api/crawlers/<id>/trigger/) and runs history (/api/crawlers/<id>/runs/)
  • Create Ad, AdEvaluation, and NotificationLog serializers and views
  • Implement ads filtering by crawl_task and is_flagged (/api/ads/)
  • Write backend unit tests for all implemented API endpoints and ensure they pass

4. Background Tasks & Scheduler (Celery)

  • Configure Celery app and connect to Django settings (core/celery.py)
  • Setup Celery dynamic database scheduler (django-celery-beat)
  • Implement crawler pipeline task (run_crawl_pipeline in crawler/tasks.py)
  • Add scraping/extraction logic for Divar public listings

5. AI Flagging & Telegram Notification

  • Implement AI analysis with LLM Structured Outputs in ads/tasks.py
  • Implement Telegram Bot notification sender in ads/tasks.py
  • Connect full pipeline (Scrape -> AI Flag -> Telegram Alert)

6. Frontend Dashboard (React + Vite)

  • Initialize React project in frontend/
  • Create layout and custom CSS styling
  • Build Crawler Management page (list, create, edit, trigger, log view)
  • Build Ads Feed page (display filtered ads with AI evaluation details)
  • Connect React frontend to Django REST APIs

7. Verification & Production Readiness

  • Run full system manual QA (Edge cases, errors, network failure)
  • Optimize logging, performance, and rate limiting