divar-crawler/next-steps.md

Divar Crawler - Deployment & Next Steps Guide

This document provides clear instructions for running, testing, and deploying the Intelligent Divar Ads Crawler on local development machines (Windows) and production environments (Linux / Linux Server with Docker Compose).

---

1. Quick Start with Docker Compose (Linux / Linux Server - Recommended)

The simplest way to run the entire stack (PostgreSQL, Redis, Django API, Celery Worker, React Dashboard) on a Linux machine or server is via Docker Compose:

Step 1: Prepare Environment Variables

Copy .env.example to .env:

cp .env.example .env

Fill in your API credentials:

• OPENAI_API_KEY: Your OpenAI or OpenRouter API key.
• OPENAI_MODEL: Model identifier (e.g. openrouter/free or gpt-4o-mini).
• TELEGRAM_BOT_TOKEN: Telegram bot token from @BotFather.

Step 2: Start All Services

docker compose up --build -d

Step 3: Verify Status & Access Dashboard

• View running containers: docker compose ps
• View container logs: docker compose logs -f
• Open React Dashboard: http://<SERVER_IP>:5173/
• Check API Health: http://<SERVER_IP>:8000/api/health/

---

2. Local Windows Setup (Development Mode)

If developing locally without Docker on Windows:

Step 1: Configure .env

copy .env.example .env

Step 2: Start Django Backend

.venv\Scripts\Activate.ps1
pip install -r requirements.txt
python backend/manage.py migrate
python backend/manage.py runserver

Backend will run at http://127.0.0.1:8000/.

Step 3: Start React Frontend

In a new terminal:

cd frontend
..\.node\npm.cmd install
..\.node\npm.cmd run dev

Dashboard will run at http://localhost:5173/.

---

3. Running Unit Tests

Run the full Django test suite (covering crawlers, models, serializers, views, and health checks):

.venv\Scripts\python backend/manage.py test backend

---

4. Setting up a Crawl Task & Telegram Alert

1. Open the dashboard at http://localhost:5173/ (or http://<SERVER_IP>:5173/).
2. Click ایجاد کرال جدید (Create Task).
3. Provide:
   • Title: Descriptive name (e.g., Tehran Buy Apartment Under Market Value).
   • Divar Category URL: Divar public search URL (e.g., https://divar.ir/s/tehran/buy-apartment).
   • Detection Prompt: Evaluation criteria in Persian (e.g. "بررسی کن آیا آگهی فوری و زیر قیمت منطقه است یا خیر").
   • Telegram Channel: Bot target channel handle (e.g., @my_alerts_channel).
4. Save the task.
5. Click Trigger (Play button) next to the task to execute an immediate run and verify the scraping, AI evaluation, and Telegram notification pipeline.