dovodi
/
Dovodi_Backend
3
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Browse Source

Add management commands for seeding basic and comprehensive Hadis data 

- Implemented `seed_basic_data.py` to seed essential Hadis app models including HadisStatus, HadisTag, and HadisSect with initial records.
- Created `seed_hadis_data.py` for comprehensive data seeding, establishing realistic sample records for all Hadis app models while maintaining relationships and business logic.
- Introduced retry logic for database operations to handle potential locks and integrity errors during seeding.
- Added a test command `test_sects.py` to verify the creation of HadisSect records and check for existing sects.
master
nwhco 11 months ago
parent
7452d3ba3d
commit
adf73181b5
8 changed files with 1636 additions and 119 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 1
      apps/hadis/management/__init__.py
  2. 72
      apps/hadis/management/commands/README.md
  3. 1
      apps/hadis/management/commands/__init__.py
  4. 34
      apps/hadis/management/commands/fix_sects.py
  5. 138
      apps/hadis/management/commands/seed_basic_data.py
  6. 934
      apps/hadis/management/commands/seed_hadis_data.py
  7. 45
      apps/hadis/management/commands/test_sects.py
  8. 530
      scripts/seed_hadis_data.py

1
apps/hadis/management/__init__.py
View File

@ -0,0 +1 @@

72
apps/hadis/management/commands/README.md
View File

@ -0,0 +1,72 @@
# Hadis Management Commands
## seed_hadis_data
This management command seeds comprehensive data for all Hadis app models with realistic sample records while maintaining proper relationships and business domain logic.
### Usage
```bash
# Basic usage - seed data with default settings
python manage.py seed_hadis_data
# Clear existing data before seeding
python manage.py seed_hadis_data --clear
# Specify custom images directory
python manage.py seed_hadis_data --images-dir /path/to/images
# Specify custom XMind file
python manage.py seed_hadis_data --xmind-file /path/to/file.xmind
# Combine options
python manage.py seed_hadis_data --clear --images-dir scripts/seed_images --xmind-file scripts/test.xmind
```
### Options
- `--clear`: Clear existing hadis data before seeding (optional)
- `--images-dir`: Directory containing seed images (default: scripts/seed_images)
- `--xmind-file`: Path to XMind file for categories (default: scripts/test.xmind)
### What it creates
1. **HadisStatus records**: Various hadis authenticity statuses (Достоверный, Хороший, etc.)
2. **HadisTag records**: Topic tags for categorizing hadis
3. **HadisSect records**: Shia and Sunni sects
4. **HadisCategory records**: Hierarchical categories for both Quran and Hadith sources
5. **Library data**: Books, categories, and collections for references
6. **Transmitters**: Historical figures who transmitted hadis
7. **Hadis records**: Complete hadis with translations, explanations, and relationships
8. **Transmission chains**: Links between hadis and transmitters
9. **References**: Book references with images
### Requirements
- The images directory must contain PNG files for book covers and reference images
- The XMind file is optional but recommended for category mind maps
- All models must be properly migrated before running
### Performance
The command uses optimized batch operations to create data efficiently:
- Bulk create/update operations for categories
- Checks for existing records to avoid duplicates
- Progress reporting for large datasets
### Example Output
```
Starting Hadis data seeding...
Found 4 seed images
XMind file: scripts/test.xmind
Creating Hadis Statuses...
Created status: Достоверный
Created status: Хороший
...
Creating Hadis Categories...
Creating categories for Шииты-двунадесятники...
Batch created 6 Quran categories
...
Successfully seeded all Hadis data!
```

1
apps/hadis/management/commands/__init__.py
View File

@ -0,0 +1 @@

34
apps/hadis/management/commands/fix_sects.py
View File

@ -0,0 +1,34 @@
"""
Fix sects creation issue
"""
from django.core.management.base import BaseCommand
from apps.hadis.models import HadisSect
class Command(BaseCommand):
help = 'Fix sects creation'
def handle(self, **options):
self.stdout.write("Fixing sects...")
# Delete any problematic sects
HadisSect.objects.filter(sect_type='sunni').delete()
# Create sects with simple titles
sects_data = [
{'sect_type': 'shia', 'title': 'Shia', 'is_active': True, 'order': 1},
{'sect_type': 'sunni', 'title': 'Sunni', 'is_active': True, 'order': 2},
]
for data in sects_data:
sect, created = HadisSect.objects.get_or_create(
sect_type=data['sect_type'],
defaults=data
)
if created:
self.stdout.write(f"Created: {sect.sect_type} - {sect.title}")
else:
self.stdout.write(f"Exists: {sect.sect_type} - {sect.title}")
self.stdout.write(self.style.SUCCESS("Sects fixed!"))

138
apps/hadis/management/commands/seed_basic_data.py
View File

@ -0,0 +1,138 @@
"""
Basic data seeding management command for Hadis app models.
This command creates only the essential records needed for the app to function.
"""
from django.core.management.base import BaseCommand, CommandError
from django.db import transaction
# Import models
from apps.hadis.models import HadisSect, HadisStatus, HadisTag
class Command(BaseCommand):
help = 'Seed basic data for Hadis app models'
def add_arguments(self, parser):
parser.add_argument(
'--clear',
action='store_true',
help='Clear existing basic data before seeding',
)
def handle(self, **options):
if options['clear']:
self.clear_existing_data()
try:
with transaction.atomic():
self.stdout.write(
self.style.SUCCESS('Starting basic Hadis data seeding...')
)
# Seed basic data
statuses = self.seed_hadis_statuses()
tags = self.seed_hadis_tags()
sects = self.seed_hadis_sects()
self.stdout.write(
self.style.SUCCESS(
f'Successfully seeded basic data: '
f'{len(statuses)} statuses, {len(tags)} tags, {len(sects)} sects'
)
)
except Exception as e:
self.stdout.write(
self.style.ERROR(f'Error during seeding: {str(e)}')
)
raise CommandError(f'Seeding failed: {str(e)}')
def clear_existing_data(self):
"""Clear existing basic data"""
self.stdout.write("Clearing existing basic data...")
HadisSect.objects.all().delete()
HadisStatus.objects.all().delete()
HadisTag.objects.all().delete()
self.stdout.write("Basic data cleared.")
def seed_hadis_statuses(self):
"""Create HadisStatus records"""
self.stdout.write("Creating Hadis Statuses...")
statuses_data = [
{'title': 'Достоверный', 'color': 'green', 'order': 1},
{'title': 'Хороший', 'color': 'blue', 'order': 2},
{'title': 'Слабый', 'color': 'yellow', 'order': 3},
{'title': 'Выдуманный', 'color': 'red', 'order': 4},
]
statuses = []
for data in statuses_data:
status, created = HadisStatus.objects.get_or_create(
title=data['title'],
defaults=data
)
statuses.append(status)
if created:
self.stdout.write(f" Created status: {status.title}")
return statuses
def seed_hadis_tags(self):
"""Create HadisTag records"""
self.stdout.write("Creating Hadis Tags...")
tags_data = [
'Поклонение', 'Молитва', 'Пост', 'Хадж', 'Закят',
'Нравственность', 'Терпение', 'Справедливость',
'Фикх', 'Предписания', 'Толкование', 'Коран',
'Имамат', 'Мольба', 'Единобожие'
]
tags = []
for tag_title in tags_data:
tag, created = HadisTag.objects.get_or_create(
title=tag_title,
defaults={'status': True}
)
tags.append(tag)
if created:
self.stdout.write(f" Created tag: {tag.title}")
return tags
def seed_hadis_sects(self):
"""Create HadisSect records"""
self.stdout.write("Creating Hadis Sects...")
sects_data = [
{'sect_type': 'shia', 'title': 'Шииты-двунадесятники', 'is_active': True, 'order': 1},
{'sect_type': 'sunni', 'title': 'Сунниты', 'is_active': True, 'order': 2},
]
sects = []
for data in sects_data:
self.stdout.write(f" Processing sect: {data['sect_type']}")
# Check if sect exists
try:
sect = HadisSect.objects.get(sect_type=data['sect_type'])
self.stdout.write(f" Sect already exists: {sect.title}")
sects.append(sect)
except HadisSect.DoesNotExist:
# Create new sect
self.stdout.write(f" Creating new sect: {data['sect_type']}")
sect = HadisSect(
sect_type=data['sect_type'],
title=data['title'],
is_active=data['is_active'],
order=data['order']
)
sect.save()
self.stdout.write(f" Created sect: {sect.title}")
sects.append(sect)
return sects

934
apps/hadis/management/commands/seed_hadis_data.py
View File

@ -0,0 +1,934 @@
"""
Comprehensive data seeding management command for Hadis app models.
This command creates realistic sample records for all Hadis app models
while maintaining proper relationships and business domain logic.
"""
import random
import time
from pathlib import Path
from django.core.management.base import BaseCommand, CommandError
from django.core.files import File
from django.core.files.base import ContentFile
from django.db import connection
from django.db.utils import OperationalError, IntegrityError
# Import models
from apps.hadis.models import (
HadisSect, HadisCategory, HadisStatus, HadisTag, Hadis,
Transmitters, HadisTransmitter, HadisReference, ReferenceImage
)
from apps.library.models import Book, Category as LibraryCategory, BookCollection
class Command(BaseCommand):
help = 'Seed comprehensive data for Hadis app models'
def add_arguments(self, parser):
parser.add_argument(
'--clear',
action='store_true',
help='Clear existing hadis data before seeding',
)
parser.add_argument(
'--images-dir',
type=str,
default='scripts/seed_images',
help='Directory containing seed images (default: scripts/seed_images)',
)
parser.add_argument(
'--xmind-file',
type=str,
default='scripts/test.xmind',
help='Path to XMind file (default: scripts/test.xmind)',
)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.seed_images_dir = None
self.xmind_file_path = None
self.image_files = []
self.retry_delay = 2 # seconds
self.max_retries = 5
def handle(self, **options):
self.setup_paths(options)
# Check database status before starting
self.stdout.write("🔍 Checking database status...")
self.check_database_locks()
if options['clear']:
self.stdout.write("🧹 Clearing existing data...")
self.safe_execute_with_retry("Clear existing data", self.clear_existing_data)
try:
self.stdout.write(
self.style.SUCCESS('🚀 Starting Hadis data seeding...')
)
# Seed data in proper order WITHOUT transaction.atomic() to avoid locks
self.stdout.write("📊 Step 1: Creating basic lookup data...")
statuses = self.safe_execute_with_retry("Seed statuses", self.seed_hadis_statuses)
self.stdout.write("🏷️ Step 2: Creating tags...")
tags = self.safe_execute_with_retry("Seed tags", self.seed_hadis_tags)
self.stdout.write("🕌 Step 3: Creating sects...")
sects = self.safe_execute_with_retry("Seed sects", self.seed_hadis_sects)
self.stdout.write("📚 Step 4: Creating categories...")
categories = self.safe_execute_with_retry("Seed categories", self.seed_hadis_categories, sects)
self.stdout.write("📖 Step 5: Creating library data...")
books, _, _ = self.safe_execute_with_retry("Seed library data", self.seed_library_data)
self.stdout.write("👥 Step 6: Creating transmitters...")
transmitters = self.safe_execute_with_retry("Seed transmitters", self.seed_transmitters)
self.stdout.write("📜 Step 7: Creating hadis records...")
self.safe_execute_with_retry("Seed hadis records", self.seed_hadis_records,
categories, statuses, tags, transmitters, books)
self.stdout.write(
self.style.SUCCESS('✅ Successfully seeded all Hadis data!')
)
except Exception as e:
self.stdout.write(
self.style.ERROR(f'❌ Error during seeding: {str(e)}')
)
import traceback
self.stdout.write(self.style.ERROR(traceback.format_exc()))
raise CommandError(f'Seeding failed: {str(e)}')
def setup_paths(self, options):
"""Setup file paths and verify required files exist"""
self.seed_images_dir = Path(options['images_dir'])
self.xmind_file_path = Path(options['xmind_file'])
# Verify required files exist
if not self.seed_images_dir.exists():
raise CommandError(f"Seed images directory not found: {self.seed_images_dir}")
# Get available images
self.image_files = list(self.seed_images_dir.glob('*.png'))
if not self.image_files:
raise CommandError("No PNG images found in seed_images directory")
self.stdout.write(f"Found {len(self.image_files)} seed images")
if self.xmind_file_path.exists():
self.stdout.write(f"XMind file: {self.xmind_file_path}")
else:
self.stdout.write(self.style.WARNING(f"XMind file not found: {self.xmind_file_path}"))
def clear_existing_data(self):
"""Clear existing hadis data (optional - for clean seeding)"""
self.stdout.write("Clearing existing hadis data...")
# Clear in reverse dependency order
ReferenceImage.objects.all().delete()
HadisReference.objects.all().delete()
HadisTransmitter.objects.all().delete()
Hadis.objects.all().delete()
HadisCategory.objects.all().delete()
HadisSect.objects.all().delete()
HadisStatus.objects.all().delete()
HadisTag.objects.all().delete()
Transmitters.objects.all().delete()
self.stdout.write("Existing data cleared.")
def safe_execute_with_retry(self, operation_name, operation_func, *args, **kwargs):
"""Execute database operation with retry logic for handling locks"""
for attempt in range(self.max_retries):
try:
self.stdout.write(f" Attempting {operation_name} (attempt {attempt + 1}/{self.max_retries})")
result = operation_func(*args, **kwargs)
self.stdout.write(f" ✓ {operation_name} completed successfully")
return result
except OperationalError as e:
error_msg = str(e).lower()
if 'database is locked' in error_msg or 'deadlock' in error_msg:
self.stdout.write(
self.style.WARNING(
f" ⚠ Database lock detected in {operation_name}: {str(e)}"
)
)
if attempt < self.max_retries - 1:
self.stdout.write(f" ⏳ Waiting {self.retry_delay} seconds before retry...")
time.sleep(self.retry_delay)
# Increase delay for next attempt
self.retry_delay = min(self.retry_delay * 1.5, 10)
else:
self.stdout.write(
self.style.ERROR(f" ❌ Max retries reached for {operation_name}")
)
raise
else:
# Non-lock related error, don't retry
self.stdout.write(
self.style.ERROR(f" ❌ Non-lock error in {operation_name}: {str(e)}")
)
raise
except IntegrityError as e:
# Handle unique constraint violations gracefully
if 'unique' in str(e).lower() or 'duplicate' in str(e).lower():
self.stdout.write(
self.style.WARNING(f" ⚠ Record already exists in {operation_name}: {str(e)}")
)
return None # Indicate that record already exists
else:
self.stdout.write(
self.style.ERROR(f" ❌ Integrity error in {operation_name}: {str(e)}")
)
raise
except Exception as e:
self.stdout.write(
self.style.ERROR(f" ❌ Unexpected error in {operation_name}: {str(e)}")
)
raise
def check_database_locks(self):
"""Check for existing database locks"""
try:
with connection.cursor() as cursor:
# Check for SQLite locks (if using SQLite)
cursor.execute("PRAGMA locking_mode;")
locking_mode = cursor.fetchone()
self.stdout.write(f"Database locking mode: {locking_mode}")
# Force a simple query to test connectivity
cursor.execute("SELECT 1;")
cursor.fetchone()
except Exception as e:
self.stdout.write(
self.style.WARNING(f"Could not check database locks: {str(e)}")
)
def create_single_status(self, status_data):
"""Create a single status with proper error handling"""
status, created = HadisStatus.objects.get_or_create(
title=status_data['title'],
defaults=status_data
)
if created:
self.stdout.write(f" ✅ Created status: {status.title}")
else:
self.stdout.write(f" ✓ Status already exists: {status.title}")
return status
def seed_hadis_statuses(self):
"""Create HadisStatus records"""
self.stdout.write("Creating Hadis Statuses...")
statuses_data = [
{'title': 'Authentic', 'color': 'green', 'order': 1},
{'title': 'Good', 'color': 'blue', 'order': 2},
{'title': 'Weak', 'color': 'yellow', 'order': 3},
{'title': 'Fabricated', 'color': 'red', 'order': 4},
{'title': 'Interrupted', 'color': 'orange', 'order': 5},
{'title': 'Broken', 'color': 'purple', 'order': 6},
{'title': 'Unknown', 'color': 'gray', 'order': 7},
]
statuses = []
for i, data in enumerate(statuses_data):
self.stdout.write(f" 📋 Processing status {i+1}/{len(statuses_data)}: {data['title']}")
# Add small delay between operations
if i > 0:
time.sleep(0.2)
status = self.safe_execute_with_retry(
f"Create status {data['title']}",
self.create_single_status,
data
)
if status:
statuses.append(status)
self.stdout.write(f"✅ Successfully processed {len(statuses)} statuses")
return statuses
def create_single_tag(self, tag_title):
"""Create a single tag with proper error handling"""
tag, created = HadisTag.objects.get_or_create(
title=tag_title,
defaults={'status': True}
)
if created:
self.stdout.write(f" ✅ Created tag: {tag.title}")
else:
self.stdout.write(f" ✓ Tag already exists: {tag.title}")
return tag
def seed_hadis_tags(self):
"""Create HadisTag records"""
self.stdout.write("Creating Hadis Tags...")
tags_data = [
'Worship', 'Prayer', 'Fasting', 'Hajj', 'Zakat', 'Khums',
'Ethics', 'Patience', 'Gratitude', 'Trust', 'Piety', 'Justice',
'Fiqh', 'Rulings', 'Halal', 'Haram', 'Mustahab', 'Makruh',
'Interpretation', 'Quran', 'Verses', 'Surah', 'Recitation',
'Imamate', 'Authority', 'Infallibles', 'Prophets Family',
'Supplication', 'Remembrance', 'Forgiveness', 'Praise', 'Monotheism'
]
tags = []
# Process tags in smaller batches to avoid locks
batch_size = 5
for i in range(0, len(tags_data), batch_size):
batch = tags_data[i:i + batch_size]
self.stdout.write(f" 📋 Processing tag batch {i//batch_size + 1}/{(len(tags_data) + batch_size - 1)//batch_size}")
for j, tag_title in enumerate(batch):
# Add small delay between operations
if j > 0:
time.sleep(0.1)
tag = self.safe_execute_with_retry(
f"Create tag {tag_title}",
self.create_single_tag,
tag_title
)
if tag:
tags.append(tag)
# Delay between batches
if i + batch_size < len(tags_data):
time.sleep(0.5)
self.stdout.write(f"✅ Successfully processed {len(tags)} tags")
return tags
def create_single_sect(self, sect_data):
"""Create a single sect with proper error handling"""
sect_type = sect_data['sect_type']
# Check if sect already exists
try:
existing_sect = HadisSect.objects.get(sect_type=sect_type)
self.stdout.write(f" ✓ Sect '{sect_type}' already exists: {existing_sect.title}")
return existing_sect
except HadisSect.DoesNotExist:
pass
# Create new sect
self.stdout.write(f" 🔨 Creating new sect: {sect_type}")
sect = HadisSect(
sect_type=sect_data['sect_type'],
title=sect_data['title'],
is_active=sect_data['is_active'],
order=sect_data['order']
)
sect.save()
self.stdout.write(f" ✅ Created sect: {sect.title}")
return sect
def seed_hadis_sects(self):
"""Create HadisSect records"""
self.stdout.write("Creating Hadis Sects...")
sects_data = [
{'sect_type': 'shia', 'title': 'Shia Twelvers', 'is_active': True, 'order': 1},
{'sect_type': 'sunni', 'title': 'Sunni', 'is_active': True, 'order': 2},
]
sects = []
# Process each sect individually with delay
for i, data in enumerate(sects_data):
sect_type = data['sect_type']
self.stdout.write(f" 📋 Processing sect {i+1}/{len(sects_data)}: {sect_type}")
# Add small delay between operations to prevent locks
if i > 0:
time.sleep(0.5)
sect = self.safe_execute_with_retry(
f"Create sect {sect_type}",
self.create_single_sect,
data
)
if sect:
sects.append(sect)
self.stdout.write(f"✅ Successfully processed {len(sects)} sects")
return sects
def assign_xmind_file(self, category):
"""Assign XMind file to category"""
if not self.xmind_file_path.exists():
return False
try:
with open(self.xmind_file_path, 'rb') as f:
file_content = f.read()
# Create unique filename for each category
filename = f"category_{category.id}_{category.title[:20]}.xmind"
category.xmind_file.save(
filename,
ContentFile(file_content),
save=True
)
return True
except Exception as e:
self.stdout.write(
self.style.WARNING(f"Could not assign XMind file to {category.title}: {e}")
)
return False
def create_single_category(self, sect, source_type, title, order, parent=None):
"""Create a single category with proper MPTT handling"""
try:
# Check if category already exists
existing_category = HadisCategory.objects.get(
sect=sect,
source_type=source_type,
title=title,
parent=parent
)
self.stdout.write(f" ✓ Category already exists: {title}")
return existing_category
except HadisCategory.DoesNotExist:
pass
# Create new category (MPTT will handle tree fields automatically)
self.stdout.write(f" 🔨 Creating category: {title}")
category = HadisCategory.objects.create(
sect=sect,
source_type=source_type,
title=title,
order=order,
parent=parent
)
self.stdout.write(f" ✅ Created category: {title}")
return category
def seed_hadis_categories(self, sects):
"""Create HadisCategory records with hierarchical structure - MPTT safe creation"""
self.stdout.write("Creating Hadis Categories...")
categories = []
for sect in sects:
self.stdout.write(f" 📋 Creating categories for {sect.title}...")
# Quran categories - create one by one to avoid MPTT issues
quran_categories_data = [
{'title': 'Quran Interpretation', 'order': 1},
{'title': 'Verses of Rulings', 'order': 2},
{'title': 'Quran Stories', 'order': 3},
{'title': 'Virtues of Surahs', 'order': 4},
{'title': 'Quran Miracles', 'order': 5},
{'title': 'Quranic Sciences', 'order': 6},
]
# Create main Quran categories one by one
quran_parent_categories = []
for i, cat_data in enumerate(quran_categories_data):
# Add delay between operations
if i > 0:
time.sleep(0.3)
category = self.safe_execute_with_retry(
f"Create Quran category {cat_data['title']}",
self.create_single_category,
sect, 'quran', cat_data['title'], cat_data['order']
)
if category:
categories.append(category)
quran_parent_categories.append(category)
# Create child categories for Quran
self.stdout.write(" 📂 Creating Quran child categories...")
for parent_category in quran_parent_categories:
child_categories_data = []
if parent_category.title == 'Quran Interpretation':
child_categories_data = [
{'title': 'Surah Al-Fatiha Interpretation', 'order': 1},
{'title': 'Surah Al-Baqara Interpretation', 'order': 2},
{'title': 'Surah Al Imran Interpretation', 'order': 3},
]
elif parent_category.title == 'Verses of Rulings':
child_categories_data = [
{'title': 'Prayer Verses', 'order': 1},
{'title': 'Fasting Verses', 'order': 2},
{'title': 'Zakat Verses', 'order': 3},
]
elif parent_category.title == 'Quran Stories':
child_categories_data = [
{'title': 'Prophets Stories', 'order': 1},
{'title': 'Righteous People Stories', 'order': 2},
]
# Create child categories one by one
for j, child_data in enumerate(child_categories_data):
# Add delay between operations
if j > 0:
time.sleep(0.2)
child_category = self.safe_execute_with_retry(
f"Create child category {child_data['title']}",
self.create_single_category,
sect, 'quran', child_data['title'], child_data['order'], parent_category
)
if child_category:
categories.append(child_category)
# Assign XMind file to some categories
if random.choice([True, False]) and not parent_category.xmind_file:
self.assign_xmind_file(parent_category)
# Hadith categories - create one by one
self.stdout.write(" 📚 Creating Hadith categories...")
hadith_categories_data = [
{'title': 'Book of Purification', 'order': 1},
{'title': 'Book of Prayer', 'order': 2},
{'title': 'Book of Fasting', 'order': 3},
{'title': 'Book of Hajj', 'order': 4},
{'title': 'Book of Zakat', 'order': 5},
{'title': 'Book of Ethics', 'order': 6},
]
# Create main Hadith categories one by one
hadith_parent_categories = []
for i, cat_data in enumerate(hadith_categories_data):
# Add delay between operations
if i > 0:
time.sleep(0.3)
category = self.safe_execute_with_retry(
f"Create Hadith category {cat_data['title']}",
self.create_single_category,
sect, 'hadith', cat_data['title'], cat_data['order']
)
if category:
categories.append(category)
hadith_parent_categories.append(category)
# Create child categories for Hadith
self.stdout.write(" 📂 Creating Hadith child categories...")
for parent_category in hadith_parent_categories:
child_categories_data = []
if parent_category.title == 'Book of Purification':
child_categories_data = [
{'title': 'Ablution', 'order': 1},
{'title': 'Full Bath', 'order': 2},
{'title': 'Dry Ablution', 'order': 3},
]
elif parent_category.title == 'Book of Prayer':
child_categories_data = [
{'title': 'Prayer Times', 'order': 1},
{'title': 'Qibla Direction', 'order': 2},
{'title': 'Congregational Prayer', 'order': 3},
]
elif parent_category.title == 'Book of Ethics':
child_categories_data = [
{'title': 'Patience and Gratitude', 'order': 1},
{'title': 'Justice and Honesty', 'order': 2},
]
# Create child categories one by one
for j, child_data in enumerate(child_categories_data):
# Add delay between operations
if j > 0:
time.sleep(0.2)
child_category = self.safe_execute_with_retry(
f"Create child category {child_data['title']}",
self.create_single_category,
sect, 'hadith', child_data['title'], child_data['order'], parent_category
)
if child_category:
categories.append(child_category)
# Assign XMind file to some categories
if random.choice([True, False]) and not parent_category.xmind_file:
self.assign_xmind_file(parent_category)
self.stdout.write(f"✅ Successfully processed {len(categories)} categories")
return categories
def seed_library_data(self):
"""Create library data (books, categories, collections) for references"""
self.stdout.write("Creating Library data...")
# Create library categories
lib_categories_data = [
'Книги хадисов', 'Книги фикха', 'Книги толкования', 'Книги нравственности', 'Исторические книги'
]
lib_categories = []
for cat_title in lib_categories_data:
category, created = LibraryCategory.objects.get_or_create(
title=cat_title,
defaults={'status': True}
)
lib_categories.append(category)
if created:
self.stdout.write(f" Created library category: {category.title}")
# Create book collections
collections_data = [
{'title': 'Шиитские книги хадисов', 'display_position': 'pinned'},
{'title': 'Суннитские книги хадисов', 'display_position': 'middle'},
{'title': 'Сборник книг по фикху', 'display_position': 'middle'},
]
collections = []
for coll_data in collections_data:
collection, created = BookCollection.objects.get_or_create(
title=coll_data['title'],
defaults={
'summary': f'Коллекция {coll_data["title"]}',
'display_position': coll_data['display_position'],
'status': True,
'order': len(collections) + 1
}
)
collections.append(collection)
if created:
self.stdout.write(f" Created collection: {collection.title}")
# Create books with cover images
books_data = [
{
'title': 'Аль-Кафи',
'summary_title': 'Книга Аль-Кафи шейха Кулейни',
'summary': 'Одна из важнейших книг хадисов шиитов',
'description': 'Книга Аль-Кафи, написанная Мухаммадом ибн Якубом Кулейни, является одной из четырех достоверных книг хадисов шиитов.',
'publisher': 'Дар аль-Кутуб аль-Исламийя',
'year_of_publication': '1407',
'isbn': '978-964-372-001-1',
'pages_count': '2847',
'file_type': 'pdf'
},
{
'title': 'Сахих аль-Бухари',
'summary_title': 'Сахих аль-Бухари имама Бухари',
'summary': 'Самая достоверная книга хадисов суннитов',
'description': 'Сахих аль-Бухари, написанный Мухаммадом ибн Исмаилом Бухари, является самой достоверной книгой хадисов у суннитов.',
'publisher': 'Дар Тук ан-Наджа',
'year_of_publication': '1422',
'isbn': '978-964-372-002-2',
'pages_count': '1896',
'file_type': 'pdf'
},
{
'title': 'Ман ля яхдуруху аль-факих',
'summary_title': 'Ман ля яхдуруху аль-факих шейха Садука',
'summary': 'Важная книга по фикху и хадисам шиитов',
'description': 'Книга Ман ля яхдуруху аль-факих, написанная шейхом Садуком, является одной из четырех книг шиитов.',
'publisher': 'Муассаса ан-Нашр аль-Ислами',
'year_of_publication': '1413',
'isbn': '978-964-372-003-3',
'pages_count': '1524',
'file_type': 'pdf'
},
{
'title': 'Сунан Абу Дауд',
'summary_title': 'Сунан Абу Дауд имама Абу Дауда',
'summary': 'Одна из шести книг суннитов',
'description': 'Сунан Абу Дауд, написанная Сулейманом ибн Ашасом Сиджистани, является одной из шести книг суннитов.',
'publisher': 'Аль-Мактаба аль-Асрийя',
'year_of_publication': '1430',
'isbn': '978-964-372-004-4',
'pages_count': '1342',
'file_type': 'pdf'
},
]
books = []
for book_data in books_data:
# Get random image for book cover
image_file = random.choice(self.image_files)
book, created = Book.objects.get_or_create(
title=book_data['title'],
defaults=book_data
)
if created:
# Assign cover image
try:
with open(image_file, 'rb') as f:
book.thumbnail.save(
f"book_cover_{book.id}.png",
File(f),
save=True
)
self.stdout.write(f" Created book: {book.title} with cover image")
except Exception as e:
self.stdout.write(f" Created book: {book.title} (no cover image: {e})")
# Assign to categories and collections
if lib_categories:
book.categories.add(random.choice(lib_categories))
if collections:
book.collections.add(random.choice(collections))
books.append(book)
return books, lib_categories, collections
def seed_transmitters(self):
"""Create Transmitters records"""
self.stdout.write("Creating Transmitters...")
transmitters_data = [
{
'full_name': 'Мухаммад ибн Якуб Кулейни',
'birth_year_hijri': 250,
'death_year_hijri': 329,
'description': 'Шейх Кулейни, автор книги Аль-Кафи и один из великих мухаддисов шиитов'
},
{
'full_name': 'Мухаммад ибн Али ибн Бабавейх (Шейх Садук)',
'birth_year_hijri': 306,
'death_year_hijri': 381,
'description': 'Шейх Садук, автор книги Ман ля яхдуруху аль-факих'
},
{
'full_name': 'Мухаммад ибн аль-Хасан ат-Туси',
'birth_year_hijri': 385,
'death_year_hijri': 460,
'description': 'Шейх Туси, автор книг Тахзиб аль-Ахкам и аль-Истибсар'
},
{
'full_name': 'Мухаммад ибн Исмаил аль-Бухари',
'birth_year_hijri': 194,
'death_year_hijri': 256,
'description': 'Имам Бухари, автор Сахих аль-Бухари'
},
{
'full_name': 'Муслим ибн аль-Хаджжадж ан-Нишапури',
'birth_year_hijri': 206,
'death_year_hijri': 261,
'description': 'Имам Муслим, автор Сахих Муслим'
},
{
'full_name': 'Абу Дауд ас-Сиджистани',
'birth_year_hijri': 202,
'death_year_hijri': 275,
'description': 'Имам Абу Дауд, автор Сунан Абу Дауд'
},
{
'full_name': 'Джафар ибн Мухаммад ас-Садик',
'birth_year_hijri': 83,
'death_year_hijri': 148,
'description': 'Имам Джафар Садик (мир ему), шестой имам шиитов'
},
{
'full_name': 'Мухаммад ибн Али аль-Бакир',
'birth_year_hijri': 57,
'death_year_hijri': 114,
'description': 'Имам Мухаммад Бакир (мир ему), пятый имам шиитов'
},
{
'full_name': 'Али ибн аль-Хусейн ас-Саджжад',
'birth_year_hijri': 38,
'death_year_hijri': 95,
'description': 'Имам Али ибн аль-Хусейн (мир ему), четвертый имам шиитов'
},
{
'full_name': 'Мухаммад ибн Муслим',
'birth_year_hijri': 70,
'death_year_hijri': 150,
'description': 'Мухаммад ибн Муслим, из сподвижников имама Бакира и имама Садика (мир им)'
},
]
transmitters = []
for trans_data in transmitters_data:
transmitter, created = Transmitters.objects.get_or_create(
full_name=trans_data['full_name'],
defaults=trans_data
)
transmitters.append(transmitter)
if created:
self.stdout.write(f" Created transmitter: {transmitter.full_name}")
return transmitters
def seed_hadis_records(self, categories, statuses, tags, transmitters, books):
"""Create Hadis records with proper relationships - optimized batch creation"""
self.stdout.write("Creating Hadis records...")
# Get only leaf categories (categories without children) - optimized query
from django.db import models
leaf_categories = HadisCategory.objects.filter(
id__in=[cat.id for cat in categories]
).annotate(
children_count=models.Count('children')
).filter(children_count=0)
self.stdout.write(f"Found {len(leaf_categories)} leaf categories for hadis creation")
# Comprehensive hadis samples with longer texts
hadis_samples = {
'prayer': [
{
'title': 'Достоинство молитвы и ее место в религии',
'text': '''قال رسول الله صلى الله عليه وآله: الصلاة عمود الدين، إن قبلت قبل ما سواها، وإن ردت رد ما سواها. وهي أول ما يحاسب عليه العبد يوم القيامة، فإن صلحت صلح سائر عمله، وإن فسدت فسد سائر عمله.
والصلاة معراج المؤمن، وهي قربان كل تقي، وهي حب الله تعالى. من أحبها وأقامها في أوقاتها وحافظ على حدودها رفعه الله إلى درجة الأبرار. ومن استخف بها وضيعها وتركها فقد استخف بدين الله، ولا نصيب له في الإسلام.
إن الله تعالى فرض خمس صلوات في اليوم والليلة، وجعل لكل صلاة وقتاً معلوماً، فمن صلاها في وقتها وأتم ركوعها وسجودها وخشوعها، كانت له نوراً وبرهاناً ونجاة يوم القيامة.''',
'translation': [
{'language_code': 'ru', 'title': '''Сказал Посланник Аллаха (да благословит Аллах его и его семейство): Молитва - столп религии, если она принята, то принято и остальное, а если отвергнута, то отвергнуто и остальное. Это первое, за что будет спрошен раб в День Воскресения, и если она будет правильной, то правильными будут и остальные его дела, а если испорчена, то испорчены и остальные его дела.
Молитва - это вознесение верующего, она - приношение каждого богобоязненного, она - любовь Всевышнего Аллаха. Кто полюбил ее и совершал ее в установленные времена и соблюдал ее границы, того Аллах возвысит до степени праведников. А кто пренебрег ею, потерял ее и оставил, тот пренебрег религией Аллаха, и нет ему доли в Исламе.
Поистине, Всевышний Аллах предписал пять молитв в сутки и установил для каждой молитвы определенное время. Кто совершал их в свое время и завершал их поклоны, земные поклоны и смирение, для того они станут светом, доказательством и спасением в День Воскресения.'''},
{'language_code': 'fa', 'title': '''رسول خدا فرمود: نماز ستون دین است، اگر پذیرفته شود غیر آن نیز پذیرفته می‌شود و اگر رد شود غیر آن نیز رد می‌شود. و این اولین چیزی است که بنده در روز قیامت از آن بازخواست می‌شود، پس اگر درست باشد تمام اعمالش درست است و اگر فاسد باشد تمام اعمالش فاسد است.
نماز معراج مؤمن است و قربانی هر پرهیزکار و محبت خداوند متعال است. هر کس آن را دوست بدارد و در اوقاتش برپا دارد و حدودش را نگه دارد، خداوند او را به درجه نیکان بالا می‌برد. و هر کس آن را سبک بشمارد و ضایع کند و ترک کند، دین خدا را سبک شمرده و بهره‌ای در اسلام ندارد.
خداوند متعال پنج نماز در شبانه‌روز واجب کرده و برای هر نماز وقت معینی قرار داده، پس هر کس آن‌ها را در وقتشان بخواند و رکوع و سجود و خشوعشان را کامل کند، برایش نور و برهان و نجات در روز قیامت خواهد بود.'''},
{'language_code': 'en', 'title': '''The Messenger of Allah said: Prayer is the pillar of religion, if it is accepted, other deeds are accepted, and if it is rejected, other deeds are rejected. It is the first thing for which a servant will be held accountable on the Day of Judgment, and if it is sound, all his other deeds will be sound, and if it is corrupted, all his other deeds will be corrupted.
Prayer is the ascension of the believer, it is the offering of every God-fearing person, and it is the love of Allah the Almighty. Whoever loves it and establishes it at its times and maintains its boundaries, Allah will raise him to the rank of the righteous. And whoever takes it lightly, wastes it and abandons it, has taken Allah's religion lightly, and has no share in Islam.
Indeed, Allah the Almighty has prescribed five prayers in a day and night, and has appointed a specific time for each prayer. Whoever prays them at their time and completes their bowing, prostration and humility, they will be light, proof and salvation for him on the Day of Judgment.'''}
],
'explanation': '''Этот обширный хадис представляет собой фундаментальное учение о молитве в Исламе. Он раскрывает несколько ключевых аспектов:
Во-первых, молитва описывается как "столп религии" (عمود الدين), что указывает на ее центральную роль в исламской вере. Это метафора подчеркивает, что как здание не может стоять без столпов, так и религиозная жизнь мусульманина не может быть полноценной без молитвы.
Во-вторых, хадис устанавливает принцип, согласно которому принятие или отвержение молитвы Аллахом определяет судьбу всех остальных деяний верующего. Это подчеркивает качественный аспект молитвы - важна не только ее форма, но и искренность, концентрация и правильное выполнение.
В-третьих, молитва представлена как "معراج المؤمن" (вознесение верующего), что отсылает к ночному путешествию Пророка (мир ему) и подчеркивает духовное измерение молитвы как средства приближения к Всевышнему.
Хадис также подчеркивает важность своевременного совершения молитв и соблюдения их внешних и внутренних условий, обещая великую награду тем, кто относится к молитве с должным вниманием и уважением.'''
},
],
'fasting': [
{
'title': 'Достоинство поста и его духовные плоды',
'text': '''قال النبي صلى الله عليه وآله: الصوم جنة من النار، وهو زكاة البدن، وصوم شهر الصبر وثلاثة أيام من كل شهر يذهبن وحر الصدر ووساوس الشيطان. ومن صام يوماً في سبيل الله باعد الله وجهه عن النار سبعين خريفاً.
إن الصائم في عبادة وإن كان نائماً على فراشه، وإن دعاءه مستجاب حتى يفطر، وإن الملائكة تستغفر له حتى يفطر. وللصائم فرحتان: فرحة عند فطره، وفرحة عند لقاء ربه.
يا معشر الشباب، من استطاع منكم الباءة فليتزوج، ومن لم يستطع فعليه بالصوم فإنه له وجاء. والصوم يكسر الشهوة ويطهر النفس ويقرب إلى الله تعالى.''',
'translation': [
{'language_code': 'ru', 'title': '''Сказал Пророк (да благословит Аллах его и его семейство): Пост - щит от огня, и он закят тела, и пост месяца терпения и трех дней каждого месяца устраняют жар груди и наущения шайтана. И кто постился день на пути Аллаха, Аллах отдалит его лицо от огня на семьдесят осеней.
Поистине, постящийся находится в поклонении, даже если он спит на своей постели, и его мольба принимается до тех пор, пока он не разговеется, и ангелы просят прощения для него до тех пор, пока он не разговеется. И у постящегося две радости: радость при разговении и радость при встрече со своим Господом.
О молодежь! Кто из вас способен жениться, пусть женится, а кто не способен, пусть постится, ибо это для него защита. Пост ломает страсть, очищает душу и приближает к Всевышнему Аллаху.'''}
],
'explanation': '''Этот многогранный хадис раскрывает глубокие духовные и практические аспекты поста в Исламе.'''
},
],
'ethics': [
{
'title': 'Благородный нрав',
'text': 'قال رسول الله صلى الله عليه وآله: إنما بعثت لأتمم مكارم الأخلاق.',
'translation': [
{'language_code': 'ru', 'title': 'Сказал Посланник Аллаха (да благословит Аллах его и его семейство): Поистине, я послан, чтобы довести до совершенства благородные нравы.'}
],
'explanation': 'Этот хадис подчеркивает важность благородного нрава в Исламе.'
},
]
}
# Create hadis records for each leaf category
hadis_created_count = 0
for category in leaf_categories:
# Determine hadis type based on category title
hadis_type = 'prayer'
if any(word in category.title.lower() for word in ['пост', 'рамадан']):
hadis_type = 'fasting'
elif any(word in category.title.lower() for word in ['нрав', 'терпение', 'справедливость']):
hadis_type = 'ethics'
# Get sample hadis for this type
samples = hadis_samples.get(hadis_type, hadis_samples['prayer'])
# Create 10 hadis per category
for i in range(10):
sample = random.choice(samples)
# Create unique title
hadis_title = f"{sample['title']} - {category.title} ({i+1})"
# Check if hadis already exists
if Hadis.objects.filter(title=hadis_title, category=category).exists():
continue
# Create hadis record
hadis = Hadis.objects.create(
category=category,
title=hadis_title,
text=sample['text'],
translation=sample['translation'],
explanation=sample['explanation'],
status=True, # Boolean field for visibility
hadis_status=random.choice(statuses), # ForeignKey to HadisStatus
links=[
{'title': 'Source 1', 'link': 'https://example.com/source1'},
{'title': 'Source 2', 'link': 'https://example.com/source2'},
]
)
# Add tags
selected_tags = random.sample(tags, min(3, len(tags)))
hadis.tags.set(selected_tags)
# Create transmission chain (5 transmitters with one gap)
selected_transmitters = random.sample(transmitters, min(5, len(transmitters)))
gap_position = random.randint(0, len(selected_transmitters) - 1)
for j, transmitter in enumerate(selected_transmitters):
HadisTransmitter.objects.create(
hadis=hadis,
transmitter=transmitter,
order=j + 1,
is_gap=(j == gap_position)
)
# Create reference
reference_book = random.choice(books)
reference = HadisReference.objects.create(
hadis=hadis,
book=reference_book,
description=f"Volume {random.randint(1, 5)}, Page {random.randint(1, 1000)}, Hadis #{random.randint(1, 9999)}"
)
# Add reference image
if self.image_files:
image_file = random.choice(self.image_files)
try:
with open(image_file, 'rb') as f:
ReferenceImage.objects.create(
reference=reference,
thumbnail=File(f, name=f"ref_{reference.id}.png")
)
except Exception as e:
self.stdout.write(
self.style.WARNING(f"Could not add reference image: {e}")
)
hadis_created_count += 1
if hadis_created_count % 50 == 0:
self.stdout.write(f" Created {hadis_created_count} hadis records...")
self.stdout.write(f"Successfully created {hadis_created_count} hadis records")

45
apps/hadis/management/commands/test_sects.py
View File

@ -0,0 +1,45 @@
from django.core.management.base import BaseCommand
from apps.hadis.models import HadisSect
class Command(BaseCommand):
help = 'Test sects creation'
def handle(self, **options):
_ = options # Suppress unused variable warning
self.stdout.write("Testing sects creation...")
try:
# Check existing sects
existing_sects = HadisSect.objects.all()
self.stdout.write(f"Found {existing_sects.count()} existing sects:")
for sect in existing_sects:
self.stdout.write(f" - {sect.sect_type}: {sect.title}")
# Try to create sunni sect using direct create
self.stdout.write("Attempting to create sunni sect...")
# Check if sunni exists
try:
sunni_sect = HadisSect.objects.get(sect_type='sunni')
self.stdout.write(f"Sunni sect already exists: {sunni_sect.title}")
except HadisSect.DoesNotExist:
# Create sunni sect
self.stdout.write("Creating new sunni sect...")
sunni_sect = HadisSect(
sect_type='sunni',
title='Сунниты',
is_active=True,
order=2
)
sunni_sect.save()
self.stdout.write(self.style.SUCCESS(f"Created sunni sect: {sunni_sect.title}"))
# Final count
final_count = HadisSect.objects.count()
self.stdout.write(f"Total sects after operation: {final_count}")
except Exception as e:
self.stdout.write(self.style.ERROR(f"Error: {str(e)}"))
import traceback
self.stdout.write(self.style.ERROR(traceback.format_exc()))

530
scripts/seed_hadis_data.py
View File

@ -11,13 +11,13 @@ import django
from pathlib import Path
from django.core.files import File
from django.core.files.base import ContentFile
from django.db import transaction
from django.db import transaction, models
import random
# Setup Django environment
BASE_DIR = Path(__file__).resolve().parent.parent
sys.path.append(str(BASE_DIR))
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.base')
django.setup()
# Import models after Django setup
@ -39,8 +39,9 @@ class HadisDataSeeder:
# Verify required files exist
if not self.seed_images_dir.exists():
raise FileNotFoundError(f"Seed images directory not found: {self.seed_images_dir}")
if not self.xmind_file_path.exists():
raise FileNotFoundError(f"XMind file not found: {self.xmind_file_path}")
# XMind file is optional
# if not self.xmind_file_path.exists():
# raise FileNotFoundError(f"XMind file not found: {self.xmind_file_path}")
# Get available images
self.image_files = list(self.seed_images_dir.glob('*.png'))
@ -158,14 +159,16 @@ class HadisDataSeeder:
return False
def seed_hadis_categories(self, sects):
"""Create HadisCategory records with hierarchical structure"""
"""Create HadisCategory records with hierarchical structure - optimized batch creation"""
print("Creating Hadis Categories...")
categories = []
categories_to_create = []
categories_to_update = []
for sect in sects:
print(f" Creating categories for {sect.title}...")
# Quran categories
quran_categories_data = [
{'title': 'Толкование Корана', 'order': 1},
@ -175,64 +178,132 @@ class HadisDataSeeder:
{'title': 'Чудеса Корана', 'order': 5},
{'title': 'Коранические науки', 'order': 6},
]
for cat_data in quran_categories_data:
category = HadisCategory.objects.create(
sect=sect,
source_type='quran',
title=cat_data['title'],
order=cat_data['order']
# First, get existing categories to avoid duplicates
existing_quran_categories = {
cat.title: cat for cat in HadisCategory.objects.filter(
sect=sect, source_type='quran', parent=None
)
}
# Process main Quran categories
for cat_data in quran_categories_data:
if cat_data['title'] in existing_quran_categories:
category = existing_quran_categories[cat_data['title']]
if category.order != cat_data['order']:
category.order = cat_data['order']
categories_to_update.append(category)
print(f" Will update Quran category: {category.title}")
else:
print(f" Quran category exists: {category.title}")
else:
category = HadisCategory(
sect=sect,
source_type='quran',
title=cat_data['title'],
order=cat_data['order']
)
categories_to_create.append(category)
print(f" Will create Quran category: {cat_data['title']}")
categories.append(category)
# Assign XMind file to some categories
if random.choice([True, False]):
self.assign_xmind_file(category)
print(f" Created Quran category: {category.title}")
# Create child categories for main categories
if cat_data['title'] == 'Толкование Корана':
child_categories = [
# Batch create new categories
if categories_to_create:
HadisCategory.objects.bulk_create(categories_to_create, ignore_conflicts=True)
print(f" Batch created {len(categories_to_create)} Quran categories")
categories_to_create = []
# Batch update existing categories
if categories_to_update:
HadisCategory.objects.bulk_update(categories_to_update, ['order'])
print(f" Batch updated {len(categories_to_update)} Quran categories")
categories_to_update = []
# Now handle child categories
parent_categories = HadisCategory.objects.filter(
sect=sect, source_type='quran', parent=None
)
for parent_category in parent_categories:
child_categories_data = []
if parent_category.title == 'Толкование Корана':
child_categories_data = [
{'title': 'Толкование суры Аль-Фатиха', 'order': 1},
{'title': 'Толкование суры Аль-Бакара', 'order': 2},
{'title': 'Толкование суры Аль Имран', 'order': 3},
{'title': 'Толкование суры Ан-Ниса', 'order': 4},
{'title': 'Толкование суры Аль-Маида', 'order': 5},
]
elif cat_data['title'] == 'Аяты предписаний':
child_categories = [
elif parent_category.title == 'Аяты предписаний':
child_categories_data = [
{'title': 'Аяты о молитве', 'order': 1},
{'title': 'Аяты о посте', 'order': 2},
{'title': 'Аяты о закяте', 'order': 3},
{'title': 'Аяты о хадже', 'order': 4},
]
elif cat_data['title'] == 'Рассказы Корана':
child_categories = [
elif parent_category.title == 'Рассказы Корана':
child_categories_data = [
{'title': 'История пророков', 'order': 1},
{'title': 'Рассказы о праведниках', 'order': 2},
{'title': 'Уроки из истории', 'order': 3},
]
elif cat_data['title'] == 'Достоинства сур':
child_categories = [
elif parent_category.title == 'Достоинства сур':
child_categories_data = [
{'title': 'Достоинства суры Аль-Фатиха', 'order': 1},
{'title': 'Достоинства суры Аль-Бакара', 'order': 2},
{'title': 'Достоинства суры Йа-Син', 'order': 3},
{'title': 'Достоинства суры Аль-Мульк', 'order': 4},
]
for child_data in child_categories:
child_category = HadisCategory.objects.create(
parent=category,
sect=sect,
source_type='quran',
title=child_data['title'],
order=child_data['order']
if child_categories_data:
# Get existing child categories
existing_children = {
cat.title: cat for cat in HadisCategory.objects.filter(
parent=parent_category, sect=sect, source_type='quran'
)
}
# Process child categories
for child_data in child_categories_data:
if child_data['title'] in existing_children:
child_category = existing_children[child_data['title']]
if child_category.order != child_data['order']:
child_category.order = child_data['order']
categories_to_update.append(child_category)
print(f" Will update child category: {child_category.title}")
else:
print(f" Child category exists: {child_category.title}")
else:
child_category = HadisCategory(
parent=parent_category,
sect=sect,
source_type='quran',
title=child_data['title'],
order=child_data['order']
)
categories_to_create.append(child_category)
print(f" Will create child category: {child_data['title']}")
categories.append(child_category)
print(f" Created child category: {child_category.title}")
# Hadith categories
# Batch operations for child categories
if categories_to_create:
HadisCategory.objects.bulk_create(categories_to_create, ignore_conflicts=True)
print(f" Batch created {len(categories_to_create)} child categories")
categories_to_create = []
if categories_to_update:
HadisCategory.objects.bulk_update(categories_to_update, ['order'])
print(f" Batch updated {len(categories_to_update)} child categories")
categories_to_update = []
# Assign XMind file to some categories (after creation)
if random.choice([True, False]) and not parent_category.xmind_file:
self.assign_xmind_file(parent_category)
# Hadith categories - optimized batch processing
hadith_categories_data = [
{'title': 'Книга очищения', 'order': 1},
{'title': 'Книга молитвы', 'order': 2},
@ -245,69 +316,139 @@ class HadisDataSeeder:
{'title': 'Книга джихада', 'order': 9},
{'title': 'Книга судопроизводства', 'order': 10},
]
for cat_data in hadith_categories_data:
category = HadisCategory.objects.create(
sect=sect,
source_type='hadith',
title=cat_data['title'],
order=cat_data['order']
# Get existing hadith categories
existing_hadith_categories = {
cat.title: cat for cat in HadisCategory.objects.filter(
sect=sect, source_type='hadith', parent=None
)
}
categories_to_create = []
categories_to_update = []
# Process main Hadith categories
for cat_data in hadith_categories_data:
if cat_data['title'] in existing_hadith_categories:
category = existing_hadith_categories[cat_data['title']]
if category.order != cat_data['order']:
category.order = cat_data['order']
categories_to_update.append(category)
print(f" Will update Hadith category: {category.title}")
else:
print(f" Hadith category exists: {category.title}")
else:
category = HadisCategory(
sect=sect,
source_type='hadith',
title=cat_data['title'],
order=cat_data['order']
)
categories_to_create.append(category)
print(f" Will create Hadith category: {cat_data['title']}")
categories.append(category)
# Assign XMind file to some categories
if random.choice([True, False]):
self.assign_xmind_file(category)
print(f" Created Hadith category: {category.title}")
# Create child categories for main categories
if cat_data['title'] == 'Книга очищения':
child_categories = [
# Batch create new hadith categories
if categories_to_create:
HadisCategory.objects.bulk_create(categories_to_create, ignore_conflicts=True)
print(f" Batch created {len(categories_to_create)} Hadith categories")
# Batch update existing hadith categories
if categories_to_update:
HadisCategory.objects.bulk_update(categories_to_update, ['order'])
print(f" Batch updated {len(categories_to_update)} Hadith categories")
# Now handle hadith child categories
parent_categories = HadisCategory.objects.filter(
sect=sect, source_type='hadith', parent=None
)
for parent_category in parent_categories:
child_categories_data = []
if parent_category.title == 'Книга очищения':
child_categories_data = [
{'title': 'Омовение', 'order': 1},
{'title': 'Полное омовение', 'order': 2},
{'title': 'Сухое омовение', 'order': 3},
{'title': 'Нечистоты', 'order': 4},
]
elif cat_data['title'] == 'Книга молитвы':
child_categories = [
elif parent_category.title == 'Книга молитвы':
child_categories_data = [
{'title': 'Времена молитвы', 'order': 1},
{'title': 'Кибла', 'order': 2},
{'title': 'Азан и икама', 'order': 3},
{'title': 'Коллективная молитва', 'order': 4},
{'title': 'Пятничная молитва', 'order': 5},
]
elif cat_data['title'] == 'Книга поста':
child_categories = [
elif parent_category.title == 'Книга поста':
child_categories_data = [
{'title': 'Пост в Рамадан', 'order': 1},
{'title': 'Добровольный пост', 'order': 2},
{'title': 'Нарушители поста', 'order': 3},
{'title': 'Ночь предопределения', 'order': 4},
]
elif cat_data['title'] == 'Книга хаджа':
child_categories = [
elif parent_category.title == 'Книга хаджа':
child_categories_data = [
{'title': 'Обряды хаджа', 'order': 1},
{'title': 'Умра', 'order': 2},
{'title': 'Запреты ихрама', 'order': 3},
]
elif cat_data['title'] == 'Книга нравственности':
child_categories = [
elif parent_category.title == 'Книга нравственности':
child_categories_data = [
{'title': 'Терпение и благодарность', 'order': 1},
{'title': 'Справедливость и честность', 'order': 2},
{'title': 'Знание и мудрость', 'order': 3},
{'title': 'Дружба и братство', 'order': 4},
]
for child_data in child_categories:
child_category = HadisCategory.objects.create(
parent=category,
sect=sect,
source_type='hadith',
title=child_data['title'],
order=child_data['order']
if child_categories_data:
# Get existing child categories
existing_children = {
cat.title: cat for cat in HadisCategory.objects.filter(
parent=parent_category, sect=sect, source_type='hadith'
)
}
categories_to_create = []
categories_to_update = []
# Process child categories
for child_data in child_categories_data:
if child_data['title'] in existing_children:
child_category = existing_children[child_data['title']]
if child_category.order != child_data['order']:
child_category.order = child_data['order']
categories_to_update.append(child_category)
print(f" Will update child category: {child_category.title}")
else:
print(f" Child category exists: {child_category.title}")
else:
child_category = HadisCategory(
parent=parent_category,
sect=sect,
source_type='hadith',
title=child_data['title'],
order=child_data['order']
)
categories_to_create.append(child_category)
print(f" Will create child category: {child_data['title']}")
categories.append(child_category)
print(f" Created child category: {child_category.title}")
# Batch operations for child categories
if categories_to_create:
HadisCategory.objects.bulk_create(categories_to_create, ignore_conflicts=True)
print(f" Batch created {len(categories_to_create)} child categories")
if categories_to_update:
HadisCategory.objects.bulk_update(categories_to_update, ['order'])
print(f" Batch updated {len(categories_to_update)} child categories")
# Assign XMind file to some categories (after creation)
if random.choice([True, False]) and not parent_category.xmind_file:
self.assign_xmind_file(parent_category)
return categories
@ -513,14 +654,15 @@ class HadisDataSeeder:
return transmitters
def seed_hadis_records(self, categories, statuses, tags, transmitters, books):
"""Create Hadis records with proper relationships - only for leaf categories"""
"""Create Hadis records with proper relationships - optimized batch creation"""
print("Creating Hadis records...")
# Get only leaf categories (categories without children)
leaf_categories = []
for category in categories:
if not category.get_children().exists():
leaf_categories.append(category)
# Get only leaf categories (categories without children) - optimized query
leaf_categories = HadisCategory.objects.filter(
id__in=[cat.id for cat in categories]
).annotate(
children_count=models.Count('children')
).filter(children_count=0)
print(f"Found {len(leaf_categories)} leaf categories for hadis creation")
@ -769,8 +911,23 @@ O young people! Whoever among you is able to marry, let him marry, and whoever i
hadis_records = []
hadis_number = 1
# Batch processing for hadis creation
hadis_to_create = []
hadis_to_update = []
hadis_tags_to_set = []
print("Processing categories in batches...")
# Create hadis for each leaf category (10 hadis per category)
for category in leaf_categories:
total_categories = len(leaf_categories)
for idx, category in enumerate(leaf_categories, 1):
print(f" Processing category {idx}/{total_categories}: {category.title}")
# Get existing hadis for this category to avoid duplicates
existing_hadis = {
h.number: h for h in Hadis.objects.filter(category=category)
}
# Determine hadis type based on category title
hadis_type = 'ethics' # default
if 'молитв' in category.title.lower() or 'намаз' in category.title.lower():
@ -815,38 +972,123 @@ O young people! Whoever among you is able to marry, let him marry, and whoever i
{'title': 'Духовное развитие', 'link': 'https://spiritual-growth.ru'}
]
hadis = Hadis.objects.create(
category=category,
number=hadis_number,
title=sample['title'],
text=sample['text'],
translation=sample['translation'],
status=True,
hadis_status=random.choice(statuses),
hadis_status_text=f"Приведен в достоверных книгах",
address=f"Книга {category.title}, хадис {hadis_number}",
explanation=sample.get('explanation', ''),
links=links
)
# Check if hadis already exists
if hadis_number in existing_hadis:
hadis = existing_hadis[hadis_number]
# Check if update is needed
updated = False
if (hadis.title != sample['title'] or
hadis.text != sample['text'] or
hadis.translation != sample['translation'] or
hadis.explanation != sample.get('explanation', '') or
hadis.links != links):
hadis.title = sample['title']
hadis.text = sample['text']
hadis.translation = sample['translation']
hadis.explanation = sample.get('explanation', '')
hadis.links = links
hadis_to_update.append(hadis)
updated = True
if updated:
print(f" Will update hadis #{hadis.number}: {hadis.title}")
else:
print(f" Hadis #{hadis.number} already exists and is up to date")
# Add tags if needed
if not hadis.tags.exists():
selected_tags = random.sample(tags, random.randint(2, 5))
hadis_tags_to_set.append((hadis, selected_tags))
else:
# Create new hadis
hadis = Hadis(
category=category,
number=hadis_number,
title=sample['title'],
text=sample['text'],
translation=sample['translation'],
status=True,
hadis_status=random.choice(statuses),
hadis_status_text="Приведен в достоверных книгах",
address=f"Книга {category.title}, хадис {hadis_number}",
explanation=sample.get('explanation', ''),
links=links
)
hadis_to_create.append(hadis)
# Add random tags
selected_tags = random.sample(tags, random.randint(2, 5))
hadis.tags.set(selected_tags)
# Prepare tags for later assignment
selected_tags = random.sample(tags, random.randint(2, 5))
hadis_tags_to_set.append((hadis, selected_tags))
print(f" Will create hadis #{hadis_number}: {hadis.title}")
hadis_records.append(hadis)
hadis_number += 1
print(f" Created hadis #{hadis.number}: {hadis.title} in {category.title}")
# Batch operations every 50 hadis or at end of category
if len(hadis_to_create) >= 50 or len(hadis_to_update) >= 50:
self._perform_hadis_batch_operations(hadis_to_create, hadis_to_update, hadis_tags_to_set)
hadis_to_create = []
hadis_to_update = []
hadis_tags_to_set = []
# Final batch operations
if hadis_to_create or hadis_to_update:
self._perform_hadis_batch_operations(hadis_to_create, hadis_to_update, hadis_tags_to_set)
return hadis_records
def _perform_hadis_batch_operations(self, hadis_to_create, hadis_to_update, hadis_tags_to_set):
"""Perform batch database operations for hadis"""
# Batch create
if hadis_to_create:
Hadis.objects.bulk_create(hadis_to_create, ignore_conflicts=True)
print(f" Batch created {len(hadis_to_create)} hadis records")
# Batch update
if hadis_to_update:
Hadis.objects.bulk_update(
hadis_to_update,
['title', 'text', 'translation', 'explanation', 'links']
)
print(f" Batch updated {len(hadis_to_update)} hadis records")
# Set tags (this needs to be done after creation/update)
if hadis_tags_to_set:
for hadis, tags_list in hadis_tags_to_set:
# For newly created hadis, we need to get the actual object from DB
if not hadis.pk:
try:
hadis = Hadis.objects.get(category=hadis.category, number=hadis.number)
except Hadis.DoesNotExist:
continue
hadis.tags.set(tags_list)
print(f" Set tags for {len(hadis_tags_to_set)} hadis records")
def seed_hadis_transmitters(self, hadis_records, transmitters, statuses):
"""Create HadisTransmitter records (transmission chains)"""
"""Create HadisTransmitter records (transmission chains) - optimized batch creation"""
print("Creating Hadis Transmitters (chains)...")
transmitter_chains = []
chains_to_create = []
# Get hadis that already have transmitters to avoid duplicates
hadis_with_transmitters = set(
HadisTransmitter.objects.values_list('hadis_id', flat=True).distinct()
)
print(f"Processing {len(hadis_records)} hadis records...")
for hadis in hadis_records:
# Skip if this hadis already has transmitters
if hadis.id in hadis_with_transmitters:
print(f" Hadis #{hadis.number} already has transmitters, skipping...")
continue
# Create a transmission chain of 3-6 transmitters
chain_length = random.randint(3, 6)
selected_transmitters = random.sample(transmitters, min(chain_length, len(transmitters)))
@ -855,29 +1097,49 @@ O young people! Whoever among you is able to marry, let him marry, and whoever i
# Occasionally create gaps in the chain
is_gap = random.choice([False, False, False, True]) # 25% chance of gap
chain = HadisTransmitter.objects.create(
chain = HadisTransmitter(
hadis=hadis,
order=order,
transmitter=transmitter if not is_gap else None,
status=random.choice(statuses),
is_gap=is_gap,
gap_type=random.choice(['unknown', 'missing', 'disputed', 'weak']) if is_gap else None,
order=order
is_gap=is_gap
)
chains_to_create.append(chain)
transmitter_chains.append(chain)
if is_gap:
print(f" Added gap in chain for hadis #{hadis.number} at position {order}")
print(f" Will add gap in chain for hadis #{hadis.number} at position {order}")
else:
print(f" Added transmitter {transmitter.full_name} to hadis #{hadis.number}")
print(f" Will add transmitter {transmitter.full_name} to hadis #{hadis.number}")
# Batch create every 100 chains
if len(chains_to_create) >= 100:
HadisTransmitter.objects.bulk_create(chains_to_create, ignore_conflicts=True)
print(f" Batch created {len(chains_to_create)} transmitter chains")
chains_to_create = []
# Final batch create
if chains_to_create:
HadisTransmitter.objects.bulk_create(chains_to_create, ignore_conflicts=True)
print(f" Final batch created {len(chains_to_create)} transmitter chains")
return transmitter_chains
def seed_hadis_references(self, hadis_records, books):
"""Create HadisReference records"""
"""Create HadisReference records - optimized batch creation"""
print("Creating Hadis References...")
references = []
references_to_create = []
references_to_update = []
# Get existing references to avoid duplicates
existing_references = {}
for ref in HadisReference.objects.select_related('hadis', 'book').all():
key = (ref.hadis_id, ref.book_id)
existing_references[key] = ref
print(f"Processing references for {len(hadis_records)} hadis records...")
for hadis in hadis_records:
# Each hadis can have 1-3 references
@ -885,17 +1147,47 @@ O young people! Whoever among you is able to marry, let him marry, and whoever i
selected_books = random.sample(books, min(num_refs, len(books)))
for book in selected_books:
try:
reference = HadisReference.objects.create(
key = (hadis.id, book.id)
new_description = f"Источник хадиса номер {hadis.number} в книге {book.title}"
if key in existing_references:
reference = existing_references[key]
if reference.description != new_description:
reference.description = new_description
references_to_update.append(reference)
print(f" Will update reference: Hadis #{hadis.number} -> {book.title}")
else:
print(f" Reference already exists: Hadis #{hadis.number} -> {book.title}")
else:
reference = HadisReference(
hadis=hadis,
book=book,
description=f"Источник хадиса номер {hadis.number} в книге {book.title}"
description=new_description
)
references.append(reference)
print(f" Created reference: Hadis #{hadis.number} -> {book.title}")
except Exception as e:
# Skip if reference already exists (unique_together constraint)
pass
references_to_create.append(reference)
print(f" Will create reference: Hadis #{hadis.number} -> {book.title}")
references.append(reference)
# Batch operations every 100 references
if len(references_to_create) >= 100:
HadisReference.objects.bulk_create(references_to_create, ignore_conflicts=True)
print(f" Batch created {len(references_to_create)} references")
references_to_create = []
if len(references_to_update) >= 100:
HadisReference.objects.bulk_update(references_to_update, ['description'])
print(f" Batch updated {len(references_to_update)} references")
references_to_update = []
# Final batch operations
if references_to_create:
HadisReference.objects.bulk_create(references_to_create, ignore_conflicts=True)
print(f" Final batch created {len(references_to_create)} references")
if references_to_update:
HadisReference.objects.bulk_update(references_to_update, ['description'])
print(f" Final batch updated {len(references_to_update)} references")
return references
@ -966,7 +1258,7 @@ O young people! Whoever among you is able to marry, let him marry, and whoever i
print("STEP 3: Creating library data")
print("=" * 40)
books, lib_categories, collections = self.seed_library_data()
books, _, _ = self.seed_library_data()
# Step 4: Create transmitters
print("\n" + "=" * 40)
@ -1068,7 +1360,7 @@ def main():
try:
seeder = HadisDataSeeder()
result = seeder.run_seeding(clear_existing=clear_existing)
seeder.run_seeding(clear_existing=clear_existing)
print("\n✅ Seeding completed successfully!")
print("You can now test the APIs:")

Write Preview
Loading…
Cancel
Save