Supporting Reference 3: Troubleshooting

Use this page when a V1.0 run fails or quality output is unexpectedly weak. It is optimized for fast diagnosis and safe recovery.

R3.1 First-response triage order

1. Read dist/logs/latest_run.log for EXE runs (or logs/latest_run.log for Python runs) and locate the first blocking error.
2. If publish is involved, read logs/db_uploader.log for first failing row.
3. Check row state in review_queue by Review_Status/QC_Status.
4. Check metadata_quality for V1.0 proof status, issue labels, and repair attempts.
5. Fix root cause before retrying (model/dependency/path/credentials).
6. Rerun safely through normal flow; do not patch ad-hoc state blindly.

R3.2 Caption/prefill failures (Ollama)

Symptoms: caption stage stalls, timeout errors, or missing model exceptions.

Checks:

• Run ollama list and confirm service responsiveness.
• Confirm configured caption models exist (primary required; fallback only if configured).
• Check startup line in latest_run.log for processor mode: processor=GPU/CPU.
• Inspect stage logs in logs/latest_run.log.
• Check continuation lines in logs: [RETRY]/[RETRY-OK] (if fallback enabled), timeout warnings, and quarantine lines for repeated native crash rows.

Fix path:

1. Pull missing model with ollama pull <model>.
2. If expected GPU mode is not used, set/verify: OLLAMA_FORCE_GPU=1, OLLAMA_RESTART_FOR_GPU=1, OLLAMA_LLM_LIBRARY=cuda, OLLAMA_NUM_GPU, OLLAMA_MAIN_GPU.
3. Restart/retry once after service health is restored.
4. If the same row repeatedly fails with quarantine markers (PrefillNativeCrash), inspect row-specific failure reason first instead of re-running full batches blindly.
5. If repeated, reduce parallel load and re-run controlled batch.

R3.3 Quality scoring failures

Symptoms: scoring stage crash, missing metric fields, or repeated stage abort.

Checks:

• Validate environment dependencies (numpy/opencv/pyiqa/torch/torchvision).
• Confirm sac+logos+ava1-l14-linearMSE.pth exists in repository root.
• For EXE runs, verify Score runtime python: ... (ver=(3, 13)) appears in stage log.
• Review error trace and failing row context in logs/latest_run.log.
• Verify score completeness in SQL checks (all score fields should be non-null for processed rows).

Fix path:

1. Rebuild or repair venv if imports fail.
2. If error mentions python312.dll conflict, relaunch EXE with AMIR_PYTHON set to the 3.13 interpreter.
3. Restore missing model weight file.
4. Retry the run from start after environment is healthy.

-- quick completeness check
SELECT
  SUM(nima_score IS NULL) AS nima_null,
  SUM(blur_score IS NULL) AS blur_null,
  SUM(brightness_score IS NULL) AS brightness_null,
  SUM(contrast_score IS NULL) AS contrast_null,
  SUM(brisque_score IS NULL) AS brisque_null,
  SUM(clip_aesthetic_score IS NULL) AS clip_null,
  SUM(QR IS NULL) AS qr_null
FROM photos_info_revamp;

R3.4 Resize failures (QC_Status=ResizeFailed)

Symptoms: rows flagged as ResizeFailed and caption prefill skipped.

Checks:

• Validate source image path readability and file integrity.
• Validate write permissions under data/ollama_tmp.
• Check available disk space.

Fix path: correct filesystem constraints, then rerun the affected set through normal pipeline entry.

R3.4A Metadata quality proof failures

Symptoms: rows reach review with empty fields, generic filler, duplicate wording, or Metadata_Needs_Work after bounded repair.

Checks:

• Inspect metadata_quality.overall_quality_status and overall_quality_issues for the affected rows.
• Check repair_attempts, fallback_used, and fallback_reason.
• Review logs/latest_run.log for metadata proof, identifier router, or model timeout messages.
• Compare the image with Caption, alt text, and Keywords directly: a proof pass validates structure, duplication, and context, not semantic correctness.

Fix path:

1. Fix the generic proof or repair rule causing the failure.
2. Do not create per-topic patches for a single batch.
3. Keep automated repair limited to Caption, alt text, and Keywords; preserve selected Subject, Location, and Folder/category.
4. Run a mixed sample before restarting a large queue.

R3.5 Publish failures (FTP/MySQL)

Symptoms: uploader errors, partial remote updates, or missing published records.

Checks:

• Inspect logs/db_uploader.log and identify first failed item.
• Validate FTP/MySQL host credentials and network reachability.
• Confirm publish target values in amir2000_config.py.

Fix path:

1. Resolve auth/connectivity issue.
2. Retry publish for approved rows.
3. Verify upsert completion by File_Name in MySQL table photos_info_revamp.

R3.6 SQLite lock or inconsistency

• Close external processes that may hold data/review.db lock.
• Back up current data/ before destructive actions.
• If reset is required, run python .\init_db.py and rerun batch.

R3.7 Filename collision issues

• Inspect data/used_filenames.json and destination folder contents.
• Use editor rename flow so reservation updates remain consistent.
• Avoid manual renames outside workflow scripts for in-flight items.

R3.8 App crash or forced close (continue safely)

Symptoms: app window closes unexpectedly, or Windows Event Viewer shows native crash events (for example 0xc0000005 / BEX64).

Safe continue path:

1. Restart the app.
2. Click Recover crash session next to Clear all.
3. Verify recovered queue size, then continue with Start Batch.

Recovery file: data/multiset_session.json. Optional backups: data/multiset_session.backup_*.json. Recovery checks both incoming and staged file locations before restore.

If crash happened while clicking Add set, inspect data/crash_runtime.log first. For high-volume queue building, you can disable automatic subject generation with AUTO_AI_SUBJECT_ON_SELECT=0 and run AI suggest manually per set.

R3.9 When to escalate to runbook flow

• Repeated failures after one clean retry.
• Unclear partial state after crash/interruption.
• Any issue that may require rollback of staged files or reserved names.

Escalation path: Step 3: Runbook.

R3.10 Keyword context pollution (geo mismatch)

Symptoms: unrelated place or terrain keywords appear in images whose selected location and visible scene do not support them.

Checks:

• Inspect row Location/Subject vs generated Keywords in review editor.
• Confirm taxonomy sources in data/location_list.json and data/folder_map.json.
• Review recent prefill behavior in caption_review_local.py and runtime logs.

Fix path:

1. Remove unrelated keywords in review editor for current run.
2. Keep keyword generation location-aware and avoid hardcoded geography pools.
3. For already-published rows, scope any remediation to the affected records, verify the selection against a backup, and republish only after validation.

Keep remediation run-scoped and reviewable. Do not apply a cleanup query to the wider catalog without first confirming the exact affected records.

R3.10A Generate button issues in review editor

Symptoms: regenerate fails, or regenerated text repeats pending-row captions.

Checks:

• Confirm caption_review_local.py is importable in the current runtime (no missing dependency errors in Generate dialog).
• Confirm row has a valid image path (ollama_path fallback to Path).
• Confirm pending-row uniqueness ledger is being built (duplicate captions should be rejected before save).

Fix path:

1. Repair missing dependency/runtime issue first (then retry Generate).
2. If regenerate still fails, inspect logs/latest_run.log and retry after confirming the primary model/runtime path.
3. Generate should refresh Caption, alt text, and Keywords only; confirm the selected Subject, Location, and Folder/category remain unchanged.
4. Keep manual review as final gate; do not publish regenerated text without validation.

R3.11 Semantic drift (terrain/time-of-day mismatch)

Symptoms: captions, alt text, or keywords conflict with obvious scene context, such as unsupported terrain or waterway terms, or a filename time-of-day cue that conflicts with generated wording.

Checks:

• Compare File_Name cue words (sunrise/sunset) against generated Caption, alt_text, and Keywords.
• Compare row context (Subject/Location/Folder) against terrain-heavy output terms.
• Flag terrain or waterway terms that are not visibly supported by the image or selected location context.
• Review prefill lines in logs/latest_run.log for the affected row IDs.

Fix path:

1. For current queue rows, edit conflicting metadata directly in review editor and continue.
2. For recurrent drift, confirm current build includes context guardrails in caption_review_local.py and rebuild the EXE.
3. Confirm fallback wording remains context-grounded and does not substitute generic placeholders for image-specific description.
4. Re-run only affected queued rows where needed.

R3.12 Text quality artifacts (hyphen/grammar)

Symptoms: malformed fragments such as City- view, view of along, article-verb starts like An crosses ..., or generic filler lines like scene appears in open terrain / scene view of scene.

Checks:

• Inspect generated Caption and alt_text in review editor before publish.
• Confirm build includes sentence sanitation and style checks in caption_review_local.py.
• Use shared text-field spellcheck in review editor (Caption/alt_text/Keywords) to catch typos while doing manual corrections.
• Run a small dry run on representative files before large batch reruns.

Fix path:

1. Update to the latest build with sentence cleanup and malformed-start rejection.
2. Reject or rewrite generic filler phrases; final output should describe visible evidence only.
3. Re-run only affected queued rows and validate output quality before publish.

R3.13 Continuation path