docs: plan OpenAI transcription implementation

docs/superpowers/plans/2026-08-30-openai-transcription.md

@@ -0,0 +1,199 @@
+# OpenAI Transcription Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add independent OpenAI transcription of prepared chunks with Brazilian-Portuguese prompting, durable incremental manifests, resumability, finite retries, cancellation, and responsive GUI progress.
+
+**Architecture:** Extend the existing preprocessing manifest into a versioned durable job stored beneath the configured output directory. A pure manifest repository owns atomic state changes; an OpenAI adapter owns one SDK request; a transcription orchestrator owns resume, retries, progress, cancellation, and ordered transcript assembly; Qt only adapts worker callbacks into UI signals.
+
+**Tech Stack:** Python 3.12+, OpenAI Python SDK 3.6.0-compatible interface, PySide6, keyring/Windows Credential Manager, pytest, pytest-qt, FFmpeg/ffprobe.
+
+**Spec:** `docs/superpowers/specs/2026-08-30-openai-transcription-design.md`
+
+## Global Constraints
+
+- Use `client.audio.transcriptions.create`, not the translation endpoint.
+- Default model is exactly `gpt-transcribe`; never silently substitute another model.
+- Seed model choices with `gpt-transcribe`, `gpt-4o-transcribe`, and `gpt-4o-mini-transcribe`, while accepting future strings.
+- Present Brazilian Portuguese as `pt-BR` and send documented ISO-639-1 `pt` to the API.
+- Read the API key from the exact Windows Credential Manager target `OPENAI_API_KEY`; never persist or log it elsewhere.
+- Every successful chunk must be atomically recorded before the next request.
+- Never retranscribe a completed chunk except through an explicit reset/retranscribe operation.
+- No overlap deduplication or translation in this milestone.
+- All network and job work remains outside the GUI thread.
+- Use test-driven development: add one failing behavior test, verify RED, implement minimally, verify GREEN, then refactor.
+
+---
+
+## File Map
+
+- `src/voice_transcriptor/models.py`: add context/vocabulary to `AppSettings`.
+- `src/voice_transcriptor/services/settings.py`: persist new defaults and migrate older files.
+- `src/voice_transcriptor/services/credentials.py`: target-based Windows credential lookup/update.
+- `src/voice_transcriptor/services/job_manifest.py`: atomic schema-2 job persistence, recovery, resume, and transcript assembly.
+- `src/voice_transcriptor/services/preprocessing.py`: durable output-root jobs and per-chunk preprocessing status updates.
+- `src/voice_transcriptor/services/transcription.py`: prompt normalization, SDK adapter, error classification, retries, cancellation, and orchestration.
+- `src/voice_transcriptor/ui/settings_dialog.py`: editable model combo and context editor.
+- `src/voice_transcriptor/ui/main_window.py`: combined job worker, progress labels, elapsed time, API errors, resume and cancellation.
+- `src/voice_transcriptor/app.py`: concrete OpenAI/job service wiring.
+- `tests/test_settings.py`, `tests/test_credentials.py`: settings and exact credential-target coverage.
+- `tests/test_job_manifest.py`: durable state and resume invariants.
+- `tests/test_preprocessing.py`: durable job integration.
+- `tests/test_transcription.py`: API arguments, retry/error behavior, persistence, resume, and cancellation.
+- `tests/ui/test_settings_dialog.py`, `tests/ui/test_main_window.py`, `tests/ui/test_app.py`: GUI settings, worker progress, errors, cancellation, and startup.
+- `README.md`: API credential, model/context, job/resume, and validation documentation.
+
+### Task 1: Settings and Existing Windows Credential Target
+
+**Files:**
+- Modify: `src/voice_transcriptor/models.py`
+- Modify: `src/voice_transcriptor/services/settings.py`
+- Modify: `src/voice_transcriptor/services/credentials.py`
+- Modify: `tests/test_settings.py`
+- Modify: `tests/test_credentials.py`
+
+**Interfaces:**
+- Produces: `DEFAULT_CONTEXT`, `SUPPORTED_MODEL_SUGGESTIONS`, and `AppSettings(..., context_vocabulary: str = DEFAULT_CONTEXT)`.
+- Produces: `CredentialService.get_api_key() -> str | None`, `has_api_key() -> bool`, and `set_api_key(value: str) -> None` using target `OPENAI_API_KEY` and backend `get_credential(service_name, username=None)`.
+- Preserves all existing public settings and credential methods.
+
+- [ ] **Step 1: Add failing settings tests** asserting the default model is `gpt-transcribe`, language remains `pt-BR`, the default context contains Brazilian guidance and vocabulary examples, older JSON files receive that context, and save/load round-trips it without an API key.
+- [ ] **Step 2: Run `python -m pytest tests/test_settings.py -v`** and verify failures report the old model or missing `context_vocabulary` field.
+- [ ] **Step 3: Implement the minimal settings/model changes** by appending the dataclass field (to preserve positional callers), updating defaults, parsing optional string context, and serializing it.
+- [ ] **Step 4: Re-run `python -m pytest tests/test_settings.py -v`** and confirm GREEN.
+- [ ] **Step 5: Add failing credential tests** with a fake backend whose `get_credential("OPENAI_API_KEY", None)` returns an object containing `username` and `password`; assert retrieval, absence, sanitized backend failures, update preserving username, and creation using username `OPENAI_API_KEY`.
+- [ ] **Step 6: Run `python -m pytest tests/test_credentials.py -v`** and verify RED because the current service queries `voice-transcriptor` / `openai-api-key` with `get_password`.
+- [ ] **Step 7: Implement exact-target lookup/update** using `TARGET_NAME = "OPENAI_API_KEY"`, `get_credential`, and `set_password(TARGET_NAME, resolved_username, value)` without exposing backend details.
+- [ ] **Step 8: Run `python -m pytest tests/test_settings.py tests/test_credentials.py -v`** and confirm GREEN.
+- [ ] **Step 9: Commit** with `git add src/voice_transcriptor/models.py src/voice_transcriptor/services/settings.py src/voice_transcriptor/services/credentials.py tests/test_settings.py tests/test_credentials.py && git commit -m "feat: configure transcription models and credential target"`.
+
+### Task 2: Atomic Resumable Job Manifest
+
+**Files:**
+- Create: `src/voice_transcriptor/services/job_manifest.py`
+- Create: `tests/test_job_manifest.py`
+
+**Interfaces:**
+- Produces: `ChunkStatus(StrEnum)` with `PENDING`, `PROCESSING`, `COMPLETED`, `FAILED`.
+- Produces: `JobManifestRepository.create(job_directory: Path, source: dict, settings: dict, chunks: Sequence[dict]) -> dict`.
+- Produces: `load(path: Path) -> dict`, `save(path: Path, manifest: dict) -> None`, `recover_for_resume(path: Path) -> dict`, `mark_processing(path, index)`, `mark_completed(path, index, text)`, `mark_failed(path, index, message)`, `mark_job_state(path, state, error=None)`, and `reset_completed(path, indexes=None)`.
+- Produces: `assemble_transcript(manifest_path: Path) -> Path` writing sibling `transcript.txt` atomically in chunk-index order.
+
+- [ ] **Step 1: Write failing manifest tests** for schema 2 creation, exact decimal timing preservation, four chunk statuses, safe relative paths, atomic `.tmp` replacement, completed counters, and rejection of malformed/out-of-directory chunk paths.
+- [ ] **Step 2: Run `python -m pytest tests/test_job_manifest.py -v`** and verify import failure.
+- [ ] **Step 3: Implement creation, validation, load, and atomic save** with UTF-8 JSON and ISO-8601 UTC timestamps.
+- [ ] **Step 4: Re-run the focused tests** and confirm the first state tests pass.
+- [ ] **Step 5: Add failing transition tests** asserting each mutator immediately changes disk state, `processing` recovers to `pending`, `completed` survives recovery, explicit resume resets `failed`, and ordinary transitions refuse to overwrite completed chunks.
+- [ ] **Step 6: Run the focused tests** and verify RED on missing transition methods.
+- [ ] **Step 7: Implement transition and recovery methods** with index validation, attempt counts, sanitized error strings, and completed-count recalculation.
+- [ ] **Step 8: Add failing assembly tests** asserting completed texts are newline-concatenated in index order, partial transcripts contain only completed chunks, and source timing remains unchanged.
+- [ ] **Step 9: Implement atomic transcript assembly** and explicit completed-reset behavior.
+- [ ] **Step 10: Run `python -m pytest tests/test_job_manifest.py -v`** and confirm GREEN.
+- [ ] **Step 11: Commit** with `git add src/voice_transcriptor/services/job_manifest.py tests/test_job_manifest.py && git commit -m "feat: add durable resumable job manifests"`.
+
+### Task 3: Durable Preprocessing Jobs
+
+**Files:**
+- Modify: `src/voice_transcriptor/services/preprocessing.py`
+- Modify: `tests/test_preprocessing.py`
+
+**Interfaces:**
+- Consumes: `JobManifestRepository` and its schema-2 chunk records.
+- Changes: `PreprocessingOptions` gains `job_root: Path | None = None` or `PreprocessingService.preprocess(..., job_root: Path | None = None)`; the implementation must receive the configured persistent root rather than use a system temporary directory for transcribable jobs.
+- Produces: `PreprocessingResult` whose directory remains durable and whose manifest contains independently reusable encoded chunks.
+
+- [ ] **Step 1: Add failing preprocessing tests** asserting jobs are created below `<output>/voice-transcriptor-jobs`, a pending transcript status is written after every successful encoded chunk, cancellation/failure retains the job, and existing valid chunks can be reused on resume.
+- [ ] **Step 2: Run `python -m pytest tests/test_preprocessing.py -v`** and verify RED against temporary-directory deletion/schema 1 behavior.
+- [ ] **Step 3: Inject `JobManifestRepository` and durable root selection** while preserving the existing FFmpeg command, exact boundaries, and atomic per-chunk writes.
+- [ ] **Step 4: Implement preprocessing resume** so valid existing chunk files are skipped, missing files are regenerated, completed transcript text is never cleared, and partial current outputs are removed after FFmpeg failure.
+- [ ] **Step 5: Run `python -m pytest tests/test_preprocessing.py tests/test_chunking.py -v`** and confirm GREEN.
+- [ ] **Step 6: Commit** with `git add src/voice_transcriptor/services/preprocessing.py tests/test_preprocessing.py && git commit -m "feat: persist and resume preprocessing jobs"`.
+
+### Task 4: OpenAI Adapter, Prompting, Retry Policy, and Orchestrator
+
+**Files:**
+- Replace: `src/voice_transcriptor/services/transcription.py`
+- Replace: `tests/test_transcription.py`
+
+**Interfaces:**
+- Produces: `normalize_language(value: str) -> str` mapping case-insensitive `pt-BR`/`pt_BR` to `pt` and otherwise using the primary ISO language subtag.
+- Produces: `build_prompt(context: str) -> str` combining the fixed no-translation Brazilian instruction with trimmed optional context.
+- Produces: `OpenAITranscriptionClient(client)` with `transcribe(path: Path, model: str, language: str, prompt: str) -> str` calling `client.audio.transcriptions.create` exactly once.
+- Produces: `RetryPolicy(max_attempts=5, initial_delay_seconds=1.0, max_delay_seconds=30.0, jitter_ratio=0.2)`.
+- Produces: `TranscriptionProgress(completed: int, total: int, current_chunk: int | None, elapsed_seconds: float, phase: str, message: str, api_error: str | None = None)`.
+- Produces: `TranscriptionService.run(manifest_path, settings, token, progress) -> Path` returning `transcript.txt`.
+- Produces typed `TranscriptionError`, `PermanentTranscriptionError`, and `RetryExhaustedError` with sanitized messages.
+
+- [ ] **Step 1: Write failing pure tests** for language normalization, fixed no-translation prompt content, optional vocabulary trimming, and arbitrary models.
+- [ ] **Step 2: Run `python -m pytest tests/test_transcription.py -v`** and verify RED because the service is still a milestone stub.
+- [ ] **Step 3: Implement only the pure helpers** and re-run those tests to GREEN.
+- [ ] **Step 4: Add a failing SDK adapter test** using a fake endpoint; assert the opened binary file, exact model, `language="pt"`, combined `prompt`, `response_format="json"`, and returned `.text`. Assert no translation method is touched.
+- [ ] **Step 5: Implement `OpenAITranscriptionClient`** with the verified installed-SDK call signature and a context manager for the chunk file.
+- [ ] **Step 6: Add failing retry-classification tests** for `RateLimitError`, `APIConnectionError`, `APITimeoutError`, 408/409/429/5xx, authentication/permission/bad request, maximum attempts, exponential delays, jitter bounds, and capped `Retry-After`.
+- [ ] **Step 7: Implement deterministic injectable retry dependencies** (`sleep`, monotonic clock, random source) and classification by SDK class/status code. Disable SDK retries during app wiring so only this policy applies.
+- [ ] **Step 8: Add failing orchestration tests** asserting one independent call per unfinished chunk, immediate disk completion after each success, skip of completed chunks, recovery of `processing`, explicit retry of failed chunks, timing retention, ordered assembly, cancellation before a request and during backoff, progress fields, and sanitized errors without key/header-like content.
+- [ ] **Step 9: Implement orchestration minimally** around `JobManifestRepository`, checking the token before each call and via cancellation-aware backoff waits.
+- [ ] **Step 10: Run `python -m pytest tests/test_transcription.py tests/test_job_manifest.py -v`** and confirm GREEN.
+- [ ] **Step 11: Commit** with `git add src/voice_transcriptor/services/transcription.py tests/test_transcription.py && git commit -m "feat: transcribe chunks with retries and resume"`.
+
+### Task 5: Settings GUI for Model and Context
+
+**Files:**
+- Modify: `src/voice_transcriptor/ui/settings_dialog.py`
+- Modify: `tests/ui/test_settings_dialog.py`
+
+**Interfaces:**
+- Changes: `model_input` becomes an editable `QComboBox` seeded from `SUPPORTED_MODEL_SUGGESTIONS` while preserving the object name `modelInput`.
+- Produces: multiline `context_input: QPlainTextEdit` with object name `contextVocabularyInput`.
+- Emits the extended `AppSettings` value without exposing the API key.
+
+- [ ] **Step 1: Add failing Qt tests** for the three seeded model suggestions, editable future model value, populated default context, round-trip user context, and validation of a blank model.
+- [ ] **Step 2: Run `python -m pytest tests/ui/test_settings_dialog.py -v`** and verify RED on the missing combo/context controls.
+- [ ] **Step 3: Implement the editable combo and context field** and adapt save logic to `currentText()` and `toPlainText()`.
+- [ ] **Step 4: Run `python -m pytest tests/ui/test_settings_dialog.py tests/test_settings.py -v`** and confirm GREEN.
+- [ ] **Step 5: Commit** with `git add src/voice_transcriptor/ui/settings_dialog.py tests/ui/test_settings_dialog.py && git commit -m "feat: add transcription model and vocabulary settings"`.
+
+### Task 6: GUI Job Worker, Progress, Resume, Errors, and Cancellation
+
+**Files:**
+- Modify: `src/voice_transcriptor/ui/main_window.py`
+- Modify: `tests/ui/test_main_window.py`
+
+**Interfaces:**
+- Consumes: preprocessing service, transcription service, settings repository, credentials.
+- Produces: a `JobWorker(QRunnable)` that preprocesses/resumes then transcribes and emits typed progress, completion, cancellation, and sanitized failure signals.
+- Produces visible widgets `chunk_progress_label`, `elapsed_label`, `current_chunk_label`, and existing read-only log entries for API errors.
+- Changes primary action text to `Transcribe`; retains `Cancel` and thread-pool execution.
+
+- [ ] **Step 1: Add failing widget tests** for Transcribe action text, `0 / 0` chunk progress, elapsed display, current chunk display, and API error log surface.
+- [ ] **Step 2: Run `python -m pytest tests/ui/test_main_window.py -v`** and verify RED.
+- [ ] **Step 3: Add the progress widgets and slots** without changing worker behavior; re-run widget tests to GREEN.
+- [ ] **Step 4: Add failing worker-flow tests** with fake services asserting the click returns without executing network work on the GUI thread, progress updates all required fields, completed results report the transcript path, and controls restore on success/failure/cancellation.
+- [ ] **Step 5: Implement `JobWorker` and connect signals**; retrieve the credential before constructing work, pass no key through signals/logs, and reuse the existing cancellation token.
+- [ ] **Step 6: Add failing resume tests** asserting a retained matching manifest is offered/reused and completed chunks are not reset; test that no implicit retranscribe path exists.
+- [ ] **Step 7: Implement resume selection** using a deterministic matching-manifest service method; retain all job data on terminal cancellation/failure.
+- [ ] **Step 8: Run `python -m pytest tests/ui/test_main_window.py -v`** and confirm GREEN.
+- [ ] **Step 9: Commit** with `git add src/voice_transcriptor/ui/main_window.py tests/ui/test_main_window.py && git commit -m "feat: add responsive transcription job interface"`.
+
+### Task 7: Application Wiring and Integrated Validation
+
+**Files:**
+- Modify: `src/voice_transcriptor/app.py`
+- Modify: `tests/ui/test_app.py`
+- Modify: `README.md`
+
+**Interfaces:**
+- `create_main_window()` wires `OpenAI(api_key=credential, max_retries=0)`, `JobManifestRepository`, durable preprocessing, and `TranscriptionService` without making an API request at startup.
+- Startup remains usable for selecting media and editing settings when the credential is absent; Transcribe reports a concise configuration error.
+
+- [ ] **Step 1: Add a failing app-wiring test** with injected factories asserting `max_retries=0`, no startup API call, exact credential retrieval, and all concrete services reach `MainWindow`.
+- [ ] **Step 2: Run `python -m pytest tests/ui/test_app.py -v`** and verify RED.
+- [ ] **Step 3: Refactor app wiring for injectable factories** and construct the concrete SDK client only when a transcription job starts.
+- [ ] **Step 4: Run `python -m pytest tests/ui/test_app.py tests/ui/test_main_window.py -v`** and confirm GREEN.
+- [ ] **Step 5: Update README** with Windows Credential Manager target `OPENAI_API_KEY`, model behavior (including the documented-model caveat for `gpt-transcribe`), Brazilian context, durable job directory, resume semantics, retries/rate limits, cancellation, and test commands.
+- [ ] **Step 6: Run the complete automated suite:** `python -m pytest -v`.
+- [ ] **Step 7: Run static/startup validation:** `python -m compileall -q src tests`, then `$env:QT_QPA_PLATFORM='offscreen'; python -c "from PySide6.QtCore import QTimer; from PySide6.QtWidgets import QApplication; from voice_transcriptor.app import create_main_window; app=QApplication([]); window=create_main_window(); window.show(); QTimer.singleShot(100, app.quit); app.exec(); window.close()"`.
+- [ ] **Step 8: Run local media-tool checks:** `ffmpeg -version` and `ffprobe -version`; if available, execute the existing generated-fixture preprocessing smoke path without a live OpenAI call.
+- [ ] **Step 9: Audit the result:** `git diff --check`, `git status --short`, and `rg -n "Authorization|Bearer |sk-[A-Za-z0-9]" src tests README.md docs` to confirm no credential/header leakage.
+- [ ] **Step 10: Use `superpowers:verification-before-completion`** and report the exact test, compile, startup, FFmpeg, and audit outputs before claiming completion.
+- [ ] **Step 11: Commit** with `git add src/voice_transcriptor/app.py tests/ui/test_app.py README.md && git commit -m "docs: wire and validate OpenAI transcription"`.