Browse Source

docs: stage guided import files

master
Codex 2 months ago
parent
commit
dce6e6102d
  1. 61
      docs/superpowers/specs/2026-09-09-guided-local-import-design.md

61
docs/superpowers/specs/2026-09-09-guided-local-import-design.md

@ -16,24 +16,34 @@ replace them or expose source contents in browser messages.
1. The owner copies supported files to the ignored `imports/` directory.
2. The owner opens **Imports** and selects **Scan imports**.
3. Gizmoduck lists each safe candidate filename, detected profile, account
3. Gizmoduck returns interrupted staging files to the incoming folder, then
lists each safe candidate filename, detected profile, account
mapping requirements, row counts, and a safe status.
4. The owner selects an account for each bank CSV and every detected card
statement section that needs mapping.
5. The owner selects **Import ready files**.
6. Each ready file completes its own preview, automatic-decision, and commit
lifecycle. The page reports created, matched, and skipped counts.
6. Each ready file moves through staging, preview, automatic-decision, and
commit. A committed file moves to processed. The page reports created,
matched, and skipped counts.
7. Files needing attention remain uncommitted. The owner corrects the account
choice or required context, rescans, and imports when the file becomes
ready.
## Scope and file handling
The scanner examines only direct files under `IMPORT_ROOT`, never follows
symlinks, rejects traversal, and applies the existing 10 MiB input limit. It
supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats.
OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe
exceptions and never parsed or committed.
`IMPORT_ROOT` has three local, ignored folders: incoming files at `imports/`,
temporary work at `imports/staging/`, and successful source documents at
`imports/processed/`. The scanner examines only direct files in the incoming
folder; it has no fixed file-count limit. It never follows symlinks, rejects
traversal, and applies the existing 10 MiB input limit. It supports only the
existing Banco do Brasil bank CSV and Ourocard TXT formats. OFX, PDFs, unknown
extensions, and unrecognized layouts are reported as safe exceptions and never
parsed or committed.
Before each scan, Gizmoduck moves direct regular files left in `staging/` back
to incoming. This recovers a run interrupted between staging and its final
move. Neither `staging/` nor `processed/` is included in normal candidate
scanning.
The scanner retains no source rows, descriptions, amounts, or review exports in
templates, messages, or logs. Those remain local source material governed by
@ -63,6 +73,22 @@ Every file retains the import service's atomic commit boundary; a failed file
creates no partial transactions or provenance. All import lifecycle actions use
the local human actor and retain existing audits.
## Local file lifecycle
Before creating a preview, Gizmoduck atomically moves each selected incoming
file to `staging/` on the same local volume. It then reads and verifies the
staged file, prepares it through the existing import lifecycle, and commits it.
After a successful commit, it moves the staged source file to `processed/`.
An existing processed file with the same name is overwritten only at this final
successful step.
If preparation, validation, duplicate review, commit, or a pre-commit file
operation fails, Gizmoduck moves the staged file back to incoming and reports a
safe exception. If the process stops after a database commit but before the
processed move, the next scan restores that file to incoming. Existing
provenance then prevents duplicate transactions; the automatic exact-source
skip path may move it to processed.
## Account mapping
The scan presents an account choice for every source section needed by the
@ -104,12 +130,12 @@ context before invoking the existing parser contract.
## Error handling and security
All POST actions use CSRF protection and revalidate scan tokens, file identity,
profile, mappings, and source metadata before creating a preview or committing.
The UI returns field errors and safe status codes. It does not reveal source
text, paths outside the allowed folder, raw parser exceptions, or database
errors. Stale scans require a new scan; the server does not reuse a changed
file under an old readiness result.
All POST actions use CSRF protection and revalidate file identity, profile,
mappings, and source metadata before creating a preview or committing. The UI
returns field errors and safe status codes. It does not reveal source text,
paths outside the allowed folder, raw parser exceptions, or database errors.
Staging identity is verified before every lifecycle step; a changed staged file
returns to incoming without commitment.
## Verification
@ -117,12 +143,15 @@ Tests use only synthetic fixtures and cover:
- direct-folder scanning and supported-profile detection;
- rejected path/symlink/oversize/unsupported input without source disclosure;
- staged success moving to processed, exception return to incoming, interrupted
staging recovery, and processed-name overwrite only after a successful
commit;
- bank and multi-section card account mapping with active BRL references;
- clean automatic creation through the existing audit and import lifecycle;
- exact reimport skipping through source provenance;
- an exceptional file not blocking an unrelated ready file;
- ambiguous duplicates, malformed candidates, stale scans, mapping tampering,
and CSRF rejection;
- ambiguous duplicates, malformed candidates, mapping tampering, and CSRF
rejection;
- line-21 partial-statement date extraction, invalid context, and New Year
rollover; and
- focused web tests plus full Django, migration, permission, and integration

Loading…
Cancel
Save