1 changed files with 129 additions and 0 deletions
@ -0,0 +1,129 @@
|
||||
# Guided Local Import Design |
||||
|
||||
## Purpose |
||||
|
||||
Let the local owner place supported financial documents in the ignored |
||||
`imports/` directory and load clean, new transactions through the web app with |
||||
minimal interaction. The owner scans the folder, selects destination accounts |
||||
where needed, imports all ready files, and receives safe status summaries. |
||||
|
||||
The workflow preserves Gizmoduck's existing review boundaries: parsing, |
||||
normalization, validation, duplicate detection, preview, decisions, and |
||||
persistence remain distinct. The UI orchestrates those boundaries; it does not |
||||
replace them or expose source contents in browser messages. |
||||
|
||||
## User workflow |
||||
|
||||
1. The owner copies supported files to the ignored `imports/` directory. |
||||
2. The owner opens **Imports** and selects **Scan imports**. |
||||
3. Gizmoduck lists each safe candidate filename, detected profile, account |
||||
mapping requirements, row counts, and a safe status. |
||||
4. The owner selects an account for each bank CSV and every detected card |
||||
statement section that needs mapping. |
||||
5. The owner selects **Import ready files**. |
||||
6. Each ready file completes its own preview, automatic-decision, and commit |
||||
lifecycle. The page reports created, matched, and skipped counts. |
||||
7. Files needing attention remain uncommitted. The owner corrects the account |
||||
choice or required context, rescans, and imports when the file becomes |
||||
ready. |
||||
|
||||
## Scope and file handling |
||||
|
||||
The scanner examines only direct files under `IMPORT_ROOT`, never follows |
||||
symlinks, rejects traversal, and applies the existing 10 MiB input limit. It |
||||
supports only the existing Banco do Brasil bank CSV and Ourocard TXT formats. |
||||
OFX, PDFs, unknown extensions, and unrecognized layouts are reported as safe |
||||
exceptions and never parsed or committed. |
||||
|
||||
The scanner retains no source rows, descriptions, amounts, or review exports in |
||||
templates, messages, or logs. Those remain local source material governed by |
||||
the data policy. The web UI displays filenames, profile labels, account mapping |
||||
state, safe reason codes, and counts only. |
||||
|
||||
## Readiness and automatic decisions |
||||
|
||||
Each scanned file has one of these statuses: |
||||
|
||||
- **Ready:** every candidate can be created without a validation issue or |
||||
unresolved duplicate. |
||||
- **Already imported:** every source candidate is already attached through |
||||
existing provenance; the file is skipped without financial writes. |
||||
- **Needs attention:** the file has an unsupported/unrecognized profile, |
||||
malformed or unresolved candidate, missing account mapping, ambiguous |
||||
potential duplicate, or missing/invalid statement context. |
||||
|
||||
The import action creates a normal existing preview per ready file. It applies |
||||
`CREATE` to clean candidates. Exact source reimports receive |
||||
`SKIP` with `ALREADY_IMPORTED`; they produce no transaction and are not an |
||||
exception. Any potential duplicate that is not an exact existing source record |
||||
requires review and prevents that file from committing automatically. |
||||
|
||||
An exception in one file does not prevent other ready files from committing. |
||||
Every file retains the import service's atomic commit boundary; a failed file |
||||
creates no partial transactions or provenance. All import lifecycle actions use |
||||
the local human actor and retain existing audits. |
||||
|
||||
## Account mapping |
||||
|
||||
The scan presents an account choice for every source section needed by the |
||||
detected profile. Bank CSV has one section. Ourocard documents may have |
||||
multiple sections and require a choice for each section. The UI uses active |
||||
BRL accounts and cards only. It never maps an owner name or filename to an |
||||
account automatically. |
||||
|
||||
The server validates that every submitted mapping corresponds to the current |
||||
scan and to an active account. Missing, archived, stale, or altered mappings |
||||
are exceptions; they cannot be committed. |
||||
|
||||
## Ourocard partial-statement dates |
||||
|
||||
`OUROCARD_PARTIAL_V1` gains a bounded context extractor. It reads and validates |
||||
the statement creation date from line 21 of the TXT document. That date supplies |
||||
the statement year and an explicit import interval. Candidate month/day values |
||||
later than the creation month belong to the preceding calendar year; remaining |
||||
values belong to the creation year. This handles statements crossing New Year |
||||
without using the computer clock or filesystem metadata. |
||||
|
||||
If line 21 is absent, malformed, ambiguous, or does not yield an interval that |
||||
resolves every candidate date uniquely, the file is **Needs attention** and is |
||||
not committed. The existing explicit parser context remains available for the |
||||
management command; this UI-only extractor supplies equivalent validated |
||||
context before invoking the existing parser contract. |
||||
|
||||
## Components |
||||
|
||||
- `ingestion` receives a folder scanner, safe profile detector, and a |
||||
UI-orchestration service that constructs existing previews and applies only |
||||
the approved automatic decisions. |
||||
- `core` receives Import navigation, scan/mapping/import forms and views, and |
||||
server-rendered status templates. |
||||
- Existing parser, normalization, validation, duplicate, persistence, audit, |
||||
and import-batch models remain their own boundaries. |
||||
- The management command remains supported for advanced/manual reviewed import |
||||
workflows. |
||||
|
||||
## Error handling and security |
||||
|
||||
All POST actions use CSRF protection and revalidate scan tokens, file identity, |
||||
profile, mappings, and source metadata before creating a preview or committing. |
||||
The UI returns field errors and safe status codes. It does not reveal source |
||||
text, paths outside the allowed folder, raw parser exceptions, or database |
||||
errors. Stale scans require a new scan; the server does not reuse a changed |
||||
file under an old readiness result. |
||||
|
||||
## Verification |
||||
|
||||
Tests use only synthetic fixtures and cover: |
||||
|
||||
- direct-folder scanning and supported-profile detection; |
||||
- rejected path/symlink/oversize/unsupported input without source disclosure; |
||||
- bank and multi-section card account mapping with active BRL references; |
||||
- clean automatic creation through the existing audit and import lifecycle; |
||||
- exact reimport skipping through source provenance; |
||||
- an exceptional file not blocking an unrelated ready file; |
||||
- ambiguous duplicates, malformed candidates, stale scans, mapping tampering, |
||||
and CSRF rejection; |
||||
- line-21 partial-statement date extraction, invalid context, and New Year |
||||
rollover; and |
||||
- focused web tests plus full Django, migration, permission, and integration |
||||
checks. |
||||
Loading…
Reference in new issue