Commit Graph
12 Commits
Author SHA1 Message Date
Herculino Trotta d763d0ffb8 fix(import): leave QIF headers out of the hash and skip non-transaction sections
The !Type header line was buffered into the first transaction's raw
lines, so a transaction hashed differently depending on whether it
opened the export. An overlapping export with a different first
transaction imported it twice. Header lines are now left out of the
internal ID. The previous-hash check still hashes the full lines,
headers included, and runs on the first appearance of each of those,
so a first transaction and an identical later one imported before
this change are both still recognized.

Account blocks (!Account and !Option:AutoSwitch account lists) and
non-transaction sections (!Type:Cat, !Type:Class, !Type:Memorized and
any type not in QIF_TRANSACTION_SECTIONS) were read with the
transaction codes, where T is an account's type and D a description,
so files containing them failed to import. They are now skipped, and
each skipped section is logged.
2026-11-14 12:16:15 -03:00
Asish Kumar 70be7b3161 fix(import): make QIF deduplication account-aware and keep identical transactions
QIF imports deduplicate each transaction by storing a SHA-256 of its raw
lines as the transaction's internal_id and skipping rows whose hash is
already taken. The hash didn't include the account, which broke two
cases:

- Importing the same file into two accounts (possible since the account
  can be picked per run) skipped every row of the second import.
- Two real transactions with the same date, amount and payee in one file
  hashed to the same ID, so the second one was always dropped.

The internal ID now hashes the account's ID and the transaction's
occurrence (how many identical transactions came before it in the file)
together with the raw lines. Using the occurrence instead of the
position in the file keeps the IDs stable when a later export of the
same statement has more transactions before or between the repeated
ones.

Transactions imported before this change keep their old IDs. So that
re-importing an old file doesn't duplicate them, the first occurrence
of a transaction is also matched against the previous hash, limited to
the account being imported into. Repeats of identical transactions,
which the old hash always skipped, are imported now.

The duplicate check also looks at soft-deleted transactions, like the
CSV/Excel deduplication does. internal_id is unique, so a soft-deleted
transaction with the same ID used to fail the import with an integrity
error instead of being skipped.

Fixes #671
2026-11-12 15:44:27 +05:30
Herculino Trotta df6ccda7b3 feat(import): read any line of the file with a line transformation
Some banks name the account once in a header cell above the table, which
skip_lines and start_row throw away. A `line` transformation returns a line
of the file (or one of its columns) as text, so a mapping can read it and
clean it up with regex or replace.

CSV keeps only the requested lines during the existing counting pass; Excel
reads them straight from each sheet. An account picked on the run still
overrides the mapped one.
2026-11-12 00:17:14 -03:00
Herculino Trotta e617e809c0 fix(import): reject archived accounts when importing through the API
The upload form only lists active accounts, but the API accepted any account
the user can access, archived ones included.
2026-11-12 00:14:06 -03:00
Asish Kumar 1ed5dc5d31 feat(import): pick the account when running an import
An import run can now be given an account, and every transaction it
creates goes into that account instead of the one the profile maps. This
lets a single profile serve several accounts at the same bank, for
statements that only name the account in a header cell or not at all,
instead of keeping one copy of the profile per account.

- ImportRun gets an optional `account` (SET_NULL), shown on the run card.
- The upload form offers the user's active accounts when the profile
  imports transactions, and leaves the field out for profiles that import
  accounts, currencies, categories, tags or entities.
- CSV and Excel rows take the run's account before deduplication, so
  rules comparing `account` check against the picked account. The
  profile's account mapping is skipped for these rows.
- QIF files use the run's account instead of the account named after the
  file.
- The import API accepts an optional `account_id`, limited to accounts
  the user can access, rejected for profiles that don't import
  transactions, and returns `account` on import runs.

Leaving the account empty keeps the current behaviour.

Closes #538
2026-11-11 22:41:37 +05:30
Herculino Trotta e0c1e4d5ff fix(import): resolve symlinks when checking the upload path
The temp root is resolve()d in settings but the file path was only
abspath()ed, so a symlink inside the temp folder could point outside it
and still pass the check. Resolve both sides the same way.

Also drop the unused /temp folder from the prod image.
2026-11-08 03:09:04 -03:00
Asish Kumar 6cf8cbf786 fix(import): stop hardcoding the Docker image's paths
Import uploads were always written to /usr/src/app/temp, presets were
read from /usr/src/app/import_presets and the caches lived in /var/tmp,
so WYGIWYH only worked from the Docker image's layout. Running it as a
systemd service with a strict sandbox, or on an immutable distro like
NixOS, needs these folders to follow the app or be configurable.

- Presets are read from IMPORT_PRESETS_ROOT, the import_presets folder
  next to the app.
- Uploads wait in IMPORT_TEMP_ROOT, the temp folder next to the app by
  default, and movable with the IMPORT_TEMP_ROOT env var. The web views,
  the API and the import service all use it.
- The file-based caches live under CACHE_ROOT, /var/tmp by default, and
  movable with the CACHE_ROOT env var.

In the Docker image the app sits in /usr/src/app, so every default
resolves to the same folder as before.

The import service checked that a file was inside the temp folder with a
plain string prefix, which also accepted sibling folders such as
/usr/src/app/temp-other. It now compares paths.

CI no longer needs to create /usr/src/app before running the tests.

Fixes #472
2026-11-07 14:56:55 +05:30
Herculino Trotta 1e486a608e fix(import): enforce object-level authorization on import profiles and runs
ImportProfile and ImportRun had no owner, and neither the web views nor the
REST viewsets filtered by the requesting user, so any authenticated user could
read, edit and delete every other user's import profiles and runs
(GHSA-q668-h56g-mfp4).

- ImportProfile is now a SharedObject, with share and take-ownership views
  like transaction rules; the global unique constraint on name is dropped.
- ImportRun is now an OwnedObject owned by the uploader, so runs made with a
  shared profile stay private to whoever ran them.
- Existing runs are backfilled with the owner of their imported
  transactions; existing profiles stay unowned and usable by everyone.
2026-10-24 13:26:37 -03:00
Herculino Trotta 5527389196 fix: deduplication breaks when given m2m fields 2026-04-18 14:47:19 +00:00
Herculino Trotta d5bbad7887 feat: add QIF import 2026-01-25 16:46:56 -03:00
Herculino Trotta c7ff6db0bf feat(app): add sanity checks for env variables 2025-12-26 09:55:57 -03:00
Herculino Trotta 05ede58c36 fix: "lax" deduplication fails if the comparison field has a numeric value 2025-12-20 00:17:22 -03:00