Commit Graph
28 Commits
Author SHA1 Message Date
Herculino Trotta df6ccda7b3 feat(import): read any line of the file with a line transformation
Some banks name the account once in a header cell above the table, which
skip_lines and start_row throw away. A `line` transformation returns a line
of the file (or one of its columns) as text, so a mapping can read it and
clean it up with regex or replace.

CSV keeps only the requested lines during the existing counting pass; Excel
reads them straight from each sheet. An account picked on the run still
overrides the mapped one.
2026-11-12 00:17:14 -03:00
Asish Kumar 1ed5dc5d31 feat(import): pick the account when running an import
An import run can now be given an account, and every transaction it
creates goes into that account instead of the one the profile maps. This
lets a single profile serve several accounts at the same bank, for
statements that only name the account in a header cell or not at all,
instead of keeping one copy of the profile per account.

- ImportRun gets an optional `account` (SET_NULL), shown on the run card.
- The upload form offers the user's active accounts when the profile
  imports transactions, and leaves the field out for profiles that import
  accounts, currencies, categories, tags or entities.
- CSV and Excel rows take the run's account before deduplication, so
  rules comparing `account` check against the picked account. The
  profile's account mapping is skipped for these rows.
- QIF files use the run's account instead of the account named after the
  file.
- The import API accepts an optional `account_id`, limited to accounts
  the user can access, rejected for profiles that don't import
  transactions, and returns `account` on import runs.

Leaving the account empty keeps the current behaviour.

Closes #538
2026-11-11 22:41:37 +05:30
Herculino Trotta e0c1e4d5ff fix(import): resolve symlinks when checking the upload path
The temp root is resolve()d in settings but the file path was only
abspath()ed, so a symlink inside the temp folder could point outside it
and still pass the check. Resolve both sides the same way.

Also drop the unused /temp folder from the prod image.
2026-11-08 03:09:04 -03:00
Asish Kumar 6cf8cbf786 fix(import): stop hardcoding the Docker image's paths
Import uploads were always written to /usr/src/app/temp, presets were
read from /usr/src/app/import_presets and the caches lived in /var/tmp,
so WYGIWYH only worked from the Docker image's layout. Running it as a
systemd service with a strict sandbox, or on an immutable distro like
NixOS, needs these folders to follow the app or be configurable.

- Presets are read from IMPORT_PRESETS_ROOT, the import_presets folder
  next to the app.
- Uploads wait in IMPORT_TEMP_ROOT, the temp folder next to the app by
  default, and movable with the IMPORT_TEMP_ROOT env var. The web views,
  the API and the import service all use it.
- The file-based caches live under CACHE_ROOT, /var/tmp by default, and
  movable with the CACHE_ROOT env var.

In the Docker image the app sits in /usr/src/app, so every default
resolves to the same folder as before.

The import service checked that a file was inside the temp folder with a
plain string prefix, which also accepted sibling folders such as
/usr/src/app/temp-other. It now compares paths.

CI no longer needs to create /usr/src/app before running the tests.

Fixes #472
2026-11-07 14:56:55 +05:30
Herculino Trotta 5527389196 fix: deduplication breaks when given m2m fields 2026-04-18 14:47:19 +00:00
Herculino Trotta d5bbad7887 feat: add QIF import 2026-01-25 16:46:56 -03:00
Herculino Trotta 05ede58c36 fix: "lax" deduplication fails if the comparison field has a numeric value 2025-12-20 00:17:22 -03:00
Herculino Trotta c55d688956 fix(import:v1): always_* types for is_paid and type requires assigning a source 2025-12-06 17:52:46 -03:00
Herculino Trotta 020dd74f80 feat: multi tenancy support 2025-03-08 12:03:17 -03:00
Herculino Trotta 6ec5b5df1e feat(import:v1): add XLS and XLSX support
Closes #47
2025-02-09 00:51:26 -03:00
Herculino Trotta 9e27fef5e5 feat(import:v1): add "add" and "subtract" transformations 2025-02-08 18:30:06 -03:00
Herculino Trotta d9e8be7efb feat(import:v1): allow to source previously mapped data by prefixing it with "__" on transformations 2025-02-08 16:38:36 -03:00
Herculino Trotta 43eb87d3ba feat(import:v1): allow to source previously mapped data by prefixing it with "__" 2025-02-08 15:59:27 -03:00
Herculino Trotta a7d8f94412 fix: empty strings not considered as None when importing 2025-01-31 16:54:04 -03:00
Herculino Trotta 9fa704811c feat(import:v1): accept list as source, first valid one will be used. 2025-01-28 21:24:23 -03:00
Herculino Trotta d50c84f8e6 refactor: remove debug prints 2025-01-24 00:36:33 -03:00
Herculino Trotta f2d32fd7e9 feat(import): final changes for release 2025-01-23 23:52:54 -03:00
Herculino Trotta 1dc03b0a84 feat(import:v1:service): respect create and type fields 2025-01-23 22:48:23 -03:00
Herculino Trotta 38379ab2b1 feat(import): try to be more aggressive on cache invalidation 2025-01-23 21:12:13 -03:00
Herculino Trotta 928ad33111 feat(import): move required field check to end of process 2025-01-23 21:09:53 -03:00
Herculino Trotta d0172b5524 feat(import): convert deduplicate fields field into list 2025-01-23 21:09:21 -03:00
Herculino Trotta 096f24e0a2 feat(import): cleanup 2025-01-23 16:32:08 -03:00
Herculino Trotta cabd03e7e6 feat: presets 2025-01-23 11:43:35 -03:00
Herculino Trotta ece44f2726 feat(import): more UI and endpoints 2025-01-22 10:43:19 -03:00
Herculino Trotta 76df16e489 feat(import:v1:schema): add option for triggering rules 2025-01-22 10:43:18 -03:00
Herculino Trotta 2ff33526ae feat(import): disable cache when running 2025-01-22 10:43:18 -03:00
Herculino Trotta 50efc51f87 feat(import): improve schema definition 2025-01-22 10:43:17 -03:00
Herculino Trotta 493bf268bb feat: rename app, some work on schema 2025-01-22 10:43:17 -03:00