starred/WYGIWYH
1
0
Fork 0
mirror of https://github.com/eitchtee/WYGIWYH.git synced 2026-01-15 05:33:26 +01:00
Code Issues 14 Packages Projects Releases 10 Wiki Activity

Cannot import transactions with comma as amount separator #53

New Issue
Closed
opened 2025-12-28 23:24:48 +01:00 by adam · 12 comments
adam commented 2025-12-28 23:24:48 +01:00
Owner
Copy Link

Originally created by @Rdlgrmpf on GitHub (Aug 14, 2025).

It seems that the import of a csv with amounts like "-2,29" fails.

[2025-08-14 21:13:26] - ERROR - apps.import_app.services.v1 - Fatal error processing row 3

Traceback (most recent call last):

File "/usr/src/app/apps/import_app/services/v1.py", line 623, in _process_row

mapped_data = self._map_row(row)

              ^^^^^^^^^^^^^^^^^^

File "/usr/src/app/apps/import_app/services/v1.py", line 592, in _map_row

value = self._coerce_type(value, mapping)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/src/app/apps/import_app/services/v1.py", line 483, in _coerce_type

return self._coerce_single_type(value, coerce_to, mapping)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/src/app/apps/import_app/services/v1.py", line 503, in _coerce_single_type

return abs(Decimal(value))

           ^^^^^^^^^^^^^^

decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]

Originally created by @Rdlgrmpf on GitHub (Aug 14, 2025). It seems that the import of a csv with amounts like "-2,29" fails. [2025-08-14 21:13:26] - ERROR - apps.import_app.services.v1 - Fatal error processing row 3 Traceback (most recent call last): File "/usr/src/app/apps/import_app/services/v1.py", line 623, in _process_row mapped_data = self._map_row(row) ^^^^^^^^^^^^^^^^^^ File "/usr/src/app/apps/import_app/services/v1.py", line 592, in _map_row value = self._coerce_type(value, mapping) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/apps/import_app/services/v1.py", line 483, in _coerce_type return self._coerce_single_type(value, coerce_to, mapping) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/apps/import_app/services/v1.py", line 503, in _coerce_single_type return abs(Decimal(value)) ^^^^^^^^^^^^^^ decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]
adam closed this issue 2025-12-28 23:24:49 +01:00
adam commented 2025-12-28 23:24:49 +01:00
Author
Owner
Copy Link

@eitchtee commented on GitHub (Aug 14, 2025):

You might need to use a transformation for this, replacing the comma with a dot, e.g.:

amount:
    target: amount
    source: value
    required: true
    transformations:
      - type: replace
        pattern: ","
        replacement: "."

If you csv also has a dot as a thousand separator, remove the dot then replace the comma with a dot (1.234,40):

amount:
    target: amount
    source: value
    transformations:
      - type: replace
        pattern: "."
        replacement: ""
      - type: replace
        pattern: ","
        replacement: "."

Let me know if this fixes your issue.

@eitchtee commented on GitHub (Aug 14, 2025): You might need to use a transformation for this, replacing the comma with a dot, e.g.: ``` amount: target: amount source: value required: true transformations: - type: replace pattern: "," replacement: "." ``` If you csv also has a dot as a thousand separator, remove the dot then replace the comma with a dot (1.234,40): ``` amount: target: amount source: value transformations: - type: replace pattern: "." replacement: "" - type: replace pattern: "," replacement: "." ``` Let me know if this fixes your issue.
adam commented 2025-12-28 23:24:50 +01:00
Author
Owner
Copy Link

@Rdlgrmpf commented on GitHub (Aug 14, 2025):

That worked! Thank you very much and also for the lightning fast reply. Next I will have a look into the rules. So far your tool looks really nice :)

@Rdlgrmpf commented on GitHub (Aug 14, 2025): That worked! Thank you very much and also for the lightning fast reply. Next I will have a look into the rules. So far your tool looks really nice :)
adam commented 2025-12-28 23:24:51 +01:00
Author
Owner
Copy Link

@eitchtee commented on GitHub (Aug 14, 2025):

Thank you, hope you like it. Feel free to create new issues or discussions if you have any problem or suggestions.

About rules, by design they can get really complex as you're running python in a very limited environment, I always recommend taking a look at https://github.com/eitchtee/WYGIWYH/discussions/230#discussioncomment-12795562 as it shows how to push the system farther than most other applications would allow.

@eitchtee commented on GitHub (Aug 14, 2025): Thank you, hope you like it. Feel free to create new issues or discussions if you have any problem or suggestions. About rules, by design they can get really complex as you're running python in a very limited environment, I always recommend taking a look at https://github.com/eitchtee/WYGIWYH/discussions/230#discussioncomment-12795562 as it shows how to push the system farther than most other applications would allow.
adam commented 2025-12-28 23:24:52 +01:00
Author
Owner
Copy Link

@avier99 commented on GitHub (Nov 27, 2025):

@eitchtee I am having major trouble in importing, i constantly get an error that "amount" field is required, where i do have it!!

here is my yaml and csv

CSV :

Date,Entity,Amount,Description,Account
8/22/2023,Dad,"300,000.00",Fees,DBS Bank
8/22/2023,Mom,"400,000.00",Fees,DBS Bank
8/7/2023,GROWW Return,929.24,,DBS Bank
7/28/2023,Tax,10.00,,DBS Bank

And my YAML :

settings:
  file_type: csv
  delimiter: ","
  encoding: utf-8
  skip_lines: 1
  importing: transactions
  trigger_transaction_rules: true
  skip_errors: true

mapping:
  account:
    target: account
    source: Account
    type: name

  date:
    target: date
    source: Date
    format:
      - "%B %d, %Y"

  reference_date:
    target: reference_date
    source: Date
    format:
      - "%B %d, %Y"

  amount:
    target: amount
    source: Amount
    required: true

  description:
    target: description
    source: Description

  notes:
    target: notes
    source: Entity

  entities:
    target: entities
    source: Entity
    type: name
    create: true

  category:
    target: category
    default: Uncategorized

  type:
    target: type
    detection_method: sign

  is_paid:
    target: is_paid
    detection_method: always_paid

  internal_id:
    target: internal_id
    transformations:
      - type: hash
        fields: ["Date", "Entity", "Amount", "Account"]

deduplication:
  - type: compare
    fields:
      - internal_id
    match_type: strict
@avier99 commented on GitHub (Nov 27, 2025): @eitchtee I am having major trouble in importing, i constantly get an error that "amount" field is required, where i do have it!! here is my yaml and csv CSV : ```csv Date,Entity,Amount,Description,Account 8/22/2023,Dad,"300,000.00",Fees,DBS Bank 8/22/2023,Mom,"400,000.00",Fees,DBS Bank 8/7/2023,GROWW Return,929.24,,DBS Bank 7/28/2023,Tax,10.00,,DBS Bank ``` And my YAML : ```yaml settings: file_type: csv delimiter: "," encoding: utf-8 skip_lines: 1 importing: transactions trigger_transaction_rules: true skip_errors: true mapping: account: target: account source: Account type: name date: target: date source: Date format: - "%B %d, %Y" reference_date: target: reference_date source: Date format: - "%B %d, %Y" amount: target: amount source: Amount required: true description: target: description source: Description notes: target: notes source: Entity entities: target: entities source: Entity type: name create: true category: target: category default: Uncategorized type: target: type detection_method: sign is_paid: target: is_paid detection_method: always_paid internal_id: target: internal_id transformations: - type: hash fields: ["Date", "Entity", "Amount", "Account"] deduplication: - type: compare fields: - internal_id match_type: strict ```
adam commented 2025-12-28 23:24:52 +01:00
Author
Owner
Copy Link

@eitchtee commented on GitHub (Nov 28, 2025):

@avier99 try removing the comma thousand-separator from your value using a transformation:

amount:
    target: amount
    source: Amount
    transformations:
      - type: replace
        pattern: ","
        replacement: ""

Let me know if it works.

@eitchtee commented on GitHub (Nov 28, 2025): @avier99 try removing the comma thousand-separator from your value using a transformation: ``` amount: target: amount source: Amount transformations: - type: replace pattern: "," replacement: "" ``` Let me know if it works.
adam commented 2025-12-28 23:24:52 +01:00
Author
Owner
Copy Link

@avier99 commented on GitHub (Nov 28, 2025):

@avier99 try removing the comma thousand-separator from your value using a transformation:

amount:
    target: amount
    source: Amount
    transformations:
      - type: replace
        pattern: ","
        replacement: ""

Let me know if it works.

i guess progress however :

2025-11-27 19:17:25] WARNING: Error processing row 1: 'NoneType' object has no attribute 'replace'
[2025-11-27 19:17:25] WARNING: Error processing row 2: 'NoneType' object has no attribute 'replace'
[2025-11-27 19:17:25] WARNING: Error processing row 3: 'NoneType' object has no attribute 'replace'
[2025-11-27 19:17:25] INFO: Import completed successfully. Successful: 0, Failed: 3, Skipped: 0

also why does it check 3 rows and not 4?

@avier99 commented on GitHub (Nov 28, 2025): > [@avier99](https://github.com/avier99) try removing the comma thousand-separator from your value using a transformation: > > ``` > amount: > target: amount > source: Amount > transformations: > - type: replace > pattern: "," > replacement: "" > ``` > > Let me know if it works. i guess progress however : ``` 2025-11-27 19:17:25] WARNING: Error processing row 1: 'NoneType' object has no attribute 'replace' [2025-11-27 19:17:25] WARNING: Error processing row 2: 'NoneType' object has no attribute 'replace' [2025-11-27 19:17:25] WARNING: Error processing row 3: 'NoneType' object has no attribute 'replace' [2025-11-27 19:17:25] INFO: Import completed successfully. Successful: 0, Failed: 3, Skipped: 0 ``` also why does it check 3 rows and not 4?
adam commented 2025-12-28 23:24:53 +01:00
Author
Owner
Copy Link

@eitchtee commented on GitHub (Nov 28, 2025):

Ohh, I didn't catch this earlier, remove the skip_lines: 1 configuration from settings. This is meant to be used when your csv has something at the top that isn't a header, like:

IMPORTED FROM BANK X AT 27-11-2025
Date,Entity,Amount,Description,Account
8/22/2023,Dad,"300,000.00",Fees,DBS Bank
8/22/2023,Mom,"400,000.00",Fees,DBS Bank
8/7/2023,GROWW Return,929.24,,DBS Bank
7/28/2023,Tax,10.00,,DBS Bank

You were skipping the header and the engine was getting confused, trying to use 8/22/2023,Dad,"300,000.00",Fees,DBS Bank as the column names (source), which is why it stops after 3 rows.

Also, you'll need to change the format on the date and reference_date, use "%m/%d/%Y" instead of what's there. Click here for available format options.

@eitchtee commented on GitHub (Nov 28, 2025): Ohh, I didn't catch this earlier, remove the `skip_lines: 1` configuration from `settings`. This is meant to be used when your csv has something at the top that isn't a header, like: ``` IMPORTED FROM BANK X AT 27-11-2025 Date,Entity,Amount,Description,Account 8/22/2023,Dad,"300,000.00",Fees,DBS Bank 8/22/2023,Mom,"400,000.00",Fees,DBS Bank 8/7/2023,GROWW Return,929.24,,DBS Bank 7/28/2023,Tax,10.00,,DBS Bank ``` You were skipping the header and the engine was getting confused, trying to use `8/22/2023,Dad,"300,000.00",Fees,DBS Bank` as the column names (source), which is why it stops after 3 rows. --- Also, you'll need to change the format on the date and reference_date, use `"%m/%d/%Y"` instead of what's there. [Click here for available format options](https://docs.python.org/3.11/library/datetime.html#strftime-and-strptime-behavior).
adam commented 2025-12-28 23:24:53 +01:00
Author
Owner
Copy Link

@avier99 commented on GitHub (Nov 28, 2025):

ohh it did work
thanks a ton

@avier99 commented on GitHub (Nov 28, 2025): ohh it did work thanks a ton
adam commented 2025-12-28 23:24:54 +01:00
Author
Owner
Copy Link

@avier99 commented on GitHub (Nov 28, 2025):

also on another note
for the yaml on type and is_paid

 type:
    target: type
    detection_method: sign

  is_paid:
    target: is_paid
    detection_method: always_paid

i tried detection method with sign and even using always_income ; it defaults goes to expense for some reason!!

@avier99 commented on GitHub (Nov 28, 2025): also on another note for the yaml on `type` and `is_paid` ```yaml type: target: type detection_method: sign is_paid: target: is_paid detection_method: always_paid ``` i tried detection method with sign and even using always_income ; it defaults goes to expense for some reason!!
adam commented 2025-12-28 23:24:54 +01:00
Author
Owner
Copy Link

@eitchtee commented on GitHub (Nov 28, 2025):

For "sign" to work you need to set a source (usually the same as amount), if the value starts with "-" it will be considered an expense, otherwise Income.

Now, for always_income and always_expense, it shouldn't require a source, but it seems like it does require one to work. If you could create a new issue about this, please.

In short, add a source to type, even if it is always_*. This also extends to is_paid.

type:
   target: type
   detection_method: sign
   source: Amount
@eitchtee commented on GitHub (Nov 28, 2025): For "sign" to work you need to set a source (usually the same as amount), if the value starts with "-" it will be considered an expense, otherwise Income. Now, for always_income and always_expense, it shouldn't require a source, but it seems like it does require one to work. If you could create a new issue about this, please. In short, add a source to type, even if it is always_*. This also extends to is_paid. ``` type: target: type detection_method: sign source: Amount ```
adam commented 2025-12-28 23:24:54 +01:00
Author
Owner
Copy Link

@avier99 commented on GitHub (Nov 28, 2025):

Another stupid scenario :

how would i achieve importing bulk transfers?

i have 2 specific types : one with same currency; and one with different currencies.

i am guessing the diff currencies can we mixed with exchange rates?

another question was related to Investments - do you have a specific way for handling investment accounts, maybe multiple sub portfolios within, but and sell of assets in the portfolio, etc?

@avier99 commented on GitHub (Nov 28, 2025): Another stupid scenario : how would i achieve importing bulk transfers? i have 2 specific types : one with same currency; and one with different currencies. i am guessing the diff currencies can we mixed with exchange rates? another question was related to Investments - do you have a specific way for handling investment accounts, maybe multiple sub portfolios within, but and sell of assets in the portfolio, etc?
adam commented 2025-12-28 23:24:54 +01:00
Author
Owner
Copy Link

@eitchtee commented on GitHub (Nov 28, 2025):

@avier99 I've moved this discussion to #414 to avoid further hijacking this issue, let's keep talking there.

@eitchtee commented on GitHub (Nov 28, 2025): @avier99 I've moved this discussion to #414 to avoid further hijacking this issue, let's keep talking there.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
documentation
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
 question
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/WYGIWYH#53
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?