Can't import csv with two positive rows #19

New Issue

Closed

opened 2025-12-28 23:24:09 +01:00 by adam · 6 comments

adam commented 2025-12-28 23:24:09 +01:00

Owner

Copy Link

Copy Source

Originally created by @eitchtee on GitHub (Jan 31, 2025).

Yes, I like it pretty much, just still figuring out the data for importing

  type:
    source: amount
    target: type
    detection_method: sign

  type:
    source: "Valor"
    target: "type"
    detection_method: sign

what are these for ?

I am in the middle of setup for my data import, just a bit confused about the part for own transfer , it would be something like transfer from acc A to acc B, and I have separate credit with debit table, both positive balance, so merge format not seems work

settings:
  file_type: csv
  delimiter: ","
  encoding: utf-8
  skip_lines: 0
  importing: transactions
  trigger_transaction_rules: true
  skip_errors: true

mapping:
  account:
    target: account
    default: "MYR Wallet"
    type: name

  date:
    target: date
    source: Date
    format: "%d/%m/%Y"

  amount:
    target: amount
    source: Valor

  description:
    target: description
    source: Description

  type:
    source: "Valor"
    target: "type"
    detection_method: sign

  notes:
    target: notes
    source: Notes

  is_paid:
    target: is_paid
    detection_method: always_paid

deduplicate:
  - type: compare
    fields:
      - internal_id
    match_type: strict

Originally posted by @lucius100 in #41

Originally created by @eitchtee on GitHub (Jan 31, 2025). > Yes, I like it pretty much, just still figuring out the data for importing > > ``` > type: > source: amount > target: type > detection_method: sign > > type: > source: "Valor" > target: "type" > detection_method: sign > ``` > > what are these for ? > > <img width="503" alt="Image" src="https://github.com/user-attachments/assets/faaeb476-20fc-49b8-af76-fc8eabf42a32" /> > > I am in the middle of setup for my data import, just a bit confused about the part for own transfer , it would be something like transfer from acc A to acc B, and I have separate credit with debit table, both positive balance, so merge format not seems work > > ``` > settings: > file_type: csv > delimiter: "," > encoding: utf-8 > skip_lines: 0 > importing: transactions > trigger_transaction_rules: true > skip_errors: true > > mapping: > account: > target: account > default: "MYR Wallet" > type: name > > date: > target: date > source: Date > format: "%d/%m/%Y" > > amount: > target: amount > source: Valor > > description: > target: description > source: Description > > type: > source: "Valor" > target: "type" > detection_method: sign > > notes: > target: notes > source: Notes > > is_paid: > target: is_paid > detection_method: always_paid > > deduplicate: > - type: compare > fields: > - internal_id > match_type: strict > ``` _Originally posted by @lucius100 in [#41](https://github.com/eitchtee/WYGIWYH/issues/41#issuecomment-2628057599)_

adam added the bug label 2025-12-28 23:24:09 +01:00

adam closed this issue 2025-12-28 23:24:09 +01:00

adam commented 2025-12-28 23:24:09 +01:00

Author

Owner

Copy Link

Copy Source

@eitchtee commented on GitHub (Jan 31, 2025):

@lucius100 I've opened this issue to track this as it's looking like you won't be able to import this with the available options. Let me have a look and get back to you.

@eitchtee commented on GitHub (Jan 31, 2025): @lucius100 I've opened this issue to track this as it's looking like you won't be able to import this with the available options. Let me have a look and get back to you.

adam commented 2025-12-28 23:24:10 +01:00

Author

Owner

Copy Link

Copy Source

@eitchtee commented on GitHub (Jan 31, 2025):

@lucius100 please upgrade to v0.8.4

You might want to have two different accounts one for Dompet and another one for TNG, so the Own Transfers make sense

Then you can use the following template:

settings:
  file_type: csv
  delimiter: ","
  encoding: utf-8
  skip_lines: 0
  importing: transactions
  trigger_transaction_rules: true
  skip_errors: true

mapping:
  account:
    target: account
    source: "Wallet"
    type: name

  date:
    target: date
    source: Date
    format: "%d/%m/%Y"

  amount:
    target: amount
    source: ["Credits", "Debits"]
    transformations:
        - type: "replace"
          pattern: "MYR "
          replacement: ""
        - type: "replace"
          pattern: ","
          replacement: ""
          
  description:
    target: description
    source: Description
    
  category:
    target: category
    source: Category

  type:
    source: Credits
    target: "type"
    required: false
    detection_method: always_income
   
  type2:
    source: Debits
    target: "type"
    required: false
    detection_method: always_expense

  is_paid:
    target: is_paid
    detection_method: always_paid
    
  internal_id:
    target: internal_id
    transformations:
        - type: hash
          fields: ["Date", "Description", "Category", "Wallet"]

deduplication:
  - type: compare
    fields:
      - internal_id
    match_type: strict

If you don't want to have different accounts, you can use your original account definition, just make sure MYR Wallet is a created account on the app.

account:
    target: account
    default: "MYR Wallet"
    type: name

---

Explanation:

We use two different type mappings, one for each table column, the first to have a valid value is used. If Credits is empty, Debits is used and always validates to an expense; if Credits has a value, it is used and always validates to an Income.

Also, since your file doesn't have some sort of unique identifier we create one by hashing Date, Description, Category and Wallet together, this should be enough to not duplicate entries if you re-import a file with the same values.

And finally, we use some replaces to remove "MYR " and commas from your values.

---

Let me know how this goes. If you have any more questions feel free to ask them.

@eitchtee commented on GitHub (Jan 31, 2025): @lucius100 please upgrade to v0.8.4 You might want to have two different accounts one for `Dompet` and another one for `TNG`, so the `Own Transfers` make sense Then you can use the following template: ```yaml settings: file_type: csv delimiter: "," encoding: utf-8 skip_lines: 0 importing: transactions trigger_transaction_rules: true skip_errors: true mapping: account: target: account source: "Wallet" type: name date: target: date source: Date format: "%d/%m/%Y" amount: target: amount source: ["Credits", "Debits"] transformations: - type: "replace" pattern: "MYR " replacement: "" - type: "replace" pattern: "," replacement: "" description: target: description source: Description category: target: category source: Category type: source: Credits target: "type" required: false detection_method: always_income type2: source: Debits target: "type" required: false detection_method: always_expense is_paid: target: is_paid detection_method: always_paid internal_id: target: internal_id transformations: - type: hash fields: ["Date", "Description", "Category", "Wallet"] deduplication: - type: compare fields: - internal_id match_type: strict ``` If you don't want to have different accounts, you can use your original account definition, just make sure `MYR Wallet` is a created account on the app. ```yaml account: target: account default: "MYR Wallet" type: name ``` --- Explanation: We use two different `type` mappings, one for each table column, the first to have a valid value is used. If Credits is empty, Debits is used and always validates to an expense; if Credits has a value, it is used and always validates to an Income. Also, since your file doesn't have some sort of unique identifier we create one by hashing Date, Description, Category and Wallet together, this should be enough to not duplicate entries if you re-import a file with the same values. And finally, we use some replaces to remove "MYR " and commas from your values. --- Let me know how this goes. If you have any more questions feel free to ask them.

adam commented 2025-12-28 23:24:10 +01:00

Author

Owner

Copy Link

Copy Source

@lucius100 commented on GitHub (Jan 31, 2025):

I make some changes to the data, so I have few errors while importing, have error on replace MYR and comma, so I removed them.

[2025-01-31 20:27:02] WARNING: Error processing row 1: 'NoneType' object has no attribute 'replace'
[2025-01-31 20:27:02] WARNING: Error processing row 2: 'NoneType' object has no attribute 'replace'
[2025-01-31 20:27:02] WARNING: Error processing row 3: 'NoneType' object has no attribute 'replace'

now have error on amount, can't figured out

[2025-01-31 20:28:18] WARNING: Error processing row 1: Required field amount is missing
[2025-01-31 20:28:18] WARNING: Error processing row 2: Required field amount is missing

here is the latest yaml code.

settings:
  file_type: csv
  delimiter: ","
  encoding: utf-8
  skip_lines: 0
  importing: transactions
  trigger_transaction_rules: true
  skip_errors: true

mapping:
  account:
    target: account
    source: "Wallet"
    type: name

  date:
    target: date
    source: Date
    format: "%d/%m/%Y"

  amount:
    target: amount
    source: ["Credits", "Debits"]
          
  description:
    target: description
    source: Description
    
  category:
    target: category
    source: Category

  type:
    source: Credits
    target: "type"
    required: false
    detection_method: always_income
   
  type2:
    source: Debits
    target: "type"
    required: false
    detection_method: always_expense

  is_paid:
    target: is_paid
    detection_method: always_paid
    
  internal_id:
    target: internal_id
    transformations:
        - type: hash
          fields: ["Date", "Description", "Category", "Wallet"]

deduplication:
  - type: compare
    fields:
      - internal_id
    match_type: strict

@lucius100 commented on GitHub (Jan 31, 2025): <img width="449" alt="Image" src="https://github.com/user-attachments/assets/d7dac14a-2afb-48f0-9a6b-33c3c874b42b" /> I make some changes to the data, so I have few errors while importing, have error on replace MYR and comma, so I removed them. ``` [2025-01-31 20:27:02] WARNING: Error processing row 1: 'NoneType' object has no attribute 'replace' [2025-01-31 20:27:02] WARNING: Error processing row 2: 'NoneType' object has no attribute 'replace' [2025-01-31 20:27:02] WARNING: Error processing row 3: 'NoneType' object has no attribute 'replace' ``` now have error on amount, can't figured out ``` [2025-01-31 20:28:18] WARNING: Error processing row 1: Required field amount is missing [2025-01-31 20:28:18] WARNING: Error processing row 2: Required field amount is missing ``` here is the latest yaml code. ``` settings: file_type: csv delimiter: "," encoding: utf-8 skip_lines: 0 importing: transactions trigger_transaction_rules: true skip_errors: true mapping: account: target: account source: "Wallet" type: name date: target: date source: Date format: "%d/%m/%Y" amount: target: amount source: ["Credits", "Debits"] description: target: description source: Description category: target: category source: Category type: source: Credits target: "type" required: false detection_method: always_income type2: source: Debits target: "type" required: false detection_method: always_expense is_paid: target: is_paid detection_method: always_paid internal_id: target: internal_id transformations: - type: hash fields: ["Date", "Description", "Category", "Wallet"] deduplication: - type: compare fields: - internal_id match_type: strict ```

adam commented 2025-12-28 23:24:10 +01:00

Author

Owner

Copy Link

Copy Source

@eitchtee commented on GitHub (Jan 31, 2025):

[2025-01-31 20:27:02] WARNING: Error processing row 1: 'NoneType' object has no attribute 'replace'

This error is probably because you had some empty rows, in the first screenshot you had Initial balance without a Credit or Debits value. It's ok to error in this case.

[2025-01-31 20:28:18] WARNING: Error processing row 1: Required field amount is missing

This is probably due to the same thing, a row with empty values for Credits and Debits.

---

I tried importing the data you shared just now, with your configuration and got no errors.

If you really want to import everything, you can add default: "0" to your amount mapping:

amount:
    target: amount
    source: ["Credits", "Debits"]
    default: "0"

Any transaction without a value will be imported with the 0 as the amount, just for reference sake. This way you won't even need to remove the commas and "MYR "

@eitchtee commented on GitHub (Jan 31, 2025): > [2025-01-31 20:27:02] WARNING: Error processing row 1: 'NoneType' object has no attribute 'replace' This error is probably because you had some empty rows, in the first screenshot you had `Initial balance` without a Credit or Debits value. It's ok to error in this case. > [2025-01-31 20:28:18] WARNING: Error processing row 1: Required field amount is missing This is probably due to the same thing, a row with empty values for Credits and Debits. --- I tried importing the data you shared just now, with your configuration and got no errors. If you really want to import everything, you can add `default: "0"` to your amount mapping: ``` amount: target: amount source: ["Credits", "Debits"] default: "0" ``` Any transaction without a value will be imported with the 0 as the amount, just for reference sake. This way you won't even need to remove the commas and "MYR "

adam commented 2025-12-28 23:24:10 +01:00

Author

Owner

Copy Link

Copy Source

@lucius100 commented on GitHub (Feb 1, 2025):

Edit: Finally found the issue, I just copy the header and found out instead of "Credits" "Debits" , it was " Credits " , " Debits "
After fixing the space, now it can import smoothly, haha

@lucius100 commented on GitHub (Feb 1, 2025): Edit: Finally found the issue, I just copy the header and found out instead of "Credits" "Debits" , it was " Credits " , " Debits " After fixing the space, now it can import smoothly, haha

adam commented 2025-12-28 23:24:11 +01:00

Author

Owner

Copy Link

Copy Source

@eitchtee commented on GitHub (Feb 1, 2025):

Oh man haha. It's always the last thing you expect.

Good it worked. Let me know if you have any other issue.

@eitchtee commented on GitHub (Feb 1, 2025): Oh man haha. It's always the last thing you expect. Good it worked. Let me know if you have any other issue.

adam referenced this issue 2025-12-28 23:26:15 +01:00

[PR #19] [MERGED] locale(pt-BR): update translation #111

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/npm_and_yarn/frontend/browserslist-4.28.8

dependabot/uv/djangorestframework-3.17.2

weblate

feat/filter-presets

dev

copilot/fix-code-scanning-alerts

feat/cuelume

feat/tom-select-improvements

chore/bump-deps

feat/attatchments

fix-procrastinate

feat/replace-webpack+bootstrap

internal_port

add-mcp-server-mention

feat/networth-chart

feat/better-rules

feature/untrack-account

feat/setup-screen

feat/sidebar

feat/auto-refresh

fix/amount-input-doesnt-update-on-delete

feature/add-django-tests

tests

0.23.0

0.22.1

0.22.0

0.21.0

0.20.2

0.20.1

0.20.0

0.19.1

0.19.0

0.18.3

0.18.2

0.18.1

0.18.0

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.2

0.16.1

0.16.0

0.15.1

0.15.0

0.14.1

0.14.0

0.13.1

0.13.0

0.12.11

0.12.10

0.12.9

0.12.8

0.12.7

0.12.6

0.12.5

0.12.4

0.12.3

0.12.2

0.12.1

0.12.0

0.11.8

0.11.7

0.11.6

0.11.5

0.11.4

0.11.3

0.11.2

0.11.1

0.11.0

0.10.1

0.10.0

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.0

0.5.0

0.4.0

0.3.6

0.3.5

0.3.4

0.3.3

0.3.2

0.3.1

0.3.0

0.2.0

0.1.9

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

0.1.0

Labels

Clear labels

bug

bug

documentation

enhancement

help wanted

pull-request

Mirrored from GitHub Pull Request

question

No labels bug

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

adam (Adam Melkus)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/WYGIWYH#19