starred/WYGIWYH
1
0
Fork 0
mirror of https://github.com/eitchtee/WYGIWYH.git synced 2026-07-31 08:58:49 +02:00
Code Issues 14 Packages Projects Releases 10 Wiki Activity

Unable to import: Required field date is missing / Importing UTF-8-BOM #25

New Issue
Closed
opened 2025-12-28 23:24:16 +01:00 by adam · 3 comments
adam commented 2025-12-28 23:24:16 +01:00
Owner
Copy Link
Copy Source

Originally created by @eitchtee on GitHub (Feb 6, 2025).

This is my full import yaml now:

settings:
  file_type: csv
  delimiter: ","
  encoding: utf-8
  skip_lines: 0
  importing: transactions
  trigger_transaction_rules: true
  skip_errors: false

mapping:
  account:
    target: account
    source: "Currency"
    type: name
    transformations:
        - type: "replace"
          pattern: "EUR"
          replacement: "AM"
          exclusive: true
  amount:
    target: amount
    source: "Amount"
    required: true
  description:
    target: description
    source: "TransactionDescription"
  internal_id:
    target: internal_id
    transformations:
        - type: hash
          fields: ["Timestamp", "TransactionDescription", "Amount"]  
  date:
    target: date
    source: "Timestamp"
    format: "%d/%m/%Y"
  is_paid:
    target: is_paid
    detection_method: always_paid

deduplication:
  - type: compare
    fields:
      - internal_id
    match_type: strict

I still get this error:

2025-02-06 21:20:20 2025-02-06 20:20:20,790 INFO    procrastinate.worker Starting job apps.import_app.tasks.process_import[7](file_path='/usr/src/app/temp/card_transactions_record_02052025_221347_TTLEnjQ.csv', import_run_id=4)
2025-02-06 21:20:20 2025-02-06 20:20:20,853 ERROR   procrastinate.worker Job apps.import_app.tasks.process_import[7](file_path='/usr/src/app/temp/card_transactions_record_02052025_221347_TTLEnjQ.csv', import_run_id=4) ended with status: Error, lasted 0.062 s
2025-02-06 21:20:20 Traceback (most recent call last):
2025-02-06 21:20:20   File "/usr/src/app/apps/import_app/services/v1.py", line 615, in process_file
2025-02-06 21:20:20     self._process_csv(file_path)
2025-02-06 21:20:20   File "/usr/src/app/apps/import_app/services/v1.py", line 590, in _process_csv
2025-02-06 21:20:20     self._process_row(row, row_number)
2025-02-06 21:20:20   File "/usr/src/app/apps/import_app/services/v1.py", line 529, in _process_row
2025-02-06 21:20:20     mapped_data = self._map_row(row)
2025-02-06 21:20:20                   ^^^^^^^^^^^^^^^^^^
2025-02-06 21:20:20   File "/usr/src/app/apps/import_app/services/v1.py", line 513, in _map_row
2025-02-06 21:20:20     raise ValueError(f"Required field {field} is missing")
2025-02-06 21:20:20 ValueError: Required field date is missing
2025-02-06 21:20:20 
2025-02-06 21:20:20 During handling of the above exception, another exception occurred:
2025-02-06 21:20:20 
2025-02-06 21:20:20 Traceback (most recent call last):
2025-02-06 21:20:20   File "/usr/local/lib/python3.11/site-packages/procrastinate/worker.py", line 281, in run_job
2025-02-06 21:20:20     task_result = await await_func(*job_args, **job.task_kwargs)
2025-02-06 21:20:20                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-06 21:20:20   File "/usr/local/lib/python3.11/site-packages/procrastinate/utils.py", line 108, in sync_to_async
2025-02-06 21:20:20     return await sync.sync_to_async(func, thread_sensitive=False)(*args, **kwargs)
2025-02-06 21:20:20            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-06 21:20:20   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 468, in __call__
2025-02-06 21:20:20     ret = await asyncio.shield(exec_coro)
2025-02-06 21:20:20           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-06 21:20:20   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
2025-02-06 21:20:20     result = self.fn(*self.args, **self.kwargs)
2025-02-06 21:20:20              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-06 21:20:20   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 522, in thread_handler
2025-02-06 21:20:20     return func(*args, **kwargs)
2025-02-06 21:20:20            ^^^^^^^^^^^^^^^^^^^^^
2025-02-06 21:20:20   File "/usr/src/app/apps/import_app/tasks.py", line 17, in process_import
2025-02-06 21:20:20     import_service.process_file(file_path)
2025-02-06 21:20:20   File "/usr/src/app/apps/import_app/services/v1.py", line 629, in process_file
2025-02-06 21:20:20     raise Exception("Import failed")
2025-02-06 21:20:20 Exception: Import failed

I even tried setting up a fresh instance with an empty database and just added a currency and an account. Then tried the import again. Same error.

Originally posted by @andremohrmann in #41

Originally created by @eitchtee on GitHub (Feb 6, 2025). > This is my full import yaml now: > ```yaml > settings: > file_type: csv > delimiter: "," > encoding: utf-8 > skip_lines: 0 > importing: transactions > trigger_transaction_rules: true > skip_errors: false > > mapping: > account: > target: account > source: "Currency" > type: name > transformations: > - type: "replace" > pattern: "EUR" > replacement: "AM" > exclusive: true > amount: > target: amount > source: "Amount" > required: true > description: > target: description > source: "TransactionDescription" > internal_id: > target: internal_id > transformations: > - type: hash > fields: ["Timestamp", "TransactionDescription", "Amount"] > date: > target: date > source: "Timestamp" > format: "%d/%m/%Y" > is_paid: > target: is_paid > detection_method: always_paid > > deduplication: > - type: compare > fields: > - internal_id > match_type: strict > ``` > > I still get this error: > ``` > 2025-02-06 21:20:20 2025-02-06 20:20:20,790 INFO procrastinate.worker Starting job apps.import_app.tasks.process_import[7](file_path='/usr/src/app/temp/card_transactions_record_02052025_221347_TTLEnjQ.csv', import_run_id=4) > 2025-02-06 21:20:20 2025-02-06 20:20:20,853 ERROR procrastinate.worker Job apps.import_app.tasks.process_import[7](file_path='/usr/src/app/temp/card_transactions_record_02052025_221347_TTLEnjQ.csv', import_run_id=4) ended with status: Error, lasted 0.062 s > 2025-02-06 21:20:20 Traceback (most recent call last): > 2025-02-06 21:20:20 File "/usr/src/app/apps/import_app/services/v1.py", line 615, in process_file > 2025-02-06 21:20:20 self._process_csv(file_path) > 2025-02-06 21:20:20 File "/usr/src/app/apps/import_app/services/v1.py", line 590, in _process_csv > 2025-02-06 21:20:20 self._process_row(row, row_number) > 2025-02-06 21:20:20 File "/usr/src/app/apps/import_app/services/v1.py", line 529, in _process_row > 2025-02-06 21:20:20 mapped_data = self._map_row(row) > 2025-02-06 21:20:20 ^^^^^^^^^^^^^^^^^^ > 2025-02-06 21:20:20 File "/usr/src/app/apps/import_app/services/v1.py", line 513, in _map_row > 2025-02-06 21:20:20 raise ValueError(f"Required field {field} is missing") > 2025-02-06 21:20:20 ValueError: Required field date is missing > 2025-02-06 21:20:20 > 2025-02-06 21:20:20 During handling of the above exception, another exception occurred: > 2025-02-06 21:20:20 > 2025-02-06 21:20:20 Traceback (most recent call last): > 2025-02-06 21:20:20 File "/usr/local/lib/python3.11/site-packages/procrastinate/worker.py", line 281, in run_job > 2025-02-06 21:20:20 task_result = await await_func(*job_args, **job.task_kwargs) > 2025-02-06 21:20:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 2025-02-06 21:20:20 File "/usr/local/lib/python3.11/site-packages/procrastinate/utils.py", line 108, in sync_to_async > 2025-02-06 21:20:20 return await sync.sync_to_async(func, thread_sensitive=False)(*args, **kwargs) > 2025-02-06 21:20:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 2025-02-06 21:20:20 File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 468, in __call__ > 2025-02-06 21:20:20 ret = await asyncio.shield(exec_coro) > 2025-02-06 21:20:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 2025-02-06 21:20:20 File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run > 2025-02-06 21:20:20 result = self.fn(*self.args, **self.kwargs) > 2025-02-06 21:20:20 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > 2025-02-06 21:20:20 File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 522, in thread_handler > 2025-02-06 21:20:20 return func(*args, **kwargs) > 2025-02-06 21:20:20 ^^^^^^^^^^^^^^^^^^^^^ > 2025-02-06 21:20:20 File "/usr/src/app/apps/import_app/tasks.py", line 17, in process_import > 2025-02-06 21:20:20 import_service.process_file(file_path) > 2025-02-06 21:20:20 File "/usr/src/app/apps/import_app/services/v1.py", line 629, in process_file > 2025-02-06 21:20:20 raise Exception("Import failed") > 2025-02-06 21:20:20 Exception: Import failed > ``` > > I even tried setting up a fresh instance with an empty database and just added a currency and an account. Then tried the import again. Same error. _Originally posted by @andremohrmann in [#41](https://github.com/eitchtee/WYGIWYH/issues/41#issuecomment-2640926087)_
adam closed this issue 2025-12-28 23:24:16 +01:00
adam commented 2025-12-28 23:24:17 +01:00
Author
Owner
Copy Link
Copy Source

@eitchtee commented on GitHub (Feb 6, 2025):

@andremohrmann try importing this file, and let me know if you get the same error

andremohrmann.csv

It has the same content you shared with me in the other issue. But in my instance it worked:

[2025-02-06 21:58:33] INFO: Starting import process
[2025-02-06 21:58:33] INFO: Starting import with 4 rows
[2025-02-06 21:58:33] INFO: Successfully processed row 1
[2025-02-06 21:58:33] INFO: Successfully processed row 2
[2025-02-06 21:58:33] INFO: Successfully processed row 3
[2025-02-06 21:58:33] INFO: Successfully processed row 4
[2025-02-06 21:58:33] INFO: Import completed successfully. Successful: 4, Failed: 0, Skipped: 0
[2025-02-06 21:58:33] INFO: Cleaning up temporary files

If this file doesn't work as well we have to start looking at other reasons this might be happening. If it works, this means your .csv file has something that isn't present on the example file, which should narrow things down. This could be some extra line(s) before the header, wrong encoding, an extra space in the heading, etc.

@eitchtee commented on GitHub (Feb 6, 2025): @andremohrmann try importing this file, and let me know if you get the same error [andremohrmann.csv](https://github.com/user-attachments/files/18698220/andremohrmann.csv) It has the same content you shared with me in the other issue. But in my instance it worked: ``` [2025-02-06 21:58:33] INFO: Starting import process [2025-02-06 21:58:33] INFO: Starting import with 4 rows [2025-02-06 21:58:33] INFO: Successfully processed row 1 [2025-02-06 21:58:33] INFO: Successfully processed row 2 [2025-02-06 21:58:33] INFO: Successfully processed row 3 [2025-02-06 21:58:33] INFO: Successfully processed row 4 [2025-02-06 21:58:33] INFO: Import completed successfully. Successful: 4, Failed: 0, Skipped: 0 [2025-02-06 21:58:33] INFO: Cleaning up temporary files ``` If this file doesn't work as well we have to start looking at other reasons this might be happening. If it works, this means your .csv file has something that isn't present on the example file, which should narrow things down. This could be some extra line(s) before the header, wrong encoding, an extra space in the heading, etc.
adam commented 2025-12-28 23:24:18 +01:00
Author
Owner
Copy Link
Copy Source

@andremohrmann commented on GitHub (Feb 7, 2025):

Then this seems to be an encoding error of the file I get from my bank. Edit: It's not UTF-8, it's UTF-8 with BOM...
I tried importing only the lines I shared with you and it worked.
Then I opened the original csv and just copied and pasted the full content into an empty csv file.
With that file, importing works flawlessly.

Probably nothing you can do in this case, I guess.
But if you want I can share the full csv I get from my bank with you.
Otherwise, I will just have to get used to the workaround and automate the "conversion" myself ;)
Edit: Setting encoding: utf-8-sig in the import yaml works.

Thanks for the help! Much appreciated.

@andremohrmann commented on GitHub (Feb 7, 2025): Then this seems to be an encoding error of the file I get from my bank. Edit: It's not UTF-8, it's UTF-8 with BOM... I tried importing only the lines I shared with you and it worked. Then I opened the original csv and just copied and pasted the full content into an empty csv file. With that file, importing works flawlessly. Probably nothing you can do in this case, I guess. But if you want I can share the full csv I get from my bank with you. Otherwise, I will just have to get used to the workaround and automate the "conversion" myself ;) Edit: Setting `encoding: utf-8-sig` in the import yaml works. Thanks for the help! Much appreciated.
adam commented 2025-12-28 23:24:18 +01:00
Author
Owner
Copy Link
Copy Source

@eitchtee commented on GitHub (Feb 7, 2025):

Then this seems to be an encoding error of the file I get from my bank. Edit: It's not UTF-8, it's UTF-8 with BOM...

One of those cases where it's close enough to not raise an error, but different enough to not work.

Edit: Setting encoding: utf-8-sig in the import yaml works.

I was going to tell you to try that, glad you figured it out.

@eitchtee commented on GitHub (Feb 7, 2025): > Then this seems to be an encoding error of the file I get from my bank. Edit: It's not UTF-8, it's UTF-8 with BOM... One of those cases where it's close enough to not raise an error, but different enough to not work. > Edit: Setting `encoding: utf-8-sig` in the import yaml works. I was going to tell you to try that, glad you figured it out.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
documentation
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
 question
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Melkus)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/WYGIWYH#25