mirror of
https://github.com/eitchtee/WYGIWYH.git
synced 2026-01-14 21:23:29 +01:00
[BUG] - database "wygiwyh" has a collation version mismatch #79
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @LevitateDilationEggbeater on GitHub (Dec 9, 2025).
DETAIL: The database was created using collation version 2.36, but the operating system provides version 2.41.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE wygiwyh REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
Currently rebuilding the database as per the suggestion. Not sure why the crash happened.
@eitchtee commented on GitHub (Dec 9, 2025):
Good question, the problem lies within Postgres' Docker images.
Read more about it here:
https://www.reddit.com/r/PostgreSQL/comments/1p0tsgu/dockers_official_postgres_image_is_shipping/
https://github.com/docker-library/postgres/issues/1356#issuecomment-3189418446
I should probably pin the image of the example docker-compose as suggested in the github issue.
@LevitateDilationEggbeater commented on GitHub (Dec 9, 2025):
Thanks! I'll look into that.
@LevitateDilationEggbeater commented on GitHub (Dec 9, 2025):
I'll reopen the ticket. Running postgres for bookworm and running
docker exec -it wygiwyh_pg psql -d <wygiwyh_DB> -U <wygiwyh_USER> -c "ALTER DATABASE wygiwyh REFRESH COLLATION VERSION;"
did not help.
wygiwyh_pg | 2025-12-09T18:58:02.618104141Z
wygiwyh_pg | 2025-12-09T18:58:02.618128464Z PostgreSQL Database directory appears to contain a database; Skipping initialization
wygiwyh_pg | 2025-12-09T18:58:02.618132308Z
wygiwyh_pg | 2025-12-09T18:58:02.638305458Z 2025-12-09 18:58:02.638 UTC [1] LOG: starting PostgreSQL 15.15 (Debian 15.15-1.pgdg13+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
wygiwyh_pg | 2025-12-09T18:58:02.638643246Z 2025-12-09 18:58:02.638 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
wygiwyh_pg | 2025-12-09T18:58:02.638650869Z 2025-12-09 18:58:02.638 UTC [1] LOG: listening on IPv6 address "::", port 5432
wygiwyh_pg | 2025-12-09T18:58:02.640036778Z 2025-12-09 18:58:02.639 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
wygiwyh_pg | 2025-12-09T18:58:02.643378448Z 2025-12-09 18:58:02.643 UTC [29] LOG: database system was shut down at 2025-12-09 18:58:01 UTC
wygiwyh_pg | 2025-12-09T18:58:02.646966256Z 2025-12-09 18:58:02.646 UTC [1] LOG: database system is ready to accept connections
wygiwyh | 2025-12-09T18:58:20.974008610Z [2025-12-09 18:58:20] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T18:58:20.978490782Z [2025-12-09 18:58:20] - INFO - root - Creating transaction for date: 2025-12-09
@eitchtee commented on GitHub (Dec 9, 2025):
The logs look clean, are you getting the same warning from Django as you did the first time?
@LevitateDilationEggbeater commented on GitHub (Dec 9, 2025):
I should elaborate a little more. Part "Creating transaction for date: 2025-12-09" repeats until it makes my server as a whole crash. I am not sure why it does that.
I have not added any transactions for today and I cannot delete the recurring transactions from the CLI.
Also the Django warning persists.
`
wygiwyh | 2025-12-09T19:20:26.863619715Z 2025-12-09 19:20:26,863 CRIT Server 'unix_http_server' running without any HTTP authentication checking
wygiwyh | 2025-12-09T19:20:26.863784044Z 2025-12-09 19:20:26,863 INFO supervisord started with pid 1
wygiwyh | 2025-12-09T19:20:27.868315643Z 2025-12-09 19:20:27,867 INFO spawned: 'procrastinate_01' with pid 7
wygiwyh | 2025-12-09T19:20:27.871074383Z 2025-12-09 19:20:27,870 INFO spawned: 'web' with pid 8
wygiwyh_pg | 2025-12-09T19:20:26.623003689Z
wygiwyh_pg | 2025-12-09T19:20:26.623032429Z PostgreSQL Database directory appears to contain a database; Skipping initialization
wygiwyh_pg | 2025-12-09T19:20:26.623037254Z
wygiwyh_pg | 2025-12-09T19:20:26.644879705Z 2025-12-09 19:20:26.644 UTC [1] LOG: starting PostgreSQL 15.15 (Debian 15.15-1.pgdg13+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
wygiwyh_pg | 2025-12-09T19:20:26.645150795Z 2025-12-09 19:20:26.645 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
wygiwyh_pg | 2025-12-09T19:20:26.645160126Z 2025-12-09 19:20:26.645 UTC [1] LOG: listening on IPv6 address "::", port 5432
wygiwyh_pg | 2025-12-09T19:20:26.646501760Z 2025-12-09 19:20:26.646 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
wygiwyh_pg | 2025-12-09T19:20:26.650355685Z 2025-12-09 19:20:26.649 UTC [29] LOG: database system was shut down at 2025-12-09 19:15:20 UTC
wygiwyh_pg | 2025-12-09T19:20:26.652866311Z 2025-12-09 19:20:26.652 UTC [1] LOG: database system is ready to accept connections
wygiwyh | 2025-12-09T19:20:27.873947326Z Procastinate is waiting for web app to start...
wygiwyh | 2025-12-09T19:20:28.875215300Z 2025-12-09 19:20:28,875 INFO success: procrastinate_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
wygiwyh | 2025-12-09T19:20:28.875233506Z 2025-12-09 19:20:28,875 INFO success: web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
wygiwyh | 2025-12-09T19:20:29.197496686Z System check identified some issues:
wygiwyh | 2025-12-09T19:20:29.197513799Z
wygiwyh | 2025-12-09T19:20:29.197516600Z WARNINGS:
wygiwyh | 2025-12-09T19:20:29.197519165Z ?: (django_vite.W001) Cannot read Vite manifest file for app default at /usr/src/app/static_files/manifest.json : [Errno 2] No such file or directory: '/usr/src/app/static_files/manifest.json'
wygiwyh | 2025-12-09T19:20:29.197521561Z HINT: Make sure you have generated a manifest file, and that DJANGO_VITE["default"]["manifest_path"] points to the correct location.
wygiwyh_pg | 2025-12-09T19:20:26.623003689Z
wygiwyh_pg | 2025-12-09T19:20:26.623032429Z PostgreSQL Database directory appears to contain a database; Skipping initialization
wygiwyh_pg | 2025-12-09T19:20:26.623037254Z
wygiwyh_pg | 2025-12-09T19:20:26.644879705Z 2025-12-09 19:20:26.644 UTC [1] LOG: starting PostgreSQL 15.15 (Debian 15.15-1.pgdg13+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
wygiwyh_pg | 2025-12-09T19:20:26.645150795Z 2025-12-09 19:20:26.645 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
wygiwyh_pg | 2025-12-09T19:20:26.645160126Z 2025-12-09 19:20:26.645 UTC [1] LOG: listening on IPv6 address "::", port 5432
wygiwyh_pg | 2025-12-09T19:20:26.646501760Z 2025-12-09 19:20:26.646 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
wygiwyh_pg | 2025-12-09T19:20:26.650355685Z 2025-12-09 19:20:26.649 UTC [29] LOG: database system was shut down at 2025-12-09 19:15:20 UTC
wygiwyh_pg | 2025-12-09T19:20:26.652866311Z 2025-12-09 19:20:26.652 UTC [1] LOG: database system is ready to accept connections
wygiwyh | 2025-12-09T19:20:53.133477378Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.137866736Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.142231130Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.146575358Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.150864201Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.155238575Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.159620776Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.164231104Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.168652218Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.173060973Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.177462533Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.181778326Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.186193026Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.190622304Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.195219886Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.199861711Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.204693931Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.209549175Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.214513364Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.219291763Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.223999147Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.228924639Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.233299049Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.237706153Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.242344115Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.247162619Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.251896278Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.256720987Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.261740301Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
wygiwyh | 2025-12-09T19:20:53.267013870Z [2025-12-09 19:20:53] - INFO - root - Creating transaction for date: 2025-12-09
`
@LevitateDilationEggbeater commented on GitHub (Dec 9, 2025):
Yes, I have 19 recurring transactions across two users. I would be more than happy to try to delete those.
I have daily backups of the database, so I am not too worried about breaking anything important.
@eitchtee commented on GitHub (Dec 9, 2025):
Ok, that's odd.
The
Creating transaction for date:log comes from recurring transactions, but it shouldn't get stuck like this. Maybe you have a lot of stuck jobs that are triggering now that you fixed the database?First do this:
docker exec -it wygiwyh_server python manage.py procrastinate shelllist_jobs task=generate_recurring_transactionsand press entercancel <the job id after the # char>for each of these jobsNow, if this doesn't work, you might want to remove the recurring transactions you have one by one to see what's causing the issue
docker exec -it wygiwyh_server python manage.py shellfrom apps.transactions.models import RecurringTransactionand press enterRecurringTransaction.all_objects.all()and press enter. This will give you a list of all recurring transactions in your instance.RecurringTransaction.all_objects.filter(description="<the name that's displayed after <RecurringTransaction: >").delete()or just nuke everything by runningRecurringTransaction.all_objects.all().delete()PS. I'd love if you could run
RecurringTransaction.all_objects.filter(description="<the name that's displayed after <RecurringTransaction: >").first().__dict__and send me the output so I can debug this (if deleting any recurring transaction actually fixes the problem).@eitchtee commented on GitHub (Dec 9, 2025):
A better way of getting the data from all recurring transactions would be to run:
Press enter after each line when typing on the shell
@LevitateDilationEggbeater commented on GitHub (Dec 9, 2025):
Oddly enough, It's working again now. I lost November, but the rest of the database seems to be there. I tried to manually delete the recurring tasks in the WebUI but got a failure notification earlier. Maybe those transactions went through after all.
In any case, thanks for the support.
[2025-12-09 20:20:46] - INFO - procrastinate.worker.worker - Starting job check_for_transaction_rules[657](signal='transaction_created', user_id=1, instance_id=50181)
[2025-12-09 20:20:46] - INFO - apps.rules.tasks - Starting rule execution...
[2025-12-09 20:20:46] - INFO - apps.rules.tasks - Available functions: dict_keys(['relativedelta', 'str', 'int', 'float', 'abs', 'randint', 'random', 'decimal', 'datetime', 'date', 'transactions'])
[2025-12-09 20:20:46] - INFO - apps.rules.tasks - Testing 0 rule(s)...
[2025-12-09 20:20:46] - INFO - procrastinate.worker - Job check_for_transaction_rules[657](signal='transaction_created', user_id=1, instance_id=50181) ended with status: Success, lasted 0.031 s - Result: (None, [])
[2025-12-09 20:20:46] - INFO - procrastinate.worker.worker - Starting job check_for_transaction_rules[658](signal='transaction_created', user_id=1, instance_id=50182)
[2025-12-09 20:20:46] - ERROR - apps.rules.tasks - ** Error while executing 'check_for_transaction_rules' task
Traceback (most recent call last):
File "/usr/src/app/apps/rules/tasks.py", line 558, in check_for_transaction_rules
instance = Transaction.objects.get(id=instance_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 635, in get
raise self.model.DoesNotExist(
apps.transactions.models.Transaction.DoesNotExist: Transaction matching query does not exist.
[2025-12-09 20:20:46] - ERROR - procrastinate.worker - Job check_for_transaction_rules[658](signal='transaction_created', user_id=1, instance_id=50182) ended with status: Error, lasted 0.004 s
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/procrastinate/worker.py", line 266, in _process_job
job_result.result = await ensure_async()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/procrastinate/worker.py", line 252, in ensure_async
task_result = await await_func(*job_args, **job.task_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/procrastinate/utils.py", line 109, in sync_to_async
return await sync.sync_to_async(func, thread_sensitive=False)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 504, in call
ret = await asyncio.shield(exec_coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 559, in thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/app/apps/rules/tasks.py", line 767, in check_for_transaction_rules
raise e
File "/usr/src/app/apps/rules/tasks.py", line 558, in check_for_transaction_rules
instance = Transaction.objects.get(id=instance_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 635, in get
raise self.model.DoesNotExist(
apps.transactions.models.Transaction.DoesNotExist: Transaction matching query does not exist.
[2025-12-09 20:20:46] - INFO - procrastinate.worker.worker - Starting job check_for_transaction_rules659
[2025-12-09 20:20:46] - ERROR - apps.rules.tasks - ** Error while executing 'check_for_transaction_rules' task
Traceback (most recent call last):
File "/usr/src/app/apps/rules/tasks.py", line 555, in check_for_transaction_rules
instance = Transaction.deleted_objects.get(id=instance_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 635, in get
raise self.model.DoesNotExist(
apps.transactions.models.Transaction.DoesNotExist: Transaction matching query does not exist.
[2025-12-09 20:20:46] - ERROR - procrastinate.worker - Job check_for_transaction_rules659 ended with status: Error, lasted 0.003 s
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/procrastinate/worker.py", line 266, in _process_job
job_result.result = await ensure_async()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/procrastinate/worker.py", line 252, in ensure_async
task_result = await await_func(*job_args, **job.task_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/procrastinate/utils.py", line 109, in sync_to_async
return await sync.sync_to_async(func, thread_sensitive=False)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 504, in call
ret = await asyncio.shield(exec_coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 559, in thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/app/apps/rules/tasks.py", line 767, in check_for_transaction_rules
raise e
File "/usr/src/app/apps/rules/tasks.py", line 555, in check_for_transaction_rules
instance = Transaction.deleted_objects.get(id=instance_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 635, in get
raise self.model.DoesNotExist(
apps.transactions.models.Transaction.DoesNotExist: Transaction matching query does not exist.
@eitchtee commented on GitHub (Dec 10, 2025):
Glad you got it working. My best guess would be a bunch of tasks piling up and trying to execute together, but I'm not 100% sure about this. I will try to add a lock to the task execution so only one can be ran at a time.
@max-wittig commented on GitHub (Dec 14, 2025):
Same happens here for a fresh install
@eitchtee commented on GitHub (Dec 14, 2025):
Hey @max-wittig, what happens on a fresh install?
@max-wittig commented on GitHub (Dec 15, 2025):
Hi! Thanks for the fast response! This error comes up.
@eitchtee commented on GitHub (Dec 15, 2025):
@max-wittig do you have
DEBUG=trueon your .env file? It is recommended to run on production with it set to false unless you're debugging some issue. But this is only a warning, it should work fine even if this message shows up.Let me know if you face any issues. It can take a while for the application to boot, thus the
Procastinate is waiting for web app to start...message, after a while everything should start up.@max-wittig commented on GitHub (Dec 15, 2025):
So the warning is not the reason for the crash? Debug should be off
Thanks for your help!
@eitchtee commented on GitHub (Dec 15, 2025):
@max-wittig the warning is fine, I also get it on my instance, I believe it's because the manifest.json is loaded after the the container starts.
But I'm failing to identify the crash on the logs you shared, SIGTERM indicates you or something on your system is asking for the container to be stopped. Could you elaborate a bit more on what problem you're facing? Also, can you share your .env, docker-compose.yml and the full logs (
docker compose logs)?@max-wittig commented on GitHub (Dec 15, 2025):
@eitchtee I took your
docker-compose.ymland make it to work in Kubernetes. Probably I made a mistake:It just seems to be stuck at
Procastinate is waiting for web app to start....@eitchtee commented on GitHub (Dec 15, 2025):
@max-wittig I'm afraid I can't be of much help as I have 0 experience with k8s. But I can give you some general tips on how things work so you have a better chance at debugging this:
/tmp/migrations_completeto exist, outputting "Procastinate is waiting for web app to start..." every 2 seconds so you know it's alive/tmp/migrations_completein case it exists from another run, then run two commands from Django,collectstaticandmigrate, the first just move the static files we need to their proper places, it can take a while on the first run, but you should see a252 static files copied to '/usr/src/app/static_files', 978 post-processed.message on the log files indicating it has ran successfully. Andmigrateruns the database migrations against the configure database. After this the lock file is created, and everything should startup.From your logs, you're not getting the message for the first command, so maybe the problem lies there. Maybe you're not giving it enough time to complete before shutting down the container, or maybe there's some permission issues. We use a
app:appuser/group on the Dockerfile.Another thing you could try is ditching supervisord, and have a container for the app and a container for the background worker, this is how we used to do things before I merged them into a single container for simplifying the setup, while this is not actively tested, it should still work.
See this older version of the docker compose for a glimpse of how it works:
3440d4405e/docker-compose.prod.ymlPay attention to the commands used (/start and /start-procrastinate instead of /start-single) and the shared tmp folder between the two containers.
@max-wittig commented on GitHub (Dec 16, 2025):
@eitchtee Thanks for the detailed response! Turns out, I just forgot some variables. Sorry!
@LevitateDilationEggbeater commented on GitHub (Dec 17, 2025):
@eitchtee, I ran into some issues again and this time 'round logged the debug output. The exchange rate seems to be causing the headaches here:
wygiwyh | 2025-12-17T08:26:18.728115974Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates959
wygiwyh | 2025-12-17T08:26:18.731083258Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates959 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.743679251Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates960
wygiwyh | 2025-12-17T08:26:18.746588355Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates960 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.759043698Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates961
wygiwyh | 2025-12-17T08:26:18.762009252Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates961 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.775103879Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates963
wygiwyh | 2025-12-17T08:26:18.777992091Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates963 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.791060497Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates964
wygiwyh | 2025-12-17T08:26:18.793989561Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates964 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.806917008Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates965
wygiwyh | 2025-12-17T08:26:18.809802002Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates965 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.822745885Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates966
wygiwyh | 2025-12-17T08:26:18.825654537Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates966 ended with status: Success, lasted 0.003 s
wygiwyh | 2025-12-17T08:26:18.838568040Z [2025-12-17 08:26:18] - INFO - procrastinate.worker.worker - Starting job reset_demo_data967
wygiwyh | 2025-12-17T08:26:18.839559999Z [2025-12-17 08:26:18] - INFO - procrastinate.worker - Job reset_demo_data967 ended with status: Success, lasted 0.001 s
@eitchtee commented on GitHub (Dec 20, 2025):
@LevitateDilationEggbeater did it also crash the server this time? If it didn't, it's ok I guess.
There's something funky going on tough, did this happen randomly or after some unexpected or expected shutdown (of the server or the containers)? For example,
reset_demo_datais meant to run at 8 am exactly, andautomatic_fetch_exchange_ratesevery hour on the hour, but both are starting at08:26, which indicates procrastinate is unable to start the jobs at the correct time for some reason.But from your logs, it seems like the lock I added is working as each job was started once the other was finished.
@LevitateDilationEggbeater commented on GitHub (Dec 20, 2025):
@eitchtee, it did indeed crash the server, or rather it sent so many requests that I could process about one transaction manually every 30 minutes.
I noticed the trouble starting two or three weeks again, but didn't pay it enough mind to really help you out with any useful details.
I'm a bit busy with the Christmas prep right now, but I want to set up a clean instance, see if the issue persists and if all possible migrate my database from an older backup - although I'll probably have to reindex it again which I'm not looking forward to.
@eitchtee commented on GitHub (Dec 20, 2025):
@LevitateDilationEggbeater Jeez, that's really bad. Are you using docker-compose? Could you share you compose file and env variables? (Censor the private stuff).
Also, if you go to: <YOUR URL/IP>/admin/procrastinate/procrastinatejob/959/ , <YOUR URL/IP>/admin/procrastinate/procrastinatejob/960/ and so on, can you share some screenshots of what you see on these pages?
@LevitateDilationEggbeater commented on GitHub (Dec 21, 2025):
@eitchtee, I don't think I'll manage today, but I'll give it a go tomorrow. I shutdown WYGIWYH for the time being as it was putting a fair bit of load onto my server which I don't want crashing over christmas.
@LevitateDilationEggbeater commented on GitHub (Dec 21, 2025):
I restarted the server and it's running, but very slow to respond. The logs look clean to me, but are attached anyway. I'll try my luck setting up a clean instance and then migrating the database later. I'll let you know what comes of it.
cat .env
wygiwyh | 2025-12-21T10:36:58.526792579Z [2025-12-21 10:36:58] - INFO - procrastinate.periodic - Registering task cleanup_deleted_transactions with periodic id '' to run periodically with cron 10 1 * * *
wygiwyh | 2025-12-21T10:36:58.526800149Z [2025-12-21 10:36:58] - INFO - procrastinate.periodic - Registering task automatic_fetch_exchange_rates with periodic id '' to run periodically with cron 0 * * * *
wygiwyh | 2025-12-21T10:36:58.526815060Z [2025-12-21 10:36:58] - INFO - procrastinate.periodic - Registering task remove_old_jobs with periodic id '' to run periodically with cron 0 4 * * *
wygiwyh | 2025-12-21T10:36:58.526847655Z [2025-12-21 10:36:58] - INFO - procrastinate.periodic - Registering task remove_expired_sessions with periodic id '' to run periodically with cron 0 6 1 * *
wygiwyh | 2025-12-21T10:36:58.526852122Z [2025-12-21 10:36:58] - INFO - procrastinate.periodic - Registering task reset_demo_data with periodic id '' to run periodically with cron 0 8 * * *
wygiwyh | 2025-12-21T10:36:58.526873922Z [2025-12-21 10:36:58] - INFO - procrastinate.periodic - Registering task check_for_updates with periodic id '' to run periodically with cron 0 */12 * * *
wygiwyh | 2025-12-21T10:36:58.661233395Z Starting user setup...
wygiwyh | 2025-12-21T10:36:58.661252733Z ADMIN_EMAIL or ADMIN_PASSWORD environment variables not set. Skipping superuser creation.
wygiwyh | 2025-12-21T10:36:58.661255378Z ---
wygiwyh | 2025-12-21T10:36:58.661257413Z DEMO setting is not True (or not set). Skipping demo user creation.
wygiwyh | 2025-12-21T10:36:58.661259485Z User setup command finished.
wygiwyh | 2025-12-21T10:36:58.877233040Z Launching a worker on all queues
wygiwyh | 2025-12-21T10:36:58.894346084Z [2025-12-21 10:36:58] - INFO - procrastinate.worker.worker - Starting worker on all queues
wygiwyh | 2025-12-21T10:36:59.253235158Z [2025-12-21 10:36:59 +0000] [8] [INFO] Starting gunicorn 23.0.0
wygiwyh | 2025-12-21T10:36:59.253540806Z [2025-12-21 10:36:59 +0000] [8] [INFO] Listening at: http://0.0.0.0:8000 (8)
wygiwyh | 2025-12-21T10:36:59.253547819Z [2025-12-21 10:36:59 +0000] [8] [INFO] Using worker: sync
wygiwyh | 2025-12-21T10:36:59.255535567Z [2025-12-21 10:36:59 +0000] [39] [INFO] Booting worker with pid: 39
wygiwyh | 2025-12-21T10:36:59.335463919Z [2025-12-21 10:36:59 +0000] [40] [INFO] Booting worker with pid: 40
wygiwyh | 2025-12-21T10:36:59.395234727Z [2025-12-21 10:36:59 +0000] [41] [INFO] Booting worker with pid: 41
wygiwyh | 2025-12-21T10:36:59.405088628Z [2025-12-21 10:36:59 +0000] [42] [INFO] Booting worker with pid: 42
wygiwyh | 2025-12-21T10:38:44.364925349Z [2025-12-21 10:38:44] - INFO - procrastinate.worker.worker - Starting job check_for_transaction_rules[999](signal='transaction_created', user_id=1, old_data=None, instance_id=98281508)
wygiwyh | 2025-12-21T10:39:13.208013041Z [2025-12-21 10:39:13] - INFO - apps.rules.tasks - Starting rule execution...
wygiwyh | 2025-12-21T10:39:13.208104569Z [2025-12-21 10:39:13] - INFO - apps.rules.tasks - Available functions: dict_keys(['relativedelta', 'str', 'int', 'float', 'abs', 'randint', 'random', 'decimal', 'datetime', 'date', 'transactions'])
wygiwyh | 2025-12-21T10:39:13.238456355Z [2025-12-21 10:39:13] - INFO - apps.rules.tasks - Testing 0 rule(s)...
wygiwyh | 2025-12-21T10:39:13.238834999Z [2025-12-21 10:39:13] - INFO - procrastinate.worker - Job check_for_transaction_rules[999](signal='transaction_created', user_id=1, old_data=None, instance_id=98281508) ended with status: Success, lasted 28.874 s - Result: (None, [])
wygiwyh | 2025-12-21T11:00:00.557175380Z [2025-12-21 11:00:00] - INFO - procrastinate.periodic - Periodic job automatic_fetch_exchange_rates1000 deferred for timestamp 1766314800 with id 1000
wygiwyh | 2025-12-21T11:00:00.583326234Z [2025-12-21 11:00:00] - INFO - procrastinate.worker.worker - Starting job automatic_fetch_exchange_rates1000
wygiwyh | 2025-12-21T11:00:00.603384147Z [2025-12-21 11:00:00] - INFO - procrastinate.worker - Job automatic_fetch_exchange_rates1000 ended with status: Success, lasted 0.020 s
wygiwyh | 2025-12-21T11:00:31.012866396Z Not Found: /favicon.ico
wygiwyh | 2025-12-21T11:00:31.012889573Z [2025-12-21 11:00:31] - WARNING - django.request - Not Found: /favicon.ico