Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX] fix memory constraints for postgres #778

Merged
merged 16 commits into from
Jul 12, 2024
Merged

Conversation

jdkent
Copy link
Member

@jdkent jdkent commented Jul 11, 2024

neurostore     | Traceback (most recent call last):
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
neurostore     |     self.dialect.do_execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
neurostore     |     cursor.execute(statement, parameters)
neurostore     | psycopg2.errors.DiskFull: could not resize shared memory segment "/PostgreSQL.2021357946" to 4194304 bytes: No space left on device
neurostore     | 
neurostore     | 
neurostore     | The above exception was the direct cause of the following exception:
neurostore     | 
neurostore     | Traceback (most recent call last):
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
neurostore     |     await app(scope, receive, sender)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/swagger_ui.py", line 220, in __call__
neurostore     |     await self.router(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
neurostore     |     await route.handle(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 485, in handle
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 806, in app
neurostore     |     await self.default(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/swagger_ui.py", line 233, in default_fn
neurostore     |     await self.app(original_scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 93, in __call__
neurostore     |     await self.simple_response(scope, receive, send, request_headers=headers)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 148, in simple_response
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/routing.py", line 153, in __call__
neurostore     |     await self.router(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
neurostore     |     await route.handle(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 485, in handle
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
neurostore     |     await route.handle(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/routing.py", line 47, in __call__
neurostore     |     await self.next_app(original_scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/security.py", line 100, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/request_validation.py", line 142, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/lifespan.py", line 26, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/context.py", line 25, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/apps/flask.py", line 149, in __call__
neurostore     |     return await self.asgi_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/a2wsgi/wsgi.py", line 165, in __call__
neurostore     |     return await responder(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/a2wsgi/wsgi.py", line 200, in __call__
neurostore     |     await self.loop.run_in_executor(
neurostore     |   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
neurostore     |     result = self.fn(*self.args, **self.kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/a2wsgi/wsgi.py", line 256, in wsgi
neurostore     |     iterable = self.app(environ, start_response)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2193, in wsgi_app
neurostore     |     response = self.handle_exception(e)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
neurostore     |     response = self.full_dispatch_request()
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
neurostore     |     rv = self.handle_user_exception(e)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
neurostore     |     rv = self.dispatch_request()
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
neurostore     |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/apps/flask.py", line 66, in __call__
neurostore     |     return self.fn(*args, **kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/main.py", line 134, in wrapper
neurostore     |     return decorated_function(request)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/response.py", line 171, in wrapper
neurostore     |     handler_response = function(*args, **kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/parameter.py", line 87, in wrapper
neurostore     |     return function(**kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/main.py", line 123, in wrapper
neurostore     |     return function(*args, **kwargs)
neurostore     |   File "/neurostore/neurostore/resources/base.py", line 678, in post
neurostore     |     self.update_base_studies(unique_ids.get("base-studies"))
neurostore     |   File "/neurostore/neurostore/resources/base.py", line 183, in update_base_studies
neurostore     |     affected_base_studies = db.session.execute(query).fetchall()
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/scoping.py", line 778, in execute
neurostore     |     return self._proxied.execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
neurostore     |     return self._execute_internal(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2236, in _execute_internal
neurostore     |     result: Result[Any] = compile_state_cls.orm_execute_statement(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
neurostore     |     result = conn.execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
neurostore     |     return meth(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
neurostore     |     return connection._execute_clauseelement(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
neurostore     |     ret = self._execute_context(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
neurostore     |     return self._exec_single_context(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
neurostore     |     self._handle_dbapi_exception(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2353, in _handle_dbapi_exception
neurostore     |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
neurostore     |     self.dialect.do_execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
neurostore     |     cursor.execute(statement, parameters)
neurostore     | sqlalchemy.exc.OperationalError: (psycopg2.errors.DiskFull) could not resize shared memory segment "/PostgreSQL.2021357946" to 4194304 bytes: No space left on device
neurostore     | 
neurostore     | [SQL: SELECT DISTINCT base_studies.id, base_studies.has_images, base_studies.has_coordinates, (SELECT coalesce(bool_and(points.analysis_id IS NOT NULL), %(coalesce_2)s) AS coalesce_1 
neurostore     | FROM points, analyses 
neurostore     | WHERE points.analysis_id = analyses.id) AS new_has_coordinates, (SELECT coalesce(bool_and(images.analysis_id IS NOT NULL), %(coalesce_4)s) AS coalesce_3 
neurostore     | FROM images, analyses 
neurostore     | WHERE images.analysis_id = analyses.id) AS new_has_images 
neurostore     | FROM base_studies JOIN studies ON studies.base_study_id = base_studies.id LEFT OUTER JOIN analyses ON analyses.study_id = studies.id 
neurostore     | WHERE base_studies.id IN (%(id_1_1)s)]
neurostore     | [parameters: {'coalesce_2': False, 'coalesce_4': False, 'id_1_1': 'FMfXCKVzsapg'}]
neurostore     | (Background on this error at: https://sqlalche.me/e/20/e3q8)

@jdkent jdkent merged commit a115f1e into master Jul 12, 2024
16 checks passed
@jdkent jdkent deleted the fix/postgres_memory branch July 12, 2024 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant