Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9726

validate_drp hsc dataset broken due to http request size limit

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The validate_drp hsc dataset is failing on post-qa with this error message:

      413 Client Error: Request Entity Too Large for url: https://squash.lsst.codes/dashboard/api/jobs/
      
      

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The logging fix appears to be working. This is the error that was generated.

            Mar 07 22:23:00 jenkins-squash django[12866]: Internal Server Error: /dashboard/api/jobs/
                                                          Traceback (most recent call last):
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
                                                              return self.cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute
                                                              return self.cursor.execute(query, args)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute
                                                              self.errorhandler(self, exc, value)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
                                                              raise errorvalue
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute
                                                              res = self._query(query)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query
                                                              rowcount = self._do_query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query
                                                              db.query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query
                                                              _mysql.connection.query(self, query)
                                                          _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')
                                                          
                                                          The above exception was the direct cause of the following exception:
                                                          
                                                          Traceback (most recent call last):
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 132, in get_response
                                                              response = wrapped_callback(request, *callback_args, **callback_kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
                                                              return view_func(*args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py", line 85, in view
                                                              return self.dispatch(request, *args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 456, in dispatch
                                                              response = self.handle_exception(exc)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 453, in dispatch
                                                              response = handler(request, *args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 20, in create
                                                              self.perform_create(serializer)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 25, in perform_create
                                                              serializer.save()
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py", line 169, in save
                                                              self.instance = self.create(validated_data)
                                                            File "./dashboard/serializers.py", line 152, in create
                                                              job = Job.objects.create(**data)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method
                                                              return getattr(self.get_queryset(), name)(*args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 348, in create
                                                              obj.save(force_insert=True, using=self.db)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 734, in save
                                                              force_update=force_update, update_fields=update_fields)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 762, in save_base
                                                              updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 846, in _save_table
                                                              result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 885, in _do_insert
                                                              using=using, raw=raw)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method
                                                              return getattr(self.get_queryset(), name)(*args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 920, in _insert
                                                              return query.get_compiler(using=using).execute_sql(return_id)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
                                                              cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
                                                              return self.cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py", line 97, in __exit__
                                                              six.reraise(dj_exc_type, dj_exc_value, traceback)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py", line 658, in reraise
                                                              raise value.with_traceback(tb)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
                                                              return self.cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute
                                                              return self.cursor.execute(query, args)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute
                                                              self.errorhandler(self, exc, value)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
                                                              raise errorvalue
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute
                                                              res = self._query(query)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query
                                                              rowcount = self._do_query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query
                                                              db.query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query
                                                              _mysql.connection.query(self, query)
                                                          django.db.utils.OperationalError: (2006, 'MySQL server has gone away')
            

            Show
            jhoblitt Joshua Hoblitt added a comment - The logging fix appears to be working. This is the error that was generated. Mar 07 22 : 23 : 00 jenkins-squash django[ 12866 ]: Internal Server Error: /dashboard/api/jobs/ Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) _mysql_exceptions.OperationalError: ( 2006 , 'MySQL server has gone away' ) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py" , line 132 , in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py" , line 58 , in wrapped_view return view_func(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py" , line 85 , in view return self.dispatch(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 456 , in dispatch response = self.handle_exception(exc) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 453 , in dispatch response = handler(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 20 , in create self.perform_create(serializer) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 25 , in perform_create serializer.save() File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py" , line 169 , in save self.instance = self.create(validated_data) File "./dashboard/serializers.py" , line 152 , in create job = Job.objects.create(**data) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 348 , in create obj.save(force_insert=True, using=self.db) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 734 , in save force_update=force_update, update_fields=update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 762 , in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 846 , in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 885 , in _do_insert using=using, raw=raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 920 , in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py" , line 974 , in execute_sql cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py" , line 97 , in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py" , line 658 , in reraise raise value.with_traceback(tb) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) django.db.utils.OperationalError: ( 2006 , 'MySQL server has gone away' )
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I think the exception error string is misleading. I believe we are hitting the max_allowed_packet limit.

            MariaDB [qadb]> select @@max_allowed_packet;
            +----------------------+
            | @@max_allowed_packet |
            +----------------------+
            |              4194304 |
            +----------------------+
            1 row in set (0.00 sec)
             
            MariaDB [qadb]> 
            

            Show
            jhoblitt Joshua Hoblitt added a comment - I think the exception error string is misleading. I believe we are hitting the max_allowed_packet limit. MariaDB [qadb]> select @ @max_allowed_packet ; +----------------------+ | @ @max_allowed_packet | +----------------------+ | 4194304 | +----------------------+ 1 row in set ( 0.00 sec)   MariaDB [qadb]>
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            A new rds parameter group that sets max_allowed_packet has been added to the tf config and applied to production.

            MariaDB [qadb]> select @@max_allowed_packet;
            +----------------------+
            | @@max_allowed_packet |
            +----------------------+
            |           1073741824 |
            +----------------------+
            1 row in set (0.00 sec)
             
            MariaDB [qadb]> 
            

            Show
            jhoblitt Joshua Hoblitt added a comment - A new rds parameter group that sets max_allowed_packet has been added to the tf config and applied to production. MariaDB [qadb]> select @ @max_allowed_packet ; +----------------------+ | @ @max_allowed_packet | +----------------------+ | 1073741824 | +----------------------+ 1 row in set ( 0.00 sec)   MariaDB [qadb]>
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The hsc dataset failed to POST again with a new DB error:

            Mar 09 03:51:20 jenkins-squash django[15365]: Internal Server Error: /dashboard/api/jobs/
                                                          Traceback (most recent call last):
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
                                                              return self.cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute
                                                              return self.cursor.execute(query, args)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute
                                                              self.errorhandler(self, exc, value)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
                                                              raise errorvalue
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute
                                                              res = self._query(query)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query
                                                              rowcount = self._do_query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query
                                                              db.query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query
                                                              _mysql.connection.query(self, query)
                                                          _mysql_exceptions.OperationalError: (1118, 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.')
                                                          
                                                          The above exception was the direct cause of the following exception:
                                                          
                                                          Traceback (most recent call last):
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 132, in get_response
                                                              response = wrapped_callback(request, *callback_args, **callback_kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
                                                              return view_func(*args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py", line 85, in view
                                                              return self.dispatch(request, *args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 456, in dispatch
                                                              response = self.handle_exception(exc)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 453, in dispatch
                                                              response = handler(request, *args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 20, in create
                                                              self.perform_create(serializer)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 25, in perform_create
                                                              serializer.save()
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py", line 169, in save
                                                              self.instance = self.create(validated_data)
                                                            File "./dashboard/serializers.py", line 154, in create
                                                              Measurement.objects.create(job=job, **measurement)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method
                                                              return getattr(self.get_queryset(), name)(*args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 348, in create
                                                              obj.save(force_insert=True, using=self.db)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 734, in save
                                                              force_update=force_update, update_fields=update_fields)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 762, in save_base
                                                              updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 846, in _save_table
                                                              result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 885, in _do_insert
                                                              using=using, raw=raw)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method
                                                              return getattr(self.get_queryset(), name)(*args, **kwargs)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 920, in _insert
                                                              return query.get_compiler(using=using).execute_sql(return_id)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
                                                              cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
                                                              return self.cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py", line 97, in __exit__
                                                              six.reraise(dj_exc_type, dj_exc_value, traceback)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py", line 658, in reraise
                                                              raise value.with_traceback(tb)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
                                                              return self.cursor.execute(sql, params)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute
                                                              return self.cursor.execute(query, args)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute
                                                              self.errorhandler(self, exc, value)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
                                                              raise errorvalue
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute
                                                              res = self._query(query)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query
                                                              rowcount = self._do_query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query
                                                              db.query(q)
                                                            File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query
                                                              _mysql.connection.query(self, query)
                                                          django.db.utils.OperationalError: (1118, 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.')
            

            Show
            jhoblitt Joshua Hoblitt added a comment - The hsc dataset failed to POST again with a new DB error: Mar 09 03 : 51 : 20 jenkins-squash django[ 15365 ]: Internal Server Error: /dashboard/api/jobs/ Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) _mysql_exceptions.OperationalError: ( 1118 , 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.' ) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py" , line 132 , in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py" , line 58 , in wrapped_view return view_func(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py" , line 85 , in view return self.dispatch(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 456 , in dispatch response = self.handle_exception(exc) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 453 , in dispatch response = handler(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 20 , in create self.perform_create(serializer) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 25 , in perform_create serializer.save() File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py" , line 169 , in save self.instance = self.create(validated_data) File "./dashboard/serializers.py" , line 154 , in create Measurement.objects.create(job=job, **measurement) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 348 , in create obj.save(force_insert=True, using=self.db) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 734 , in save force_update=force_update, update_fields=update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 762 , in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 846 , in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 885 , in _do_insert using=using, raw=raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 920 , in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py" , line 974 , in execute_sql cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py" , line 97 , in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py" , line 658 , in reraise raise value.with_traceback(tb) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) django.db.utils.OperationalError: ( 1118 , 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.' )
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The production rds innodb log size has been increased to 256MiB* 10. Which is hopefully significantly larger than the transaction to insert the hsc measurements blob.

            Show
            jhoblitt Joshua Hoblitt added a comment - The production rds innodb log size has been increased to 256MiB* 10. Which is hopefully significantly larger than the transaction to insert the hsc measurements blob.

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Angelo Fausti, Jonathan Sick, Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.