[DM-9726] validate_drp hsc dataset broken due to http request size limit - Jira

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: None
Labels:
None

Story Points:
3.833
Epic Link:
sqre-s17-squash-refresh-2
Team:
SQuaRE

Description

The validate_drp hsc dataset is failing on post-qa with this error message:

413 Client Error: Request Entity Too Large for url: https://squash.lsst.codes/dashboard/api/jobs/

Attachments

Issue Links

is triggering

DM-9741 Implement a minimal viable REST API in Flask with a test client

Done

relates to

DM-9778 jenkins qadb-dump job broken by new large hsc measurement results

Done

Activity

Ascending order - Click to sort in descending order

6 older comments

Hide

Permalink

Joshua Hoblitt added a comment - 08/Mar/17 9:43 AM

The logging fix appears to be working. This is the error that was generated.

Mar 07 22:23:00 jenkins-squash django[12866]: Internal Server Error: /dashboard/api/jobs/

                                              Traceback (most recent call last):

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute

                                                  return self.cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute

                                                  return self.cursor.execute(query, args)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute

                                                  self.errorhandler(self, exc, value)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler

                                                  raise errorvalue

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute

                                                  res = self._query(query)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query

                                                  rowcount = self._do_query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query

                                                  db.query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query

                                                  _mysql.connection.query(self, query)

                                              _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

                                              The above exception was the direct cause of the following exception:

                                              Traceback (most recent call last):

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 132, in get_response

                                                  response = wrapped_callback(request, *callback_args, **callback_kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

                                                  return view_func(*args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py", line 85, in view

                                                  return self.dispatch(request, *args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 456, in dispatch

                                                  response = self.handle_exception(exc)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 453, in dispatch

                                                  response = handler(request, *args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 20, in create

                                                  self.perform_create(serializer)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 25, in perform_create

                                                  serializer.save()

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py", line 169, in save

                                                  self.instance = self.create(validated_data)

                                                File "./dashboard/serializers.py", line 152, in create

                                                  job = Job.objects.create(**data)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method

                                                  return getattr(self.get_queryset(), name)(*args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 348, in create

                                                  obj.save(force_insert=True, using=self.db)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 734, in save

                                                  force_update=force_update, update_fields=update_fields)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 762, in save_base

                                                  updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 846, in _save_table

                                                  result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 885, in _do_insert

                                                  using=using, raw=raw)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method

                                                  return getattr(self.get_queryset(), name)(*args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 920, in _insert

                                                  return query.get_compiler(using=using).execute_sql(return_id)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql

                                                  cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute

                                                  return self.cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py", line 97, in __exit__

                                                  six.reraise(dj_exc_type, dj_exc_value, traceback)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py", line 658, in reraise

                                                  raise value.with_traceback(tb)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute

                                                  return self.cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute

                                                  return self.cursor.execute(query, args)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute

                                                  self.errorhandler(self, exc, value)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler

                                                  raise errorvalue

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute

                                                  res = self._query(query)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query

                                                  rowcount = self._do_query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query

                                                  db.query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query

                                                  _mysql.connection.query(self, query)

                                              django.db.utils.OperationalError: (2006, 'MySQL server has gone away')

Show

Joshua Hoblitt added a comment - 08/Mar/17 9:43 AM The logging fix appears to be working. This is the error that was generated. Mar 07 22 : 23 : 00 jenkins-squash django[ 12866 ]: Internal Server Error: /dashboard/api/jobs/ Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) _mysql_exceptions.OperationalError: ( 2006 , 'MySQL server has gone away' ) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py" , line 132 , in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py" , line 58 , in wrapped_view return view_func(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py" , line 85 , in view return self.dispatch(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 456 , in dispatch response = self.handle_exception(exc) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 453 , in dispatch response = handler(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 20 , in create self.perform_create(serializer) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 25 , in perform_create serializer.save() File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py" , line 169 , in save self.instance = self.create(validated_data) File "./dashboard/serializers.py" , line 152 , in create job = Job.objects.create(**data) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 348 , in create obj.save(force_insert=True, using=self.db) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 734 , in save force_update=force_update, update_fields=update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 762 , in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 846 , in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 885 , in _do_insert using=using, raw=raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 920 , in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py" , line 974 , in execute_sql cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py" , line 97 , in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py" , line 658 , in reraise raise value.with_traceback(tb) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) django.db.utils.OperationalError: ( 2006 , 'MySQL server has gone away' )

Hide

Permalink

Joshua Hoblitt added a comment - 08/Mar/17 11:24 AM

I think the exception error string is misleading. I believe we are hitting the max_allowed_packet limit.

MariaDB [qadb]> select @@max_allowed_packet;

+----------------------+

| @@max_allowed_packet |

+----------------------+

|              4194304 |

+----------------------+

1 row in set (0.00 sec)

MariaDB [qadb]>

Show

Joshua Hoblitt added a comment - 08/Mar/17 11:24 AM I think the exception error string is misleading. I believe we are hitting the max_allowed_packet limit. MariaDB [qadb]> select @ @max_allowed_packet ; +----------------------+ | @ @max_allowed_packet | +----------------------+ | 4194304 | +----------------------+ 1 row in set ( 0.00 sec) MariaDB [qadb]>

Hide

Permalink

Joshua Hoblitt added a comment - 08/Mar/17 12:02 PM

A new rds parameter group that sets max_allowed_packet has been added to the tf config and applied to production.

MariaDB [qadb]> select @@max_allowed_packet;

+----------------------+

| @@max_allowed_packet |

+----------------------+

|           1073741824 |

+----------------------+

1 row in set (0.00 sec)

MariaDB [qadb]>

Show

Joshua Hoblitt added a comment - 08/Mar/17 12:02 PM A new rds parameter group that sets max_allowed_packet has been added to the tf config and applied to production. MariaDB [qadb]> select @ @max_allowed_packet ; +----------------------+ | @ @max_allowed_packet | +----------------------+ | 1073741824 | +----------------------+ 1 row in set ( 0.00 sec) MariaDB [qadb]>

Hide

Permalink

Joshua Hoblitt added a comment - 09/Mar/17 10:36 AM

The hsc dataset failed to POST again with a new DB error:

Mar 09 03:51:20 jenkins-squash django[15365]: Internal Server Error: /dashboard/api/jobs/

                                              Traceback (most recent call last):

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute

                                                  return self.cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute

                                                  return self.cursor.execute(query, args)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute

                                                  self.errorhandler(self, exc, value)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler

                                                  raise errorvalue

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute

                                                  res = self._query(query)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query

                                                  rowcount = self._do_query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query

                                                  db.query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query

                                                  _mysql.connection.query(self, query)

                                              _mysql_exceptions.OperationalError: (1118, 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.')

                                              The above exception was the direct cause of the following exception:

                                              Traceback (most recent call last):

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 132, in get_response

                                                  response = wrapped_callback(request, *callback_args, **callback_kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view

                                                  return view_func(*args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py", line 85, in view

                                                  return self.dispatch(request, *args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 456, in dispatch

                                                  response = self.handle_exception(exc)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py", line 453, in dispatch

                                                  response = handler(request, *args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 20, in create

                                                  self.perform_create(serializer)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py", line 25, in perform_create

                                                  serializer.save()

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py", line 169, in save

                                                  self.instance = self.create(validated_data)

                                                File "./dashboard/serializers.py", line 154, in create

                                                  Measurement.objects.create(job=job, **measurement)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method

                                                  return getattr(self.get_queryset(), name)(*args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 348, in create

                                                  obj.save(force_insert=True, using=self.db)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 734, in save

                                                  force_update=force_update, update_fields=update_fields)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 762, in save_base

                                                  updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 846, in _save_table

                                                  result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py", line 885, in _do_insert

                                                  using=using, raw=raw)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 127, in manager_method

                                                  return getattr(self.get_queryset(), name)(*args, **kwargs)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py", line 920, in _insert

                                                  return query.get_compiler(using=using).execute_sql(return_id)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql

                                                  cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute

                                                  return self.cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py", line 97, in __exit__

                                                  six.reraise(dj_exc_type, dj_exc_value, traceback)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py", line 658, in reraise

                                                  raise value.with_traceback(tb)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute

                                                  return self.cursor.execute(sql, params)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 124, in execute

                                                  return self.cursor.execute(query, args)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 226, in execute

                                                  self.errorhandler(self, exc, value)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler

                                                  raise errorvalue

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 217, in execute

                                                  res = self._query(query)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 378, in _query

                                                  rowcount = self._do_query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 341, in _do_query

                                                  db.query(q)

                                                File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 280, in query

                                                  _mysql.connection.query(self, query)

                                              django.db.utils.OperationalError: (1118, 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.')

Show

Joshua Hoblitt added a comment - 09/Mar/17 10:36 AM The hsc dataset failed to POST again with a new DB error: Mar 09 03 : 51 : 20 jenkins-squash django[ 15365 ]: Internal Server Error: /dashboard/api/jobs/ Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) _mysql_exceptions.OperationalError: ( 1118 , 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.' ) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/core/handlers/base.py" , line 132 , in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py" , line 58 , in wrapped_view return view_func(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/viewsets.py" , line 85 , in view return self.dispatch(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 456 , in dispatch response = self.handle_exception(exc) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/views.py" , line 453 , in dispatch response = handler(request, *args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 20 , in create self.perform_create(serializer) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/mixins.py" , line 25 , in perform_create serializer.save() File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/rest_framework/serializers.py" , line 169 , in save self.instance = self.create(validated_data) File "./dashboard/serializers.py" , line 154 , in create Measurement.objects.create(job=job, **measurement) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 348 , in create obj.save(force_insert=True, using=self.db) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 734 , in save force_update=force_update, update_fields=update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 762 , in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 846 , in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/base.py" , line 885 , in _do_insert using=using, raw=raw) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/manager.py" , line 127 , in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/query.py" , line 920 , in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py" , line 974 , in execute_sql cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/utils.py" , line 97 , in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/utils/six.py" , line 658 , in reraise raise value.with_traceback(tb) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/utils.py" , line 64 , in execute return self.cursor.execute(sql, params) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py" , line 124 , in execute return self.cursor.execute(query, args) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 226 , in execute self.errorhandler(self, exc, value) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 36 , in defaulterrorhandler raise errorvalue File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 217 , in execute res = self._query(query) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 378 , in _query rowcount = self._do_query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/cursors.py" , line 341 , in _do_query db.query(q) File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/MySQLdb/connections.py" , line 280 , in query _mysql.connection.query(self, query) django.db.utils.OperationalError: ( 1118 , 'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.' )

Hide

Permalink

Joshua Hoblitt added a comment - 09/Mar/17 11:49 AM

The production rds innodb log size has been increased to 256MiB* 10. Which is hopefully significantly larger than the transaction to insert the hsc measurements blob.

Show

Joshua Hoblitt added a comment - 09/Mar/17 11:49 AM The production rds innodb log size has been increased to 256MiB* 10. Which is hopefully significantly larger than the transaction to insert the hsc measurements blob.

Data Management

validate_drp hsc dataset broken due to http request size limit

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins