Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11956

Integration tests with squash-deployment

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Miscellaneous fixes in the squash deployment.

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment - - edited
          • GCE persistent volume forbidden during squash-db deployment:

          Creating a GKE persistent volume...
          Error from server (Forbidden): error when creating "kubernetes/gke_volume.yaml": persistentvolumes "mysql-volume-1" is forbidden: error querying GCE PD volume mysql-volume-1: disk is not found
          make[1]: *** [deployment] Error 1
          

          Implemented LOCAL_VOLUME option for now.

          • Increased External IP discovery time limit for the services, 20s seems to work fine in GCE
          • Fixed typo in squash-api discovering the service
          • Dropped minikube support (replace.sh) in all the services, aff
          • Fixed DB password creation
          • FIxed SSLError due to subdomain in the services names
          • Added services names to Django

            --allowed-hosts

            and to Bokeh

            --allow-websocket-origin

          • Created remove-context target
          • Redeployed all services in sqre project at GCE

          Pending:

          • Better labels for the deployment
          • Use squash user instead of root in squash-db
          • Monitor app should not break upon Error 500 in the SQUASH_API_URL
          Show
          afausti Angelo Fausti added a comment - - edited GCE persistent volume forbidden during squash-db deployment: Creating a GKE persistent volume... Error from server (Forbidden): error when creating "kubernetes/gke_volume.yaml": persistentvolumes "mysql-volume-1" is forbidden: error querying GCE PD volume mysql-volume-1: disk is not found make[1]: *** [deployment] Error 1 Implemented LOCAL_VOLUME option for now. Increased External IP discovery time limit for the services, 20s seems to work fine in GCE Fixed typo in squash-api discovering the service Dropped minikube support ( replace.sh ) in all the services, aff Fixed DB password creation FIxed SSLError due to subdomain in the services names Added services names to Django --allowed-hosts and to Bokeh --allow-websocket-origin Created remove-context target Redeployed all services in sqre project at GCE Pending: Better labels for the deployment Use squash user instead of root in squash-db Monitor app should not break upon Error 500 in the SQUASH_API_URL
          Hide
          afausti Angelo Fausti added a comment - - edited

          I am connecting with root user and setting the MYSQL_ROOT_PASSWORD.
          The password is obtained from the k8s secret.

          Using now MYSQL_DATABASE env variable to create qadb during the container initialization.

          See: https://github.com/docker-library/mariadb/blob/master/10.3/docker-entrypoint.sh

          root@squash-db-1066012318-3t0zz:/# env | grep MYSQL
          MYSQL_DATABASE=qadb
          MYSQL_PASSWORD=noadmin
          

          Show
          afausti Angelo Fausti added a comment - - edited I am connecting with root user and setting the MYSQL_ROOT_PASSWORD . The password is obtained from the k8s secret. Using now MYSQL_DATABASE env variable to create qadb during the container initialization. See: https://github.com/docker-library/mariadb/blob/master/10.3/docker-entrypoint.sh root@squash-db-1066012318-3t0zz:/# env | grep MYSQL MYSQL_DATABASE=qadb MYSQL_PASSWORD=noadmin
          Hide
          afausti Angelo Fausti added a comment - - edited

          The issue with the persistent volume on GCE was solved, basically the problem was my misunderstanding of what a Persitent Volume Claim really does, it automagically provision the volume for you but I had a k8s configuration to create the GCE PV which clearly was not necessary and was duplicating the same volume thus the error.

          Debugging a last standing issue in the deployment. I had intermittent failures connecting to the database pod and after narrowing down the problem I found:

          • A bad configmap used to add customized DB configuration. When applied it removed configuration files existing in the same directory where the mount point is created. That's fixed now by adding all the files required in the config map.
          • The squash-db service had bad labels and selector. It was selecting the db pod and the api pod using the same labels. So I guess the route configuration was screwed up sending traffic to both pods and causing this intermittent behaviour. Labels were reviewed and it is working fine now.
          Show
          afausti Angelo Fausti added a comment - - edited The issue with the persistent volume on GCE was solved, basically the problem was my misunderstanding of what a Persitent Volume Claim really does, it automagically provision the volume for you but I had a k8s configuration to create the GCE PV which clearly was not necessary and was duplicating the same volume thus the error. Debugging a last standing issue in the deployment. I had intermittent failures connecting to the database pod and after narrowing down the problem I found: A bad configmap used to add customized DB configuration. When applied it removed configuration files existing in the same directory where the mount point is created. That's fixed now by adding all the files required in the config map. The squash-db service had bad labels and selector. It was selecting the db pod and the api pod using the same labels. So I guess the route configuration was screwed up sending traffic to both pods and causing this intermittent behaviour. Labels were reviewed and it is working fine now.
          Hide
          afausti Angelo Fausti added a comment -
          • Fixed broken links between services due to new architecture
          • Implemented new method to pass url arguments to bokeh apps emded.server_document() released in bokeh 0.12.9
            https://bokeh.pydata.org/en/latest/docs/user_guide/embed.html#server-data
          • Implemented loading bar for bokeh apps
          • Added Save button to create a png from the plot (monitor app)
          • Other miscellaneous fixes
          Show
          afausti Angelo Fausti added a comment - Fixed broken links between services due to new architecture Implemented new method to pass url arguments to bokeh apps emded.server_document() released in bokeh 0.12.9 https://bokeh.pydata.org/en/latest/docs/user_guide/embed.html#server-data Implemented loading bar for bokeh apps Added Save button to create a png from the plot (monitor app) Other miscellaneous fixes
          Hide
          afausti Angelo Fausti added a comment -

          Deployed to my sandbox https://squash-dash-demo.lsst.codes/

          Show
          afausti Angelo Fausti added a comment - Deployed to my sandbox https://squash-dash-demo.lsst.codes/

            People

            Assignee:
            afausti Angelo Fausti
            Reporter:
            afausti Angelo Fausti
            Watchers:
            Angelo Fausti
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.