Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28467

Test APDB Cassandra prototype with 2-month time partitioning

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      6
    • Sprint:
      DB_S21_12
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      My last large test used 30-day partitioning for source tables and I had to run 13 queries to cover 12 month period. It could be that performance depends on the number of concurrent queries and/or number of read operations, I want to check if there is a difference in reducing number of partitions. I don't need to run whole 12 months of visits, 100k will probably be enough to understand that.

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment - - edited

            Ran fro 124k visits with 60-day partitions.

            Standard report:

            # nodetool status
            Datacenter: datacenter1
            =======================
            Status=Up/Down
            |/ State=Normal/Leaving/Joining/Moving
            --  Address       Load        Tokens  Owns (effective)  Host ID                               Rack
            UN  10.128.0.118  676.75 GiB  256     ?                 52d05f52-d01c-40a0-8560-75e8d3e5a20d  rack1
            UN  10.128.0.91   591.55 GiB  256     ?                 d824d71d-dd3b-4c18-92d1-ef4f6dc2a866  rack1
            UN  10.128.0.119  604.98 GiB  256     ?                 9487a7fe-b6db-4d69-91b2-2fa567197d45  rack1
            UN  10.128.0.103  605.88 GiB  256     ?                 767278b0-c12d-4def-98ef-c0e9e811fcbd  rack1
            UN  10.128.0.57   632.97 GiB  256     ?                 ed1f8fff-2e9f-4656-b9e8-565df3457931  rack1
            UN  10.128.0.123  596.53 GiB  256     ?                 5e9be0a6-fcc8-4a16-a9a7-4d35d77ca85c  rack1
            UN  10.128.0.101  624.45 GiB  256     ?                 44612049-898b-4116-b12b-da30474b4181  rack1
            UN  10.128.0.121  631.38 GiB  256     ?                 ffdfacc7-8025-4978-92bf-764d56e0dd20  rack1
            UN  10.128.0.122  615.54 GiB  256     ?                 7ed00b36-3a2b-4cb2-8fbe-1f12d1d009a6  rack1
            UN  10.128.0.100  618.22 GiB  256     ?                 ae4d4462-de5d-4726-b600-e23a16b05fde  rack1
            UN  10.128.0.52   640.33 GiB  256     ?                 7284d69d-57c1-4f27-8b61-f6fb6739aae9  rack1
            UN  10.128.0.37   595.68 GiB  256     ?                 e50cf9e5-3777-4dd2-91a0-a6cef931a8dd  rack1 

            Total load size 7434.26 GiB = 7.26 TiB.

            Data sizes on each node:

            $ shmux -m -s -c "df -h /data/apdb*" -- apdb-server-{1..12}
             apdb-server-1: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-1: /dev/nvme0n1p1  375G   77G  299G  21% /data/apdb1
             apdb-server-1: /dev/nvme0n2p1  375G   75G  300G  20% /data/apdb2
             apdb-server-1: /dev/nvme0n3p1  375G   75G  301G  20% /data/apdb3
             apdb-server-1: /dev/nvme0n4p1  375G   75G  301G  20% /data/apdb4
             apdb-server-1: /dev/nvme0n5p1  375G   76G  300G  21% /data/apdb5
             apdb-server-1: /dev/nvme0n6p1  375G   76G  300G  21% /data/apdb6
             apdb-server-1: /dev/nvme0n7p1  375G   75G  300G  20% /data/apdb7
             apdb-server-1: /dev/nvme0n8p1  375G   75G  300G  20% /data/apdb8
             apdb-server-6: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-6: /dev/nvme0n1p1  375G   84G  292G  23% /data/apdb1
             apdb-server-6: /dev/nvme0n2p1  375G   80G  296G  22% /data/apdb2
             apdb-server-6: /dev/nvme0n3p1  375G   80G  296G  22% /data/apdb3
             apdb-server-6: /dev/nvme0n4p1  375G   79G  297G  21% /data/apdb4
             apdb-server-6: /dev/nvme0n5p1  375G   78G  297G  21% /data/apdb5
             apdb-server-6: /dev/nvme0n6p1  375G   79G  297G  21% /data/apdb6
             apdb-server-6: /dev/nvme0n7p1  375G   78G  297G  21% /data/apdb7
             apdb-server-6: /dev/nvme0n8p1  375G   79G  297G  21% /data/apdb8
             apdb-server-4: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-4: /dev/nvme0n1p1  375G   82G  293G  22% /data/apdb1
             apdb-server-4: /dev/nvme0n2p1  375G   74G  302G  20% /data/apdb2
             apdb-server-4: /dev/nvme0n3p1  375G   74G  301G  20% /data/apdb3
             apdb-server-4: /dev/nvme0n4p1  375G   75G  301G  20% /data/apdb4
             apdb-server-4: /dev/nvme0n5p1  375G   75G  301G  20% /data/apdb5
             apdb-server-4: /dev/nvme0n6p1  375G   76G  300G  21% /data/apdb6
             apdb-server-4: /dev/nvme0n7p1  375G   75G  301G  20% /data/apdb7
             apdb-server-4: /dev/nvme0n8p1  375G   75G  301G  20% /data/apdb8
            apdb-server-11: Filesystem      Size  Used Avail Use% Mounted on
            apdb-server-11: /dev/nvme0n1p1  375G   84G  292G  23% /data/apdb1
            apdb-server-11: /dev/nvme0n2p1  375G   77G  299G  21% /data/apdb2
            apdb-server-11: /dev/nvme0n3p1  375G   77G  298G  21% /data/apdb3
            apdb-server-11: /dev/nvme0n4p1  375G   78G  297G  21% /data/apdb4
            apdb-server-11: /dev/nvme0n5p1  375G   78G  298G  21% /data/apdb5
            apdb-server-11: /dev/nvme0n6p1  375G   78G  298G  21% /data/apdb6
            apdb-server-11: /dev/nvme0n7p1  375G   78G  298G  21% /data/apdb7
            apdb-server-11: /dev/nvme0n8p1  375G   77G  298G  21% /data/apdb8
             apdb-server-3: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-3: /dev/nvme0n1p1  375G   84G  291G  23% /data/apdb1
             apdb-server-3: /dev/nvme0n2p1  375G   80G  296G  22% /data/apdb2
             apdb-server-3: /dev/nvme0n3p1  375G   80G  295G  22% /data/apdb3
             apdb-server-3: /dev/nvme0n4p1  375G   80G  296G  22% /data/apdb4
             apdb-server-3: /dev/nvme0n5p1  375G   80G  296G  22% /data/apdb5
             apdb-server-3: /dev/nvme0n6p1  375G   80G  296G  22% /data/apdb6
             apdb-server-3: /dev/nvme0n7p1  375G   80G  295G  22% /data/apdb7
             apdb-server-3: /dev/nvme0n8p1  375G   81G  295G  22% /data/apdb8
             apdb-server-2: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-2: /dev/nvme0n1p1  375G   83G  293G  22% /data/apdb1
             apdb-server-2: /dev/nvme0n2p1  375G   81G  295G  22% /data/apdb2
             apdb-server-2: /dev/nvme0n3p1  375G   81G  295G  22% /data/apdb3
             apdb-server-2: /dev/nvme0n4p1  375G   81G  295G  22% /data/apdb4
             apdb-server-2: /dev/nvme0n5p1  375G   81G  294G  22% /data/apdb5
             apdb-server-2: /dev/nvme0n6p1  375G   81G  295G  22% /data/apdb6
             apdb-server-2: /dev/nvme0n7p1  375G   81G  295G  22% /data/apdb7
             apdb-server-2: /dev/nvme0n8p1  375G   81G  295G  22% /data/apdb8
             apdb-server-7: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-7: /dev/nvme0n1p1  375G   91G  285G  25% /data/apdb1
             apdb-server-7: /dev/nvme0n2p1  375G   85G  291G  23% /data/apdb2
             apdb-server-7: /dev/nvme0n3p1  375G   85G  290G  23% /data/apdb3
             apdb-server-7: /dev/nvme0n4p1  375G   85G  290G  23% /data/apdb4
             apdb-server-7: /dev/nvme0n5p1  375G   85G  291G  23% /data/apdb5
             apdb-server-7: /dev/nvme0n6p1  375G   86G  290G  23% /data/apdb6
             apdb-server-7: /dev/nvme0n7p1  375G   85G  291G  23% /data/apdb7
             apdb-server-7: /dev/nvme0n8p1  375G   87G  289G  24% /data/apdb8
             apdb-server-5: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-5: /dev/nvme0n1p1  375G   84G  292G  23% /data/apdb1
             apdb-server-5: /dev/nvme0n2p1  375G   78G  297G  21% /data/apdb2
             apdb-server-5: /dev/nvme0n3p1  375G   78G  298G  21% /data/apdb3
             apdb-server-5: /dev/nvme0n4p1  375G   78G  298G  21% /data/apdb4
             apdb-server-5: /dev/nvme0n5p1  375G   78G  297G  21% /data/apdb5
             apdb-server-5: /dev/nvme0n6p1  375G   78G  298G  21% /data/apdb6
             apdb-server-5: /dev/nvme0n7p1  375G   78G  298G  21% /data/apdb7
             apdb-server-5: /dev/nvme0n8p1  375G   78G  297G  21% /data/apdb8
            apdb-server-10: Filesystem      Size  Used Avail Use% Mounted on
            apdb-server-10: /dev/nvme0n1p1  375G   82G  294G  22% /data/apdb1
            apdb-server-10: /dev/nvme0n2p1  375G   79G  296G  22% /data/apdb2
            apdb-server-10: /dev/nvme0n3p1  375G   79G  297G  21% /data/apdb3
            apdb-server-10: /dev/nvme0n4p1  375G   80G  296G  22% /data/apdb4
            apdb-server-10: /dev/nvme0n5p1  375G   80G  296G  22% /data/apdb5
            apdb-server-10: /dev/nvme0n6p1  375G   80G  296G  22% /data/apdb6
            apdb-server-10: /dev/nvme0n7p1  375G   81G  295G  22% /data/apdb7
            apdb-server-10: /dev/nvme0n8p1  375G   80G  296G  22% /data/apdb8
             apdb-server-9: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-9: /dev/nvme0n1p1  375G   83G  293G  22% /data/apdb1
             apdb-server-9: /dev/nvme0n2p1  375G   78G  298G  21% /data/apdb2
             apdb-server-9: /dev/nvme0n3p1  375G   76G  300G  21% /data/apdb3
             apdb-server-9: /dev/nvme0n4p1  375G   77G  299G  21% /data/apdb4
             apdb-server-9: /dev/nvme0n5p1  375G   76G  300G  21% /data/apdb5
             apdb-server-9: /dev/nvme0n6p1  375G   77G  299G  21% /data/apdb6
             apdb-server-9: /dev/nvme0n7p1  375G   77G  299G  21% /data/apdb7
             apdb-server-9: /dev/nvme0n8p1  375G   76G  300G  21% /data/apdb8
             apdb-server-8: Filesystem      Size  Used Avail Use% Mounted on
             apdb-server-8: /dev/nvme0n1p1  375G   79G  297G  21% /data/apdb1
             apdb-server-8: /dev/nvme0n2p1  375G   77G  299G  21% /data/apdb2
             apdb-server-8: /dev/nvme0n3p1  375G   77G  299G  21% /data/apdb3
             apdb-server-8: /dev/nvme0n4p1  375G   76G  299G  21% /data/apdb4
             apdb-server-8: /dev/nvme0n5p1  375G   76G  299G  21% /data/apdb5
             apdb-server-8: /dev/nvme0n6p1  375G   76G  299G  21% /data/apdb6
             apdb-server-8: /dev/nvme0n7p1  375G   77G  298G  21% /data/apdb7
             apdb-server-8: /dev/nvme0n8p1  375G   76G  300G  21% /data/apdb8
            apdb-server-12: Filesystem      Size  Used Avail Use% Mounted on
            apdb-server-12: /dev/nvme0n1p1  375G   83G  293G  22% /data/apdb1
            apdb-server-12: /dev/nvme0n2p1  375G   76G  300G  21% /data/apdb2
            apdb-server-12: /dev/nvme0n3p1  375G   74G  301G  20% /data/apdb3
            apdb-server-12: /dev/nvme0n4p1  375G   75G  301G  20% /data/apdb4
            apdb-server-12: /dev/nvme0n5p1  375G   76G  300G  21% /data/apdb5
            apdb-server-12: /dev/nvme0n6p1  375G   75G  300G  21% /data/apdb6
            apdb-server-12: /dev/nvme0n7p1  375G   76G  300G  21% /data/apdb7
            apdb-server-12: /dev/nvme0n8p1  375G   76G  300G  21% /data/apdb8 

            Total 7895566916 kiB or 7.35 TiB

            More detailed summary with pictures later.

            Show
            salnikov Andy Salnikov added a comment - - edited Ran fro 124k visits with 60-day partitions. Standard report: # nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.128.0.118 676.75 GiB 256 ? 52d05f52-d01c-40a0-8560-75e8d3e5a20d rack1 UN 10.128.0.91 591.55 GiB 256 ? d824d71d-dd3b-4c18-92d1-ef4f6dc2a866 rack1 UN 10.128.0.119 604.98 GiB 256 ? 9487a7fe-b6db-4d69-91b2-2fa567197d45 rack1 UN 10.128.0.103 605.88 GiB 256 ? 767278b0-c12d-4def-98ef-c0e9e811fcbd rack1 UN 10.128.0.57 632.97 GiB 256 ? ed1f8fff-2e9f-4656-b9e8-565df3457931 rack1 UN 10.128.0.123 596.53 GiB 256 ? 5e9be0a6-fcc8-4a16-a9a7-4d35d77ca85c rack1 UN 10.128.0.101 624.45 GiB 256 ? 44612049-898b-4116-b12b-da30474b4181 rack1 UN 10.128.0.121 631.38 GiB 256 ? ffdfacc7-8025-4978-92bf-764d56e0dd20 rack1 UN 10.128.0.122 615.54 GiB 256 ? 7ed00b36-3a2b-4cb2-8fbe-1f12d1d009a6 rack1 UN 10.128.0.100 618.22 GiB 256 ? ae4d4462-de5d-4726-b600-e23a16b05fde rack1 UN 10.128.0.52 640.33 GiB 256 ? 7284d69d-57c1-4f27-8b61-f6fb6739aae9 rack1 UN 10.128.0.37 595.68 GiB 256 ? e50cf9e5-3777-4dd2-91a0-a6cef931a8dd rack1 Total load size 7434.26 GiB = 7.26 TiB. Data sizes on each node: $ shmux -m -s -c "df -h /data/apdb*" -- apdb-server-{1..12} apdb-server-1: Filesystem Size Used Avail Use% Mounted on apdb-server-1: /dev/nvme0n1p1 375G 77G 299G 21% /data/apdb1 apdb-server-1: /dev/nvme0n2p1 375G 75G 300G 20% /data/apdb2 apdb-server-1: /dev/nvme0n3p1 375G 75G 301G 20% /data/apdb3 apdb-server-1: /dev/nvme0n4p1 375G 75G 301G 20% /data/apdb4 apdb-server-1: /dev/nvme0n5p1 375G 76G 300G 21% /data/apdb5 apdb-server-1: /dev/nvme0n6p1 375G 76G 300G 21% /data/apdb6 apdb-server-1: /dev/nvme0n7p1 375G 75G 300G 20% /data/apdb7 apdb-server-1: /dev/nvme0n8p1 375G 75G 300G 20% /data/apdb8 apdb-server-6: Filesystem Size Used Avail Use% Mounted on apdb-server-6: /dev/nvme0n1p1 375G 84G 292G 23% /data/apdb1 apdb-server-6: /dev/nvme0n2p1 375G 80G 296G 22% /data/apdb2 apdb-server-6: /dev/nvme0n3p1 375G 80G 296G 22% /data/apdb3 apdb-server-6: /dev/nvme0n4p1 375G 79G 297G 21% /data/apdb4 apdb-server-6: /dev/nvme0n5p1 375G 78G 297G 21% /data/apdb5 apdb-server-6: /dev/nvme0n6p1 375G 79G 297G 21% /data/apdb6 apdb-server-6: /dev/nvme0n7p1 375G 78G 297G 21% /data/apdb7 apdb-server-6: /dev/nvme0n8p1 375G 79G 297G 21% /data/apdb8 apdb-server-4: Filesystem Size Used Avail Use% Mounted on apdb-server-4: /dev/nvme0n1p1 375G 82G 293G 22% /data/apdb1 apdb-server-4: /dev/nvme0n2p1 375G 74G 302G 20% /data/apdb2 apdb-server-4: /dev/nvme0n3p1 375G 74G 301G 20% /data/apdb3 apdb-server-4: /dev/nvme0n4p1 375G 75G 301G 20% /data/apdb4 apdb-server-4: /dev/nvme0n5p1 375G 75G 301G 20% /data/apdb5 apdb-server-4: /dev/nvme0n6p1 375G 76G 300G 21% /data/apdb6 apdb-server-4: /dev/nvme0n7p1 375G 75G 301G 20% /data/apdb7 apdb-server-4: /dev/nvme0n8p1 375G 75G 301G 20% /data/apdb8 apdb-server-11: Filesystem Size Used Avail Use% Mounted on apdb-server-11: /dev/nvme0n1p1 375G 84G 292G 23% /data/apdb1 apdb-server-11: /dev/nvme0n2p1 375G 77G 299G 21% /data/apdb2 apdb-server-11: /dev/nvme0n3p1 375G 77G 298G 21% /data/apdb3 apdb-server-11: /dev/nvme0n4p1 375G 78G 297G 21% /data/apdb4 apdb-server-11: /dev/nvme0n5p1 375G 78G 298G 21% /data/apdb5 apdb-server-11: /dev/nvme0n6p1 375G 78G 298G 21% /data/apdb6 apdb-server-11: /dev/nvme0n7p1 375G 78G 298G 21% /data/apdb7 apdb-server-11: /dev/nvme0n8p1 375G 77G 298G 21% /data/apdb8 apdb-server-3: Filesystem Size Used Avail Use% Mounted on apdb-server-3: /dev/nvme0n1p1 375G 84G 291G 23% /data/apdb1 apdb-server-3: /dev/nvme0n2p1 375G 80G 296G 22% /data/apdb2 apdb-server-3: /dev/nvme0n3p1 375G 80G 295G 22% /data/apdb3 apdb-server-3: /dev/nvme0n4p1 375G 80G 296G 22% /data/apdb4 apdb-server-3: /dev/nvme0n5p1 375G 80G 296G 22% /data/apdb5 apdb-server-3: /dev/nvme0n6p1 375G 80G 296G 22% /data/apdb6 apdb-server-3: /dev/nvme0n7p1 375G 80G 295G 22% /data/apdb7 apdb-server-3: /dev/nvme0n8p1 375G 81G 295G 22% /data/apdb8 apdb-server-2: Filesystem Size Used Avail Use% Mounted on apdb-server-2: /dev/nvme0n1p1 375G 83G 293G 22% /data/apdb1 apdb-server-2: /dev/nvme0n2p1 375G 81G 295G 22% /data/apdb2 apdb-server-2: /dev/nvme0n3p1 375G 81G 295G 22% /data/apdb3 apdb-server-2: /dev/nvme0n4p1 375G 81G 295G 22% /data/apdb4 apdb-server-2: /dev/nvme0n5p1 375G 81G 294G 22% /data/apdb5 apdb-server-2: /dev/nvme0n6p1 375G 81G 295G 22% /data/apdb6 apdb-server-2: /dev/nvme0n7p1 375G 81G 295G 22% /data/apdb7 apdb-server-2: /dev/nvme0n8p1 375G 81G 295G 22% /data/apdb8 apdb-server-7: Filesystem Size Used Avail Use% Mounted on apdb-server-7: /dev/nvme0n1p1 375G 91G 285G 25% /data/apdb1 apdb-server-7: /dev/nvme0n2p1 375G 85G 291G 23% /data/apdb2 apdb-server-7: /dev/nvme0n3p1 375G 85G 290G 23% /data/apdb3 apdb-server-7: /dev/nvme0n4p1 375G 85G 290G 23% /data/apdb4 apdb-server-7: /dev/nvme0n5p1 375G 85G 291G 23% /data/apdb5 apdb-server-7: /dev/nvme0n6p1 375G 86G 290G 23% /data/apdb6 apdb-server-7: /dev/nvme0n7p1 375G 85G 291G 23% /data/apdb7 apdb-server-7: /dev/nvme0n8p1 375G 87G 289G 24% /data/apdb8 apdb-server-5: Filesystem Size Used Avail Use% Mounted on apdb-server-5: /dev/nvme0n1p1 375G 84G 292G 23% /data/apdb1 apdb-server-5: /dev/nvme0n2p1 375G 78G 297G 21% /data/apdb2 apdb-server-5: /dev/nvme0n3p1 375G 78G 298G 21% /data/apdb3 apdb-server-5: /dev/nvme0n4p1 375G 78G 298G 21% /data/apdb4 apdb-server-5: /dev/nvme0n5p1 375G 78G 297G 21% /data/apdb5 apdb-server-5: /dev/nvme0n6p1 375G 78G 298G 21% /data/apdb6 apdb-server-5: /dev/nvme0n7p1 375G 78G 298G 21% /data/apdb7 apdb-server-5: /dev/nvme0n8p1 375G 78G 297G 21% /data/apdb8 apdb-server-10: Filesystem Size Used Avail Use% Mounted on apdb-server-10: /dev/nvme0n1p1 375G 82G 294G 22% /data/apdb1 apdb-server-10: /dev/nvme0n2p1 375G 79G 296G 22% /data/apdb2 apdb-server-10: /dev/nvme0n3p1 375G 79G 297G 21% /data/apdb3 apdb-server-10: /dev/nvme0n4p1 375G 80G 296G 22% /data/apdb4 apdb-server-10: /dev/nvme0n5p1 375G 80G 296G 22% /data/apdb5 apdb-server-10: /dev/nvme0n6p1 375G 80G 296G 22% /data/apdb6 apdb-server-10: /dev/nvme0n7p1 375G 81G 295G 22% /data/apdb7 apdb-server-10: /dev/nvme0n8p1 375G 80G 296G 22% /data/apdb8 apdb-server-9: Filesystem Size Used Avail Use% Mounted on apdb-server-9: /dev/nvme0n1p1 375G 83G 293G 22% /data/apdb1 apdb-server-9: /dev/nvme0n2p1 375G 78G 298G 21% /data/apdb2 apdb-server-9: /dev/nvme0n3p1 375G 76G 300G 21% /data/apdb3 apdb-server-9: /dev/nvme0n4p1 375G 77G 299G 21% /data/apdb4 apdb-server-9: /dev/nvme0n5p1 375G 76G 300G 21% /data/apdb5 apdb-server-9: /dev/nvme0n6p1 375G 77G 299G 21% /data/apdb6 apdb-server-9: /dev/nvme0n7p1 375G 77G 299G 21% /data/apdb7 apdb-server-9: /dev/nvme0n8p1 375G 76G 300G 21% /data/apdb8 apdb-server-8: Filesystem Size Used Avail Use% Mounted on apdb-server-8: /dev/nvme0n1p1 375G 79G 297G 21% /data/apdb1 apdb-server-8: /dev/nvme0n2p1 375G 77G 299G 21% /data/apdb2 apdb-server-8: /dev/nvme0n3p1 375G 77G 299G 21% /data/apdb3 apdb-server-8: /dev/nvme0n4p1 375G 76G 299G 21% /data/apdb4 apdb-server-8: /dev/nvme0n5p1 375G 76G 299G 21% /data/apdb5 apdb-server-8: /dev/nvme0n6p1 375G 76G 299G 21% /data/apdb6 apdb-server-8: /dev/nvme0n7p1 375G 77G 298G 21% /data/apdb7 apdb-server-8: /dev/nvme0n8p1 375G 76G 300G 21% /data/apdb8 apdb-server-12: Filesystem Size Used Avail Use% Mounted on apdb-server-12: /dev/nvme0n1p1 375G 83G 293G 22% /data/apdb1 apdb-server-12: /dev/nvme0n2p1 375G 76G 300G 21% /data/apdb2 apdb-server-12: /dev/nvme0n3p1 375G 74G 301G 20% /data/apdb3 apdb-server-12: /dev/nvme0n4p1 375G 75G 301G 20% /data/apdb4 apdb-server-12: /dev/nvme0n5p1 375G 76G 300G 21% /data/apdb5 apdb-server-12: /dev/nvme0n6p1 375G 75G 300G 21% /data/apdb6 apdb-server-12: /dev/nvme0n7p1 375G 76G 300G 21% /data/apdb7 apdb-server-12: /dev/nvme0n8p1 375G 76G 300G 21% /data/apdb8 Total 7895566916 kiB or 7.35 TiB More detailed summary with pictures later.
            Hide
            salnikov Andy Salnikov added a comment -

            Summary of the test:

            • 12 server nodes, 32 cores, 64GB RAM each
            • 8x375GiB local SSD storage on each node
            • replication factor 3, QUORUM consistency for reads and writes
            • 6 client machines, 32 cores each
            • 124k visits generated
            • Partition size for time variable is set to 60 days (partitioning as before is done vie separate tables, no with Cassandra partitioning)

            (note for the future test option - try Cassandra partitioning instead of separate tables, that would complicate storage management, but I'm just curios if that can be faster than table-per-partiton approach)

            The purpose of the test was to see if different time partition size changes performance due to difference in number of concurrent queries, the result is that I do not see significant changes after doubling partition size. With this doubled size every client runs 7 parallel queries instead of 13. Here is the plot for initial 10k visits, it is practically identical to the same plot from DM-28172:

            Given that increased partition size does not improve timing, but it will increase overhead after reaching 12 months it seems reasonable to keep partition size at 1 month (which should translate into 4% average overhead at 12 months).

            One interesting and puzzling thing that I noticed during this test (and I thing it's the same issue I saw already in earlier test) - two of the 12 server nodes were showing higher then average CPU load, check plots below. After noticing this on monitoring plots I looked at what was happening on those hosts and did not see anything strange there. It looked like the additional CPU (or actually many extra threads) were due to Cassandra itself. There was no memory pressure ad GC monitoring did not show any difference in activity between these two machines and remaining cluster. After messing around for a while I stopped ap_proto and restarted Cassandra server on just those two machines. After re-starting ap_proto the situation normalized and those two hosts behaved identically to other machines. Maybe something to ask Cassandra people about.

            And here is the bunch of plots.

            Scatter plot for select and store times vs visit number:

            Real time for selects for individual tables (total time is just the sum of three other values):

            Real store times for individual tables:

            Monitoring plots from grafana:

            System load average, clearly shows higher load from apdb-server-6 and apdb-server-7, dropped to average after restarting Cassandra:

            Cassandra metrics for active tasks/threads for "mutation stage" also shows very high count for those two nodes:

            Write latency for r each individual table shows the effect of that on store latency (not a huge increase but very noticeable):




            Timing from ap_proto logs for store operation per table:

            Timing from ap_proto logs for select operation per table, this is not affected by whatever happened to those two hosts:

            Show
            salnikov Andy Salnikov added a comment - Summary of the test: 12 server nodes, 32 cores, 64GB RAM each 8x375GiB local SSD storage on each node replication factor 3, QUORUM consistency for reads and writes 6 client machines, 32 cores each 124k visits generated Partition size for time variable is set to 60 days (partitioning as before is done vie separate tables, no with Cassandra partitioning) (note for the future test option - try Cassandra partitioning instead of separate tables, that would complicate storage management, but I'm just curios if that can be faster than table-per-partiton approach) The purpose of the test was to see if different time partition size changes performance due to difference in number of concurrent queries, the result is that I do not see significant changes after doubling partition size. With this doubled size every client runs 7 parallel queries instead of 13. Here is the plot for initial 10k visits, it is practically identical to the same plot from DM-28172 : Given that increased partition size does not improve timing, but it will increase overhead after reaching 12 months it seems reasonable to keep partition size at 1 month (which should translate into 4% average overhead at 12 months). One interesting and puzzling thing that I noticed during this test (and I thing it's the same issue I saw already in earlier test) - two of the 12 server nodes were showing higher then average CPU load, check plots below. After noticing this on monitoring plots I looked at what was happening on those hosts and did not see anything strange there. It looked like the additional CPU (or actually many extra threads) were due to Cassandra itself. There was no memory pressure ad GC monitoring did not show any difference in activity between these two machines and remaining cluster. After messing around for a while I stopped ap_proto and restarted Cassandra server on just those two machines. After re-starting ap_proto the situation normalized and those two hosts behaved identically to other machines. Maybe something to ask Cassandra people about. And here is the bunch of plots. Scatter plot for select and store times vs visit number: Real time for selects for individual tables (total time is just the sum of three other values): Real store times for individual tables: Monitoring plots from grafana: System load average, clearly shows higher load from apdb-server-6 and apdb-server-7, dropped to average after restarting Cassandra: Cassandra metrics for active tasks/threads for "mutation stage" also shows very high count for those two nodes: Write latency for r each individual table shows the effect of that on store latency (not a huge increase but very noticeable): Timing from ap_proto logs for store operation per table: Timing from ap_proto logs for select operation per table, this is not affected by whatever happened to those two hosts:
            Hide
            salnikov Andy Salnikov added a comment -

            The conclusion for this test seems to be that doubling the size of the timing partition in a table-per-partition approach does not change performance numbers at all. Good thing is that we can chose partition size to satisfy other requirements and it should not affect performance. I believe 30-day partition size is probably reasonable at this point. For next test I want to check how native Cassandra partitioning behaves compared to this.

            Closing this ticket.

            Show
            salnikov Andy Salnikov added a comment - The conclusion for this test seems to be that doubling the size of the timing partition in a table-per-partition approach does not change performance numbers at all. Good thing is that we can chose partition size to satisfy other requirements and it should not affect performance. I believe 30-day partition size is probably reasonable at this point. For next test I want to check how native Cassandra partitioning behaves compared to this. Closing this ticket.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Watchers:
              Andy Salnikov, Colin Slater, Fritz Mueller
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.