Summary of all tests (just for my own record for which ticket did what). Each "gcp" test starts from scratch with empty database.
gcp-1:
- 2020-12-18 23:04 to 2020-12-20 18:30 (all times in UTC)
- 50k visits
- initial test, same setup as PDAC, 3 node cluster
DM-28136
gcp-2:
- 2020-12-23 00:50 to 2020-12-25 15:57
- 6-node cluster
- still using afw.Table for in-memory data
- 75k visits of useful data, storage errors later, 89k visits total
DM-28154
gcp-3:
- 2020-12-30 01:51 to 2021-01-02 04:20
- 6-node cluster
- switched to pandas for in-memory representation
- 100k visits
DM-28154
gcp-4:
- 2021-01-05 19:23 to 2021-01-19 23:44
- 12-node cluster, long test >1 year
- 450k visits
DM-28172
gcp-5:
- 2021-01-23 04:11 to 2021-01-26 00:42
- 2-month temporal partitioning
- 124k visits
DM-28467
gcp-6:
- 2021-01-27 22:02 to 2021-01-30 02:07
- native time partitioning
- one single SELECT (WHERE apdb_part IN (...) AND apdb_time_part IN (...))
- 100k visits
DM-28522
gcp-7:
- 2021-02-01 19:49 to 2021-02-04 22:00
- native time partitioning
- first 100k visits with one query per time partition (13 queries)
- remaining 30k visits with one query per spatial partition (~200 queries)
- 130542 visits
DM-28522
gcp-8:
- 2021-02-09 00:02 to 2021-02-11 00:24
- native time partitioning
- optimized result to pandas conversion (concatenate results then convert)
- first 79k visits with one query per time partition (13 queries)
- remaining visits with one query per spatial partition (~200 queries)
- 91242 visits total
DM-28522
gcp-9:
- 2021-02-17 03:08 to 2021-02-20 00:17
- packing fields into BLOBS with CBOR
- also check timing without converting sources into pandas
- 139352 visits
DM-28820
Summary of all tests (just for my own record for which ticket did what). Each "gcp" test starts from scratch with empty database.
gcp-1:
DM-28136gcp-2:
DM-28154gcp-3:
DM-28154gcp-4:
DM-28172gcp-5:
DM-28467gcp-6:
DM-28522gcp-7:
DM-28522gcp-8:
DM-28522gcp-9:
DM-28820