diff --git a/doc/bugs/significant_performance_regression_impacting_datal/comment_12_c841c9db2000c65c56ce8e26e58a3f62._comment b/doc/bugs/significant_performance_regression_impacting_datal/comment_12_c841c9db2000c65c56ce8e26e58a3f62._comment new file mode 100644 index 0000000000..4d84790533 --- /dev/null +++ b/doc/bugs/significant_performance_regression_impacting_datal/comment_12_c841c9db2000c65c56ce8e26e58a3f62._comment @@ -0,0 +1,153 @@ +[[!comment format=mdwn + username="yarikoptic" + avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4" + subject="all recent builds/logs are fetched to smaug" + date="2021-06-08T16:50:11Z" + content=""" +Finished fetching all the builds (logs + artifacts, i.e. built installers and packages) to smaug. FWIW `joey` user on smaug should be able to access them at /mnt/datasets/datalad/ci/git-annex as well (it is all git/git-annex with submodules happen you want to get a \"clone\" ;)) + +
+grep confirms timing for datalad testson OSX + +```shell +(tinuous-dev) datalad@smaug:/mnt/datasets/datalad/ci/git-annex$ grep -h -i 'Ran .*tests in' builds/2021/0[56]/*/cron-*/*/*macOS*/*_test-datalad\ \(master\).txt +2021-05-01T05:47:43.6623300Z Ran 1269 tests in 10381.041s +2021-05-02T04:27:47.4468150Z Ran 1269 tests in 5835.199s +2021-05-03T04:25:06.4741780Z Ran 1269 tests in 5817.953s +2021-05-04T04:26:25.2725400Z Ran 1269 tests in 5925.938s +2021-05-05T04:15:45.7241110Z Ran 1270 tests in 6003.093s +2021-05-06T04:14:18.9203140Z Ran 1270 tests in 5821.084s +2021-05-07T04:19:12.5257430Z Ran 1270 tests in 5887.912s +2021-05-08T04:17:21.4790320Z Ran 1270 tests in 5775.691s +2021-05-09T04:20:53.2085730Z Ran 1270 tests in 5961.437s +2021-05-10T04:21:52.2492300Z Ran 1270 tests in 5698.905s +2021-05-11T04:28:39.1451570Z Ran 1270 tests in 5899.918s +2021-05-12T04:28:26.6181110Z Ran 1270 tests in 5964.851s +2021-05-13T04:30:47.2978230Z Ran 1277 tests in 5938.297s +2021-05-14T04:37:51.3172970Z Ran 1277 tests in 5921.571s +2021-05-15T04:31:29.6466840Z Ran 1277 tests in 6060.762s +2021-05-16T04:33:26.5552670Z Ran 1277 tests in 5789.405s +2021-05-17T04:36:52.4460760Z Ran 1277 tests in 5855.382s +2021-05-18T04:37:24.5613580Z Ran 1278 tests in 6007.685s +2021-05-19T04:34:37.3253830Z Ran 1279 tests in 5940.204s +2021-05-20T04:36:10.3184010Z Ran 1280 tests in 5941.388s +2021-05-21T04:41:10.2664330Z Ran 1280 tests in 6218.775s +2021-05-22T04:40:39.2754540Z Ran 1289 tests in 5884.267s +2021-05-23T04:37:51.5005200Z Ran 1289 tests in 5750.672s +2021-05-24T04:38:24.5894000Z Ran 1289 tests in 5911.655s +2021-05-25T04:51:35.2189100Z Ran 1289 tests in 6266.836s +2021-05-26T06:56:33.6660300Z Ran 1293 tests in 12584.992s +2021-05-27T07:08:27.4015580Z Ran 1293 tests in 11901.552s +2021-05-28T07:39:17.3481450Z Ran 1292 tests in 12094.343s +2021-05-29T08:05:14.3586440Z Ran 1294 tests in 12205.434s +2021-05-30T08:09:33.0780990Z Ran 1294 tests in 12028.089s +2021-05-31T08:16:36.3910830Z Ran 1294 tests in 12329.455s +2021-06-01T08:42:24.3167200Z Ran 1294 tests in 12115.378s +2021-06-02T08:42:23.1985610Z Ran 1294 tests in 12432.309s +2021-06-03T08:00:08.2576030Z Ran 1294 tests in 11556.974s +2021-06-04T07:25:39.1674160Z Ran 1294 tests in 11946.195s +2021-06-05T07:14:00.5262620Z Ran 1294 tests in 12456.432s +2021-06-06T06:53:14.7001370Z Ran 1294 tests in 11677.647s +2021-06-07T07:12:57.5076610Z Ran 1294 tests in 12042.332s +2021-06-08T06:48:01.3977250Z Ran 1294 tests in 12126.463 +``` + +Interestingly the top one is in 10k, but prior ones were ok + +```shell +(tinuous-dev) datalad@smaug:/mnt/datasets/datalad/ci/git-annex$ grep -h -i 'Ran .*tests in' builds/2021/04/*/cron-*/*/*macOS*/*_test-datalad\ \(master\).txt +2021-04-27T03:58:30.4829510Z Ran 1265 tests in 6312.819s +2021-04-28T03:44:23.1655040Z Ran 1265 tests in 5622.562s +2021-04-29T03:50:14.0670840Z Ran 1269 tests in 6196.774s +2021-04-30T04:21:45.1727310Z Ran 1269 tests in 5879.034s +``` +
+ +
+and absence of the drastic effect confirmed for linux and windows + +```shell +(tinuous-dev) datalad@smaug:/mnt/datasets/datalad/ci/git-annex$ grep -h -i 'Ran .*tests in' builds/2021/0[56]/*/cron-*/*/*buntu*/*_test-datalad\ \(master\).txt +2021-05-01T04:16:34.1539917Z Ran 1269 tests in 3999.905s +2021-05-02T04:17:04.5604428Z Ran 1269 tests in 3745.951s +2021-05-03T04:22:46.1508135Z Ran 1269 tests in 4133.703s +2021-05-04T04:22:19.3612212Z Ran 1269 tests in 4341.995s +2021-05-05T04:23:04.3327570Z Ran 1270 tests in 4383.268s +2021-05-06T04:07:33.2967674Z Ran 1270 tests in 3457.688s +2021-05-07T04:18:26.7140893Z Ran 1270 tests in 3778.162s +2021-05-08T04:13:13.9041377Z Ran 1270 tests in 3728.503s +2021-05-09T04:16:59.6844666Z Ran 1270 tests in 3898.714s +2021-05-11T04:21:45.5447035Z Ran 1270 tests in 4032.457s +2021-05-12T04:20:48.6349049Z Ran 1270 tests in 3940.342s +2021-05-13T04:26:53.8376010Z Ran 1277 tests in 4130.863s +2021-05-14T04:37:41.5689917Z Ran 1277 tests in 4476.880s +2021-05-15T04:20:12.0126613Z Ran 1277 tests in 4007.927s +2021-05-16T04:30:50.0329536Z Ran 1277 tests in 4077.010s +2021-05-17T04:25:03.3302428Z Ran 1277 tests in 4034.791s +2021-05-18T04:33:23.5513354Z Ran 1278 tests in 4422.760s +2021-05-19T04:19:47.5664435Z Ran 1279 tests in 3910.031s +2021-05-20T04:13:45.4617209Z Ran 1280 tests in 3508.342s +2021-05-21T04:23:56.1103373Z Ran 1280 tests in 3946.563s +2021-05-22T04:30:52.1649178Z Ran 1289 tests in 4406.863s +2021-05-23T04:35:53.5526207Z Ran 1289 tests in 3818.054s +2021-05-24T04:33:44.9770098Z Ran 1289 tests in 4069.751s +2021-05-25T04:38:28.6369074Z Ran 1289 tests in 3809.039s +2021-05-26T05:16:56.6913239Z Ran 1293 tests in 4340.973s +2021-05-27T05:21:33.5230620Z Ran 1293 tests in 4303.062s +2021-05-28T05:44:02.6763227Z Ran 1292 tests in 4013.921s +2021-05-29T06:05:40.8639499Z Ran 1294 tests in 3581.055s +2021-05-30T06:17:23.4265104Z Ran 1294 tests in 4257.736s +2021-05-31T06:36:09.6446865Z Ran 1294 tests in 4782.606s +2021-06-01T06:57:26.6049829Z Ran 1294 tests in 4391.030s +2021-06-02T06:59:17.8345547Z Ran 1294 tests in 4737.580s +2021-06-03T06:10:51.5533496Z Ran 1294 tests in 3453.557s +2021-06-04T05:44:15.7873867Z Ran 1294 tests in 4113.770s +2021-06-05T05:25:15.5789949Z Ran 1294 tests in 4383.014s +2021-06-06T05:20:48.7371175Z Ran 1294 tests in 4386.210s +2021-06-07T05:33:24.8643085Z Ran 1294 tests in 4303.855s +2021-06-08T05:03:38.3420188Z Ran 1294 tests in 4371.387s +``` + +```shell +(tinuous-dev) datalad@smaug:/mnt/datasets/datalad/ci/git-annex$ grep -h -i 'Ran .*tests in' builds/2021/0[56]/*/cron-*/*/*indows*/*_test-datalad\ \(master\).txt +2021-05-01T05:29:21.7469495Z Ran 1250 tests in 4904.324s +2021-05-02T05:33:35.7321469Z Ran 1250 tests in 4835.792s +2021-05-03T05:32:36.9342468Z Ran 1250 tests in 5040.210s +2021-05-04T05:18:53.5090364Z Ran 1250 tests in 4508.693s +2021-05-05T05:29:53.0946195Z Ran 1251 tests in 4989.032s +2021-05-06T05:26:05.6627667Z Ran 1251 tests in 4766.005s +2021-05-07T05:19:54.3788683Z Ran 1251 tests in 4305.963s +2021-05-08T05:32:49.9787816Z Ran 1251 tests in 4999.028s +2021-05-09T05:38:03.4560444Z Ran 1251 tests in 5482.509s +2021-05-10T05:34:58.3502335Z Ran 1251 tests in 5067.805s +2021-05-11T05:29:07.7916431Z Ran 1251 tests in 4797.435s +2021-05-12T05:23:10.5488731Z Ran 1251 tests in 4276.426s +2021-05-13T05:25:09.1884039Z Ran 1258 tests in 4275.409s +2021-05-14T05:39:20.0295201Z Ran 1258 tests in 4999.023s +2021-05-15T05:31:48.8021685Z Ran 1258 tests in 4736.174s +2021-05-16T05:33:12.0136713Z Ran 1258 tests in 4310.326s +2021-05-17T05:30:27.7704400Z Ran 1258 tests in 4561.060s +2021-05-18T05:31:34.3746293Z Ran 1259 tests in 4426.723s +2021-05-19T05:31:11.8604863Z Ran 1260 tests in 4903.506s +2021-05-20T05:22:56.6087010Z Ran 1261 tests in 4299.536s +2021-05-21T05:28:25.7511300Z Ran 1261 tests in 4498.286s +2021-05-22T05:44:54.2975226Z Ran 1270 tests in 5376.292s +2021-05-23T05:35:29.5504579Z Ran 1270 tests in 4289.031s +2021-05-24T05:33:40.4688984Z Ran 1270 tests in 4202.429s +2021-05-25T05:45:30.0310577Z Ran 1270 tests in 4882.292s +2021-05-26T06:03:08.4980635Z Ran 1274 tests in 4441.783s +2021-05-27T06:05:25.0347805Z Ran 1274 tests in 4381.865s +2021-05-28T06:35:10.3614775Z Ran 1273 tests in 4529.720s +2021-05-29T07:10:28.5980900Z Ran 1275 tests in 4639.335s +2021-05-30T07:01:37.6908114Z Ran 1275 tests in 5123.743s +2021-05-31T07:01:41.6742606Z Ran 1275 tests in 4672.112s +2021-06-01T07:36:38.0359361Z Ran 1275 tests in 5183.674s +2021-06-02T07:21:46.3022239Z Ran 1275 tests in 4610.975s +2021-06-03T07:20:14.4589015Z Ran 1275 tests in 5795.353s +2021-06-04T06:44:13.4142157Z Ran 1275 tests in 4574.473s +2021-06-08T06:10:00.4654735Z Ran 1275 tests in 5406.636s +``` +
+ +now will try to do timing on a local OSX to see if it is indeed just some coincidence with OSX VM getting slower or it is \"real\". +"""]]