HEALTH_ERR noout flag(s) set; 25 osds down; 7055371 scrub errors; Reduced data availability: 138 pgs inactive, 103 pgs down; Possible data damage: 30 pgs inconsistent; Degraded data redundancy: 1797720/26981188 objects degraded (6.663%), 47 pgs degraded, 130 pgs undersized; 49 daemons have recently crashed [WRN] OSDMAP_FLAGS: noout flag(s) set [WRN] OSD_DOWN: 25 osds down osd.12 (root=default,host=balin002) is down osd.39 (root=default,host=balin004) is down osd.41 (root=default,host=balin004) is down osd.58 (root=default,host=balin005) is down osd.59 (root=default,host=balin005) is down osd.62 (root=default,host=balin006) is down osd.81 (root=default,host=balin007) is down osd.93 (root=default,host=balin009) is down osd.104 (root=default,host=balin008) is down osd.109 (root=default,host=balin010) is down osd.115 (root=default,host=balin010) is down osd.121 (root=default,host=balin011) is down osd.130 (root=default,host=balin011) is down osd.134 (root=default,host=balin012) is down osd.141 (root=default,host=balin012) is down osd.155 (root=default,host=balin013) is down osd.156 (root=default,host=balin014) is down osd.163 (root=default,host=balin014) is down osd.168 (root=default,host=balin015) is down osd.174 (root=default,host=balin015) is down osd.188 (root=default,host=balin016) is down osd.195 (root=default,host=balin017) is down osd.206 (root=default,host=balin018) is down osd.212 (root=default,host=balin018) is down osd.230 (root=default,host=balin020) is down [ERR] OSD_SCRUB_ERRORS: 7055371 scrub errors [WRN] PG_AVAILABILITY: Reduced data availability: 138 pgs inactive, 103 pgs down pg 2.4 is stuck inactive for 2d, current state undersized+peered, last acting [191,84,217,208,83,NONE,NONE] pg 2.e is stuck inactive for 2d, current state undersized+peered, last acting [NONE,5,112,131,232,NONE,88] pg 2.14 is stuck inactive for 2d, current state undersized+peered, last acting [231,16,NONE,NONE,66,102,44] pg 2.1a is stuck inactive for 2d, current state undersized+peered, last acting [193,NONE,113,NONE,NONE,65,147] pg 2.1b is stuck inactive for 2d, current state undersized+peered, last acting [51,98,80,NONE,150,NONE,137] pg 3.0 is down, acting [33,82,54,198,223,63,213] pg 3.1 is down, acting [77,226,45,50,19,6,198] pg 3.2 is down, acting [149,35,NONE,84,129,99,65] pg 3.3 is down, acting [82,23,7,149,29,176,61] pg 3.4 is down, acting [11,236,146,186,95,166,70] pg 3.5 is down, acting [146,227,239,19,172,85,32] pg 3.6 is stuck inactive since forever, current state down, last acting [154,NONE,95,50,NONE,20,NONE] pg 3.7 is down, acting [80,27,101,89,200,114,20] pg 3.c is down+remapped, acting [NONE,238,NONE,24,NONE,NONE,208] pg 3.d is stuck inactive for 2d, current state undersized+degraded+peered, last acting [NONE,127,86,159,NONE,11,204] pg 3.f is down+remapped, acting [NONE,NONE,NONE,40,105,203,NONE] pg 3.10 is down+remapped, acting [NONE,NONE,128,74,217,NONE,NONE] pg 3.14 is stuck inactive for 2d, current state undersized+degraded+peered, last acting [NONE,202,49,NONE,209,NONE,9] pg 3.15 is down+remapped, acting [NONE,NONE,NONE,128,NONE,217,177] pg 3.19 is down+remapped, acting [NONE,NONE,NONE,NONE,2,99,43] pg 3.1a is down+remapped, acting [NONE,NONE,NONE,NONE,214,89,19] pg 3.1b is down+remapped, acting [NONE,NONE,NONE,NONE,NONE,113,143] pg 3.80 is down, acting [33,82,54,198,223,63,213] pg 3.81 is down, acting [77,226,45,50,19,6,198] pg 3.82 is down, acting [149,35,NONE,84,129,99,65] pg 3.83 is down, acting [82,23,7,149,29,176,61] pg 3.84 is down, acting [11,236,146,186,95,166,70] pg 3.85 is down, acting [146,227,239,19,172,85,32] pg 3.86 is down, acting [154,NONE,95,50,NONE,20,NONE] pg 3.87 is down, acting [80,27,101,89,200,114,20] pg 3.89 is stuck inactive for 2d, current state undersized+degraded+inconsistent+peered, last acting [65,NONE,NONE,103,NONE,220,24] pg 3.8a is down+remapped, acting [NONE,NONE,NONE,NONE,194,120,60] pg 3.8b is down+remapped, acting [NONE,NONE,NONE,166,NONE,19,3] pg 3.8c is down+remapped, acting [NONE,238,NONE,24,NONE,NONE,208] pg 3.8d is stuck inactive for 2d, current state undersized+degraded+peered, last acting [NONE,127,86,159,NONE,11,204] pg 3.8f is down+remapped, acting [NONE,NONE,NONE,40,105,203,NONE] pg 3.90 is down+remapped, acting [NONE,NONE,128,74,217,NONE,NONE] pg 3.94 is stuck inactive for 2d, current state undersized+degraded+peered, last acting [NONE,202,49,NONE,209,NONE,9] pg 3.95 is down+remapped, acting [NONE,NONE,NONE,128,NONE,217,177] pg 3.99 is down+remapped, acting [NONE,NONE,NONE,NONE,2,99,43] pg 3.9a is down+remapped, acting [NONE,NONE,NONE,NONE,214,89,19] pg 3.9b is down+remapped, acting [NONE,NONE,NONE,167,NONE,113,143] pg 3.9d is down+remapped, acting [NONE,NONE,NONE,NONE,161,150,225] pg 3.9e is down+remapped, acting [NONE,138,NONE,NONE,NONE,NONE,47] pg 3.a0 is down+remapped, acting [NONE,NONE,NONE,NONE,223,63,213] pg 3.a1 is down+remapped, acting [NONE,NONE,NONE,NONE,NONE,6,198] pg 3.a2 is down+remapped, acting [NONE,NONE,NONE,NONE,129,99,65] pg 3.a3 is down+remapped, acting [NONE,NONE,NONE,NONE,29,176,61] pg 5.0 is stuck inactive for 2d, current state undersized+peered, last acting [NONE,114,NONE,235,190,71] pg 5.9 is down, acting [80,215,NONE,NONE,61,NONE] pg 6.0 is stuck inactive for 2d, current state undersized+peered, last acting [204] [ERR] PG_DAMAGED: Possible data damage: 30 pgs inconsistent pg 3.9 is down+inconsistent, acting [65,NONE,NONE,103,NONE,220,24] pg 3.12 is active+clean+inconsistent, acting [107,88,147,1,158,224,63] pg 3.16 is active+clean+inconsistent, acting [19,235,128,65,136,209,117] pg 3.2e is active+clean+inconsistent, acting [219,46,94,21,144,31,160] pg 3.31 is active+clean+inconsistent, acting [33,13,123,47,197,226,108] pg 3.32 is active+clean+inconsistent, acting [107,88,147,1,158,224,63] pg 3.36 is active+clean+inconsistent, acting [19,235,128,65,136,209,117] pg 3.37 is active+undersized+degraded+inconsistent, acting [185,178,NONE,159,103,131,77] pg 3.38 is active+clean+inconsistent, acting [235,28,60,113,194,22,179] pg 3.49 is undersized+degraded+inconsistent+peered, acting [65,NONE,NONE,103,NONE,220,24] pg 3.4e is active+clean+inconsistent, acting [219,46,94,21,144,31,160] pg 3.51 is active+clean+inconsistent, acting [33,13,123,47,197,226,108] pg 3.52 is active+clean+inconsistent, acting [107,88,147,1,158,224,63] pg 3.56 is active+clean+inconsistent, acting [19,235,128,65,136,209,117] pg 3.57 is active+undersized+degraded+inconsistent, acting [185,178,NONE,159,103,131,77] pg 3.58 is active+clean+inconsistent, acting [235,28,60,113,194,22,179] pg 3.69 is undersized+degraded+inconsistent+peered, acting [65,NONE,NONE,103,NONE,220,24] pg 3.6e is active+clean+inconsistent, acting [219,46,94,21,144,31,160] pg 3.71 is active+clean+inconsistent, acting [33,13,123,47,197,226,108] pg 3.72 is active+clean+inconsistent, acting [107,88,147,1,158,224,63] pg 3.76 is active+clean+inconsistent, acting [19,235,128,65,136,209,117] pg 3.77 is active+undersized+degraded+inconsistent, acting [185,178,NONE,159,103,131,77] pg 3.78 is active+clean+inconsistent, acting [235,28,60,113,194,22,179] pg 3.89 is undersized+degraded+inconsistent+peered, acting [65,NONE,NONE,103,NONE,220,24] pg 3.8e is active+clean+inconsistent, acting [219,46,94,21,144,31,160] pg 3.91 is active+clean+inconsistent, acting [33,13,123,47,197,226,108] pg 3.92 is active+clean+inconsistent, acting [107,88,147,1,158,224,63] pg 3.96 is active+clean+inconsistent, acting [19,235,128,65,136,209,117] pg 3.97 is active+undersized+degraded+inconsistent, acting [185,178,NONE,159,103,131,77] pg 3.98 is active+clean+inconsistent, acting [235,28,60,113,194,22,179] [WRN] PG_DEGRADED: Degraded data redundancy: 1797720/26981188 objects degraded (6.663%), 47 pgs degraded, 130 pgs undersized pg 2.1 is stuck undersized for 2d, current state active+undersized, last acting [77,NONE,233,63,89,101,178] pg 2.4 is stuck undersized for 2d, current state undersized+peered, last acting [191,84,217,208,83,NONE,NONE] pg 2.5 is stuck undersized for 2d, current state active+undersized, last acting [186,8,204,138,NONE,116,80] pg 2.6 is stuck undersized for 2d, current state active+undersized, last acting [107,166,26,144,238,61,NONE] pg 2.c is stuck undersized for 2d, current state active+undersized, last acting [207,234,NONE,186,110,50,89] pg 2.d is stuck undersized for 2d, current state active+undersized, last acting [51,NONE,6,140,179,69,26] pg 2.e is stuck undersized for 2d, current state undersized+peered, last acting [NONE,5,112,131,232,NONE,88] pg 2.10 is stuck undersized for 2d, current state active+undersized, last acting [184,233,89,3,135,NONE,56] pg 2.12 is stuck undersized for 2d, current state active+undersized, last acting [35,74,221,67,57,NONE,11] pg 2.13 is stuck undersized for 2d, current state active+undersized, last acting [40,217,11,50,215,144,NONE] pg 2.14 is stuck undersized for 2d, current state undersized+peered, last acting [231,16,NONE,NONE,66,102,44] pg 2.17 is stuck undersized for 2d, current state active+undersized, last acting [105,NONE,49,181,116,71,151] pg 2.1a is stuck undersized for 2d, current state undersized+peered, last acting [193,NONE,113,NONE,NONE,65,147] pg 2.1b is stuck undersized for 2d, current state undersized+peered, last acting [51,98,80,NONE,150,NONE,137] pg 3.d is stuck undersized for 2d, current state undersized+degraded+peered, last acting [NONE,127,86,159,NONE,11,204] pg 3.13 is stuck undersized for 2d, current state active+undersized+degraded, last acting [NONE,23,180,74,177,47,94] pg 3.14 is stuck undersized for 7h, current state undersized+degraded+peered, last acting [NONE,202,49,NONE,209,NONE,9] pg 3.17 is stuck undersized for 2d, current state active+undersized+degraded, last acting [185,178,NONE,159,103,131,77] pg 3.88 is stuck undersized for 2d, current state active+undersized+degraded, last acting [64,158,239,211,85,NONE,96] pg 3.89 is stuck undersized for 2d, current state undersized+degraded+inconsistent+peered, last acting [65,NONE,NONE,103,NONE,220,24] pg 3.8d is stuck undersized for 2d, current state undersized+degraded+peered, last acting [NONE,127,86,159,NONE,11,204] pg 3.93 is stuck undersized for 2d, current state active+undersized+degraded, last acting [NONE,23,180,74,177,47,94] pg 3.94 is stuck undersized for 7h, current state undersized+degraded+peered, last acting [NONE,202,49,NONE,209,NONE,9] pg 3.97 is stuck undersized for 2d, current state active+undersized+degraded+inconsistent, last acting [185,178,NONE,159,103,131,77] pg 3.9c is stuck undersized for 2d, current state active+undersized+degraded, last acting [NONE,182,161,70,218,74,1] pg 3.9f is stuck undersized for 2d, current state active+undersized+degraded, last acting [NONE,209,151,1,89,107,63] pg 4.2 is stuck undersized for 2d, current state active+undersized+degraded, last acting [78,137] pg 4.3 is stuck undersized for 2d, current state active+undersized+degraded, last acting [205,32] pg 4.4 is stuck undersized for 2d, current state active+undersized+degraded, last acting [194,61] pg 4.5 is stuck undersized for 2d, current state active+undersized+degraded, last acting [119,133] pg 4.7 is stuck undersized for 2d, current state active+undersized+degraded, last acting [176,6] pg 4.12 is stuck undersized for 2d, current state active+undersized+degraded, last acting [71,24] pg 4.1c is stuck undersized for 2d, current state active+undersized+degraded, last acting [28,158] pg 5.0 is stuck undersized for 2d, current state undersized+peered, last acting [NONE,114,NONE,235,190,71] pg 5.2 is stuck undersized for 7h, current state active+undersized, last acting [22,123,227,NONE,60,99] pg 5.8 is stuck undersized for 2d, current state active+undersized, last acting [36,179,226,79,NONE,231] pg 5.11 is stuck undersized for 2d, current state active+undersized, last acting [45,226,24,78,NONE,209] pg 5.13 is stuck undersized for 2d, current state active+undersized, last acting [36,167,225,105,211,NONE] pg 5.14 is stuck undersized for 2d, current state active+undersized, last acting [NONE,185,224,149,102,46] pg 5.16 is stuck undersized for 2d, current state active+undersized, last acting [86,53,2,225,NONE,182] pg 5.17 is stuck undersized for 2d, current state active+undersized, last acting [54,NONE,150,80,205,179] pg 5.1c is stuck undersized for 2d, current state active+undersized, last acting [158,29,217,192,NONE,125] pg 5.1e is stuck undersized for 2d, current state active+undersized, last acting [43,215,66,NONE,238,201] pg 5.1f is stuck undersized for 2d, current state active+undersized, last acting [23,151,NONE,127,24,214] pg 6.0 is stuck undersized for 2d, current state undersized+peered, last acting [204] pg 6.5 is stuck undersized for 2d, current state active+undersized, last acting [53,14] pg 6.9 is stuck undersized for 2d, current state active+undersized, last acting [55,32] pg 6.10 is stuck undersized for 2d, current state active+undersized, last acting [126,43] pg 6.15 is stuck undersized for 2d, current state active+undersized, last acting [56,22] pg 6.16 is stuck undersized for 2d, current state active+undersized, last acting [226,231] pg 6.1e is stuck undersized for 2d, current state active+undersized, last acting [33,7] [WRN] RECENT_CRASH: 49 daemons have recently crashed osd.75 crashed on host balin007 at 2024-01-26T18:02:51.822269Z osd.25 crashed on host balin003 at 2024-01-26T18:02:54.555975Z osd.91 crashed on host balin009 at 2024-01-26T18:03:20.795065Z osd.228 crashed on host balin020 at 2024-01-26T18:03:25.567078Z osd.97 crashed on host balin008 at 2024-01-26T18:03:29.630112Z osd.199 crashed on host balin017 at 2024-01-26T18:03:31.709794Z osd.50 crashed on host balin005 at 2024-01-26T18:03:36.854869Z osd.189 crashed on host balin016 at 2024-01-26T18:03:56.659898Z osd.152 crashed on host balin013 at 2024-01-26T18:03:56.760965Z osd.132 crashed on host balin012 at 2024-01-26T18:04:02.652957Z osd.216 crashed on host balin019 at 2024-01-26T18:04:03.739005Z osd.4 crashed on host balin001 at 2024-01-26T18:04:05.681004Z osd.228 crashed on host balin020 at 2024-01-26T18:04:18.907578Z osd.15 crashed on host balin002 at 2024-01-26T18:04:20.899089Z osd.42 crashed on host balin004 at 2024-01-26T18:04:45.356823Z osd.134 crashed on host balin012 at 2024-01-26T18:14:44.142683Z osd.174 crashed on host balin015 at 2024-01-26T18:15:16.090995Z osd.206 crashed on host balin018 at 2024-01-26T18:15:17.381641Z osd.64 crashed on host balin006 at 2024-01-26T18:15:18.413870Z osd.158 crashed on host balin014 at 2024-01-26T18:15:20.332006Z osd.109 crashed on host balin010 at 2024-01-26T18:15:35.502601Z osd.86 crashed on host balin009 at 2024-01-26T18:15:45.271677Z osd.239 crashed on host balin020 at 2024-01-26T18:15:51.809723Z osd.163 crashed on host balin014 at 2024-01-26T18:15:56.247493Z osd.130 crashed on host balin011 at 2024-01-26T18:17:20.070002Z osd.39 crashed on host balin004 at 2024-01-26T18:18:15.486711Z osd.115 crashed on host balin010 at 2024-01-26T18:25:17.161003Z osd.12 crashed on host balin002 at 2024-01-26T18:25:19.453669Z osd.107 crashed on host balin008 at 2024-01-26T18:26:53.564634Z osd.88 crashed on host balin009 at 2024-01-26T18:26:55.732369Z osd.182 crashed on host balin016 at 2024-01-26T18:27:28.585407Z osd.70 crashed on host balin006 at 2024-01-26T18:27:39.749702Z osd.18 crashed on host balin002 at 2024-01-26T18:30:56.147662Z osd.182 crashed on host balin016 at 2024-01-26T18:52:53.081241Z osd.228 crashed on host balin020 at 2024-01-26T18:53:02.240886Z osd.216 crashed on host balin019 at 2024-01-26T18:53:55.596378Z osd.145 crashed on host balin013 at 2024-01-26T18:54:47.742350Z osd.50 crashed on host balin005 at 2024-01-26T18:54:57.261633Z osd.72 crashed on host balin007 at 2024-01-26T18:55:24.723138Z osd.216 crashed on host balin019 at 2024-01-26T18:55:58.005241Z osd.25 crashed on host balin003 at 2024-01-26T18:56:21.006014Z osd.228 crashed on host balin020 at 2024-01-26T18:56:21.054463Z osd.39 crashed on host balin004 at 2024-01-26T19:30:58.332574Z osd.230 crashed on host balin020 at 2024-01-26T19:31:24.437215Z osd.127 crashed on host balin011 at 2024-01-26T19:31:25.436376Z osd.115 crashed on host balin010 at 2024-01-26T19:31:52.431653Z osd.59 crashed on host balin005 at 2024-01-26T19:32:16.527357Z osd.174 crashed on host balin015 at 2024-01-26T19:32:19.548745Z osd.195 crashed on host balin017 at 2024-01-26T19:32:33.554557Z