diff options
Diffstat (limited to 'logs/2025.11.03.storage-log.txt')
| -rw-r--r-- | logs/2025.11.03.storage-log.txt | 140 |
1 files changed, 140 insertions, 0 deletions
diff --git a/logs/2025.11.03.storage-log.txt b/logs/2025.11.03.storage-log.txt new file mode 100644 index 0000000..a95dc57 --- /dev/null +++ b/logs/2025.11.03.storage-log.txt @@ -0,0 +1,140 @@ +Status +====== + - Raid controller failed on ipekatrin1 + - The system was not running stable after replacement (disk disconnect after 20-30m operation) + - ipekatrin1 was temporarily converted in the master-only node (apps scheduling disabled, glusterfs stopped) + - Heketi and gluster-blockd were disabled and will be not available further. Existing heketi volumes preserved. + - New disks (from ipekatrinbackupserv1) were assembled in the RAID, assembled in gluster, and manual (file walk-trough) healing + is executed. Expected to take about 2-3 weeks (about 2TB per day rate). No LVM configured, direct mount. + - Application node will be recovered once we replace system SSDs with larger ones (as there currently no space for images/containers) + and I don't want to put it on new RAID. + +Recovery Logs +==== + 2025.09.28 + - ipekatrin1: + * Raid controller don't see 10 disks and behaves erratically. + * Turned of the server and ordered a replacement. + - Sotrage: + * Restarted degraded GlusterFS nodes and make them work on remaining 2 nodes (1 replica + metadata for most of our storage needs). + * Turned out 'database' volume is created in Raid-0 mode and it used backend for KDB database. So, data is gone. + * Recovered KDB database from backups and moved it to glusterfs/openshift volume. Nothing left on 'database' volume. Can be turned off. + + 2025.10.23 + - ipekatrin1: + * Replaced RAID controller. Make attempt to rebuild, but disks are disconnected after about 30-40 minutes (recovered after shutoff, not reboot) + * Checked power issues: cabling bypassing PSU and monitoring voltages (12V system should not go bellow 11.9V). No change, voltages seemed fine. + * Checked cabling issues disconnecting first one cable and then another (supported mode, single cable connects all disks). No change + * Tried to imrpove cooling, setting fan speeds to maximum (kept) and even temporarily installing external cooler. Radiators were cool, also checked reported temperatures. No change, still goes down in 30-40 minutes. + * Suspect backplane problems. The radiators were quite hot before adjusting cooling. Seems known stability problems due to bad signal management in firmware if overheated. Firmware updates are suggest to stabilize. + * No support by SuperMicro. Queried Tootlec about possibility of getting firmware update or/and ordering backplane [Order RG_014523_001_Chilingaryan form 16.12.2016, Angebot 14.10, Contract: 28.11] + Hardware: Chassis CSE-846BE2C-R1K28B, Backplan BPN-SAS3-846EL2), 2x MCX353A-FCB ConnectX-3 VPI + * KATRINBackupServ1 (3-years older) has backplane with enough bays to mount disks. We still need to be able to put Raid-card and Mellanox ConnectX-3 board/boards with 2 ports (can leave with 1). + - ipekatrin2: Noticed and cleared RAID alarm attributed to the battery subsystem. + * No apparent problems at the moment. Temperatures are all in order. Battery reports healthy. Systems works as usual. + * Setup temperature monitoring of RAID card, currently 76-77C + + 2025.10.27 + - ipekatrin1: + * Disconnected all disks from the server and start preparing it as an application node + - Software: + * I have temporarily suspended all ADEI cronJobs to avoid resource contention on ipekatrin2 (as restart would be dangerous now) [clean (logs,etc.)/maintain (re-caching,etc.)/update(detecting new databases)] + - Research: + * DaemonSet/GlusterFS selects nodes based on the following nodeSelector + $ oc -n glusterfs get ds glusterfs-storage -o yaml | grep -B 5 -A 5 nodeSelector + nodeSelector: + glusterfs: storage-host + All nodes has corresponding labels in their metadata: + $ oc get node/ipekatrin1.ipe.kit.edu --show-labels -o yaml | grep -A 20 labels: + labels: + ... + glusterfs: storage-host + ... + * Thats removed now from ipekatrin1 and should be recovered if we bring storage back + oc label --dry-run node/ipekatrin1.ipe.kit.edu glusterfs- + * We further need to remove 192.168.12.1 from 'endpoints/gfs' (per namespaces) to avoid possible problems. + * On ipekatrin1, /etc/fstab glusterfs mounts should be changed from 'localhost' to some other server (or commented all-together). GlusterFS mounts + should be changed from localhost to (or probably just 12.2 as it only host containing data and going via intermediary makes no sense) + 192.168.12.2,192.168.12.3:<vol> /mnt/vol glusterfs defaults,_netdev 0 0 + * All raid volumes be also temporarily commented in /etc/fstab and systemd + systemctl list-units --type=mount | grep gluster + * Further configuration changes required to run node without glusterfs causing no damage to the rest of the system + GlusterFS might be referenced via: /etc/hosts, /etc/fstab, /etc/systemd/system/*.mount /etc/auto.*, scripts/cron + endpoints (per namespace), inline gluster volumes in PV (gloabl), + gluster-block endpoints / tcmu gateway list, sc (heketi storageclass) and controllers (ds,deploy,sts); just in case check heketi cm/secrets), + - Plan: + * Prepare application node [double-check before implementing] + + Adjust node label + + Edit 'gfs' endpoints in all namespaces. + + Check glusterblock/heketi, strange pv's. + + Check Ands monitoring & maintenance scirpts + + Adjust /etc/fstab and check systemd based mounts. Shall we do soemth with hosts? + + /etc/nfs-ganesha on ipekatrin1 & ipekatrin2 + + Check/change cron & monitoring scipts + + Check for backup scripts, it probably written on raid controller. + + Grep in OpenShift configs (and /etc globally) just in case + + Google above other possible culprits. + + Boot ipekatrin1 and check that all is fine + * cronJobs + > Set affinity to ipekatrin1. + > Restart cronJobs (maybe reduce intervals) + * copy cluster backups out + * ToDo + > Ideally eliminating cronJobs all together for rest of KaaS1 life-time and replacing with continuously running cron daemon iside container + > Rebuild ipekatrinbackupserv1 as new gluster node (using disks) and try connecting it to the cluster + + 2025.10.28-31 + - Hardware + * Re-assemled ipekatrin1 disks in ipekatrinbackupserv1 backplane using new LSI 9361-8i raid controller. Original LSI 9271-8i removed. + * Put old (SAS2) disks from ipekatrinbackupserv1 into ipekatrin1. Imported RAID configs, RAID started and seems works stable using SAS2 setup. + - Software + * Removed glusterfs & fat_storage labels from ipekatrin1.ipe.kit.edu node + oc label node/ipekatrin1.ipe.kit.edu glusterfs- + oc label node/ipekatrin1.ipe.kit.edu fat_storage- + * Indentified all endpoints used in PVs (no PV specifies IPs directly). No PV hardcode IPs directly (and it seems unsupported anyway) + Editied endpoints: gfs glusterfs-dynamic-etcd glusterfs-dynamic-metrics-cassandra-1 glusterfs-dynamic-mongodb glusterfs-dynamic-registry-claim glusterfs-dynamic-sharelatex-docker + * Verified that no glusterblock devices is used by pods or outside (no iscsi devics). Checked that heketi storageClass can be safely disabled without affecting existing volumes + Teminated heketi/glusterblock services, removed storageclasses + * Checked ands-distributed scripts & crons. No referring to gluster. Monitoring checks raid status, but this probably is not critical as it would just report error (which is true) + * Set nfsganesha cluster nodes to andstorage2 only on ipekatrin1/2 (no active server on ipekatrin3). Service is inactive at the moment + Anyway double-check to disable on ipekatrin1 on a first boot + * Found active 'block' volume in glusterfs. Checked it is empty and is not used by any active 'pv'. Stopped and deleted. + * Backup is done on /mnt/provision which should work in new configuration. So, no changes are needed. + * Mount points adjusted. + - First Boot: + * Disable nfs-ganesha on first boot on ipekatrin1 + * Verified that glusterfs is not started and gluster mounts are healthy + * etcd is running and seem healthy + ETCDCTL_API=3 /usr/bin/etcdctl --cert /etc/etcd/peer.crt --key /etc/etcd/peer.key --cacert /etc/etcd/ca.crt --endpoints https://`hostname`:2379 member list + curl -v --cert /etc/etcd/peer.crt --key /etc/etcd/peer.key --cacert /etc/etcd/ca.crt -s https://192.168.13.1:2379/v2/stats/self + * origin-master-api and origin-master-controllers are runnign + * origin-node and docker failed. /var/lib/docker is on the raid (mounted /var/lib/docker, but used via lvm thin pool). + * Created '/var/lib/docker-local for now and configured docker to user overlay2 in /etc/sysconfig/docker-storage + DOCKER_STORAGE_OPTIONS="--storage-driver=overlay2 --graph=/var/lib/docker-local" + * Adjusted selinux contexts + semanage fcontext -a -e /var/lib/docker /var/lib/docker-local + restorecon -R -v /var/lib/docker-local + * Infrastructure pods are running on ipekatrin1 + * Check Status and monitoring scripts are working [ seems reasonable to me ] + > Raid is not optimal and low data space is report (/mnt/ands is not mounted) + > Docker is not reporting available Data/Metadata space (as we are on local folder) + * Check /var/lib/docker-local space usage is monitored + > Via data space usage + - Problems + * We have '*-host' pvs bound to /mnt/hostdisk which are used adei/mysql (nodes 2&3) and as katrin temporary data folder. Currently keep node1 as master, but disable scheduling + oc adm cordon ipekatrin1.ipe.kit.edu + - Backup + * Backups from 'provision' volume are taken to 'kaas-manager' VM + - Monitor + * Usage in /var/lib/docker-local [ space usage ] + - ToDo + * Try building storage RAID in ipekatrinbackupserv1 (SFF-8643 to SFF-8087 cable needed, RAID-to-backplane). Turn on, check data is accessible and turn-off. + * We shall order larger SSD for docker (LVM) and KATRIN temporary files (/mnt/hostraid). Once done, uncordon jobs on katrin2 + oc adm uncordon ipekatrin1.ipe.kit.edu + * We might try building a smaller RAID from stable disk bays and move ADEI replica here (discuss!) or a larger from SAS2 drives if it proves more stable. + * We might be able to use Intel RES2SV240 or LSISAS2x28 expander board to reduce SAS3 to SAS2 speeds... + + 2025.11.01-03 + - Document attempts to recover storage raid + - GlusterFS changes and replication + |