Implantação #870

Script para validar saúde do WN

Added by Gabriel Winckler over 11 years ago. Updated over 11 years ago.

Status:ResolvidaStart date:10 Apr 2015
Priority:NormalDue date:
Assignee:Gabriel Winckler% Done:

0%

Category:-
Target version:-

Description

Criar um script de epilogo para verificar a saúde do WN e evitar que nós com disco cheio ou problemas de disco drenem a fila toda.

History

#1 Updated by Gabriel Winckler over 11 years ago

  • Status changed from Execução to Resolvida

Função implementada usando a opção HealthCheck:

HealthCheckProgram=/etc/slurm/slurm_health_check.sh
HealthCheckInterval=300

/etc/slurm/slurm_health_check.sh

#!/bin/bash

NODE=$(hostname -s)

action() {
  scontrol update NodeName=$NODE State=drain Reason="$1" 
  echo "$1 on $NODE" | mail -s "Health check fail on $NODE" root
  exit 1
}

# Test if /tmp is mounted
grep " /tmp ext3" /proc/mounts &> /dev/null || action "/tmp not mounted" 

# Test if /tmp is writeble
touch /tmp/.health_check &> /dev/null || action "/tmp not writeble" 

# Test if /tmp has enought free space
USED=$(df /tmp | grep -v "Filesystem" | awk '{print $5}' | cut -f1 -d'%')
if [ $USED -gt 90 ]
then
  action "Not enoght space on /tmp" 
fi

Also available in: Atom PDF