디지문닷넷 DigiMoon 맘대로 닦고 조이고 기름치는 재미가 있는 DigiMoon만의 기억 저장소

1. 환경 : RHEV 3.6 self-hosted engine


2. RHEV 복구가 필요하게 된 사연 : SAN switch 조작 미스로 engine(Manager)이 설치되어 있던 LUN을 다른 서버에 인식시켜 overwite하여 engine 유실. regular guests가 설치되어 있는 data domain용 LUN은 정상이므로 engine만 복구하면 RHEV를 정상 사용할 수 있다. 평소에 engine-backup 커맨드로 engine을 백업하고 있었다.


3. 방법

Step 1) 실패된 Self-hosted engine 설정을 클리어

How to Clean Up a Failed Self-hosted Engine Deployment?
https://access.redhat.com/solutions/2121581

[root@rhevh01 ~]# systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd
[root@rhevh01 ~]# /bin/rm /etc/ovirt-hosted-engine/hosted-engine.conf
[root@rhevh01 ~]# /bin/rm /etc/ovirt-hosted-engine/answers.conf
[root@rhevh01 ~]# /bin/rm /etc/vdsm/vdsm.conf
[root@rhevh01 ~]# /bin/rm /etc/pki/vdsm/*/*.pem
[root@rhevh01 ~]# /bin/rm /etc/pki/CA/cacert.pem
[root@rhevh01 ~]# /bin/rm /etc/pki/libvirt/*.pem
[root@rhevh01 ~]# /bin/rm /etc/pki/libvirt/private/*.pem


Step 2) engine용 LUN 클리어

[root@rhevh01 ~]# dd if=/dev/zero of=/dev/mapper/<id of lun for engine> bs=1M count=1


Step 3) RHEV 3.6 Self-hosted engine 재설치


Step 4) Host에서 Engine을 global maintenance mode로 변경
[root@rhevh01 ~]# hosted-engine --set-maintenance --mode=global


Step 5) engine DB 삭제하고 빈 DB로 재생성

[root@rhevm ~]# su postgres -c psql

postgres=# SELECT * FROM pg_stat_activity WHERE datname = 'engine';

postgres=# SELECT pg_terminate_backend (pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'engine';
※ procpid 컬럼의 값이 두번째 쿼리문의 pg_stat_activity.pid에 해당한다.

postgres=# DROP DATABASE engine;
※ 프로세스를 죽여도 spwan되서 다시 올라오므로 수 차례 동안 procpid를 재확인하면서 계속 pid를 죽이다 보면 언젠가 DB 삭제가 가능해진다.

postgres=# CREATE DATABASE engine OWNER engine template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';


Step 6) engine-backup 백업본으로 복구

Backing up and restoring RHEV with engine-backup
https://access.redhat.com/solutions/797463

[root@rhevm 20170502-08:36]# engine-backup --mode=restore --scope=all --restore-permissions --file=./20170502-08\:36-rhevm.rhev.osci.kr-engine-backup --log=./20170502-08:36-rhevm.rhev.osci.kr-engine-backup.log
Preparing to restore:
- Unpacking file './20170502-08:36-rhevm.rhev.osci.kr-engine-backup'
Restoring:
- Files
FATAL: Can't connect to database 'engine'. Please see '/usr/bin/engine-backup --help'.
[root@rhevm 20170502-08:36]#

engine DB를 restore 실패한다.
위까지 진행하면 engine에서 백업 당시의 engine DB의 engine 유저의 패스워드 정보가 담긴 /etc/ovirt-engine/engine.conf.d/10-setup-database.conf 를 확인할 수 있다. 이 파일 참고하여 engine DB의 engine 유저의 패스워드를 변경한다.

[root@rhevm ~]# cat /etc/ovirt-engine/engine.conf.d/10-setup-database.conf
ENGINE_DB_HOST="localhost"
ENGINE_DB_PORT="5432"
ENGINE_DB_USER="engine"
ENGINE_DB_PASSWORD="z0TGLgiF8dNjfSboZVqYj0"
ENGINE_DB_DATABASE="engine"
ENGINE_DB_SECURED="False"
ENGINE_DB_SECURED_VALIDATION="False"
ENGINE_DB_DRIVER="org.postgresql.Driver"
ENGINE_DB_URL="jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory"
[root@rhevm ~]#

[root@rhevm ~]# su postgres -c psql
psql (8.4.20)
Type "help" for help.

postgres=#  ALTER USER engine WITH PASSWORD 'z0TGLgiF8dNjfSboZVqYj0';


이제 다시 engine-backup --mode=restore 실행하면 잘 restore될 것이다.
restore되었으면 engine-setup하여 마무리할 차례이다.


Step 7) engine-setup 실행

engine의 repo를 아래와 같이 설정
[root@rhevm ~]# cat /etc/yum.repos.d/ftp.repo
[rhel-x86_64-server-6]
name=rhel-x86_64-server-6
baseurl=ftp://192.168.0.232/pub/rhn/rhel6/repo/rhel-x86_64-server-6/getPackage
enabled=1
gpgcheck=0

[rhel-x86_64-server-supplementary-6]
name=rhel-x86_64-server-supplementary-6
baseurl=ftp://192.168.0.232/pub/rhn/rhel6/repo/rhel-x86_64-server-supplementary-6/getPackage
enabled=1
gpgcheck=0

[rhel-6-server-rhevm-3.6-rpms]
name=rhel-6-server-rhevm-3.6-rpms
baseurl=ftp://192.168.0.232/pub/rhn/rhel6/repo/rhel-6-server-rhevm-3.6-rpms/Packages
enabled=1
gpgcheck=0

[jb-eap-6-for-rhel-6-server-rpms]
name=jb-eap-6-for-rhel-6-server-rpms
baseurl=ftp://192.168.0.232/pub/rhn/rhel6/repo/jb-eap-6-for-rhel-6-server-rpms/Packages
enabled=1
gpgcheck=0


engine의 ovirt-engine 서비스 중지
[root@rhevm ~]# service ovirt-engine stop
※ 이 때 Host에서는 engine이 global maintenance mode여야 한다. 안 그러면 Host의 ha서비스(ovirt-ha-agent, ovirt-ha-broker)가 engine VM을 shutdown한다.


engine-setup 실행

[root@rhevm ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf']
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170725080141-u9kclg.log
          Version: otopi-1.4.2 (otopi-1.4.2-1.el6ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Stage: Environment customization

          Welcome to the RHEV 3.6 setup/upgrade.
          Please read the RHEV 3.6 install guide
          https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Installation_Guide/index.html.
          Please refer to the RHEV Upgrade Helper application
          https://access.redhat.com/labs/rhevupgradehelper/ which will guide you in the upgrading process.
          Would you like to proceed? (Yes, No) [Yes]:

          --== PRODUCT OPTIONS ==--


          --== PACKAGES ==--

[ INFO  ] Checking for product updates...
[ INFO  ] No product updates found

          --== ALL IN ONE CONFIGURATION ==--


          --== NETWORK CONFIGURATION ==--

          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]:
[ INFO  ] iptables will be configured as firewall manager.

          --== DATABASE CONFIGURATION ==--


          --== OVIRT ENGINE CONFIGURATION ==--


          --== STORAGE CONFIGURATION ==--


          --== PKI CONFIGURATION ==--


          --== APACHE CONFIGURATION ==--


          --== SYSTEM CONFIGURATION ==--


          --== MISC CONFIGURATION ==--


          --== END OF CONFIGURATION ==--

[ INFO  ] Stage: Setup validation
[ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data.
[ ERROR ] Failed to execute stage 'Setup validation': Hosted Engine setup detected, but Global Maintenance is not set.
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170725080141-u9kclg.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170725080242-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed
[root@rhevm yum.repos.d]#

 실패하는 이유는 engine을 global maintenance mode로 변경하면 postgres DB에도 정보가 박히는데 global maintenance mode가 아닌 상태에서 백업받은 파일로 복구했기 때문이다. 간단하게 update 쿼리문으로 ha_global_maintenance 컬럼의 'f' 값을 't'값으로 변경하고 engine-setup 돌리면 해결된다. 향후 engine-backup 백업 시 global maintenance mode로 전환한 뒤 백업하면 복구할 때 금번과 같은 수고를 덜 수 있다.

[root@rhevm ~]# su postgres -c psql
psql (8.4.20)
Type "help" for help.

postgres=# \connect engine
psql (8.4.20)
You are now connected to database "engine".
engine=# SELECT vm_guid, run_on_vds FROM vms WHERE vm_name ='HostedEngine';
vm_guid | run_on_vds
--------------------------------------+--------------------------------------
17c7738b-2394-48b0-a1e1-3afaa0545a7f | 79fee788-05da-40c8-8afe-d81574715c87
(1 row)

engine=# SELECT vds_id, ha_global_maintenance FROM vds_statistics WHERE vds_id = '79fee788-05da-40c8-8afe-d81574715c87';
vds_id | ha_global_maintenance
--------------------------------------+-----------------------
79fee788-05da-40c8-8afe-d81574715c87 | f
(1 row)

engine=# UPDATE vds_statistics set ha_global_maintenance = 't' WHERE vds_id = '79fee788-05da-40c8-8afe-d81574715c87';


다시 engine-setup 실행하면 성공할 것이다.




Step 8) Manager dash board에서 Host를 activate

여기까지 진행하였어도 Host와 engine이 서로 sync되지 않아 Host가 activate(regular guest를 실행할 수 있는 Host 상태)되지 않는다. PostgreSQL을 약간 수정(일종의 꼼수)하여 Manager 대시보드 상에서 Host를 reinstall하는 과정을 거치면 해결된다.

Manager 상에서 Host를 reinstall 하려면 Host 모드가 maintenance mode여야 하고, guest가 0개 떠 있고, 마이그레이션 중인 guest도 0개가 되어야 한다. 이 조건을 만족하도록 DB를 수정해 보자.

engine=# select vds_id from vds_static where vds_name='rhevh01.rhev.osci.kr';
                vds_id
--------------------------------------
 79fee788-05da-40c8-8afe-d81574715c87
(1 row)

engine=# update vds_dynamic set status=2, vm_active=0, vm_count=0, vm_migrating=0 where vds_id='79fee788-05da-40c8-8afe-d81574715c87';
UPDATE 1
engine=#

아제 Manager 대시보드의 Host 메뉴를 보면 하단 서브 창에 reinstall 메뉴가 활성화될 것이다. 재설치하여 RHEV 복구 작업을 마무리한다.




부록


※ Guest의 status를 Down으로 변경하는 쿼리문

engine=# SELECT vm_guid, run_on_vds FROM vms WHERE vm_name ='HostedEngine';
vm_guid | run_on_vds
--------------------------------------+--------------------------------------
17c7738b-2394-48b0-a1e1-3afaa0545a7f | 79fee788-05da-40c8-8afe-d81574715c87
(1 row)

engine=# UPDATE vm_dynamic SET status=0 WHERE vm_guid='17c7738b-2394-48b0-a1e1-3afaa0545a7f';



※ LDAP extension을 Manager에 구성

LDAP을 Manager에 연동했었다면, IPA client를 Manager에 설치하고 구성
[root@rhevm ~]# yum install ipa-client ipa-admintools
[root@rhevm ~]# ipa-client-install
Discovery was successful!
Hostname: rhevm.rhev.osci.kr
Realm: RHEV.OSCI.KR
DNS Domain: rhev.osci.kr
IPA Server: ipaserver.rhev.osci.kr
BaseDN: dc=rhev,dc=osci,dc=kr

Continue to configure the system with these values? [no]: yes
User authorized to enroll computers: admin
Synchronizing time with KDC...
Password for admin@RHEV.OSCI.KR:
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=RHEV.OSCI.KR
    Issuer:      CN=Certificate Authority,O=RHEV.OSCI.KR
    Valid From:  Sat Dec 31 04:38:09 2016 UTC
    Valid Until: Wed Dec 31 04:38:09 2036 UTC

Enrolled in IPA realm RHEV.OSCI.KR
Attempting to get host TGT...
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm RHEV.OSCI.KR
trying https://ipaserver.rhev.osci.kr/ipa/xml
Forwarding 'env' to server u'https://ipaserver.rhev.osci.kr/ipa/xml'
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Forwarding 'host_mod' to server u'https://ipaserver.rhev.osci.kr/ipa/xml'
Could not update DNS SSHFP records.
SSSD enabled
Configuring rhev.osci.kr as NIS domain
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Client configuration complete.
[root@rhevm ~]# kinit admin
Password for admin@RHEV.OSCI.KR:
[root@rhevm ~]# kinit admin
(IPA server의 admin 패스워드 입력)

Manager용 LDAP extension을 Manager에 설치 및 셋업
[root@rhevm ~]# yum install ovirt-engine-extension-aaa-ldap-setup


ovirt-engine 서비스 재시작
[root@rhevm ~]# service ovirt-engine restart

이 저작물은 크리에이티브 커먼즈 코리아 저작자표시-비영리-변경금지 2.0 대한민국 라이센스에 따라 이용하실 수 있습니다.