Move logical partition using “migratelp” command to preserve cross site mirroring

 

Lets first understand what is cross site mirroring and why it necessary??

Answer: In cross site mirroring disks are coming from two different site. If on AIX LPAR there is logical volume “backuplv” which having 2 disk hdisk0 and hdisk1. Because of some disaster at “Disk-Mirror site 1” we lost hdisk0 availability, then we can access same data which is present on hdisk0 from other site disk “hdisk1”.

 

Disk-Mirror site 1- hdisk0, hdisk2

Disk-Mirro-site 2- hdisk1

Each site have unique identifier. AIX admin can identify Lun/disk from correct site by engaging storage admin and providing them your requirement that you need disk from cross site for file system.

 

Generally production environment have cross site mirroring, for development and test environment filesystem have only single copy of logical partition.

 

Here I am going to discuss 1 scenario where, we need had issue for “cross-site” mirroring. This incident was occurred because of admin mistake.

 

Issue description:

“backuplv”  data copy 1 and data copy 2 pointing to same site disk. To solve this issue admin need to use “migratelp” .

 

LV name : backuplv

File system name : /backup

PV- hdisk0 hdisk1 hdisk 2 

VG – backupvg

#lslv  -m  backuplv

LP    PP1      PV1               PP2  PV2               PP3  PV3

0001   0001    hdisk0          0010   hdisk1

0002   0002    hdisk0          0011   hdisk1         

0003   0003    hdisk0          0012   hdisk2                

0004   0004    hdisk0          0013   hdisk2

 

From above output we can observe that, LP 03 and 04 data copy 1 and 2 belongs to Disk-Mirror site 1- hdisk0, hdisk2. Here we need to move these LP 2nd copy to hdisk1.

This will preserve cross site mirroring. AIX admin can move LP 03 and 04  2nd copy to free PP's on hdisk1 using following command.

 

#Migratelp backuplv/03/2 hdisk1/12  

-        Move 2^nd mirror copy of LP 03 of logical volume backuplv to hdisk1 PP 12.

#Migratelp backuplv/04/2 hdisk1/13  

-        Move 2^nd mirror copy of LP 04 of logical volume backuplv to hdisk1 PP 13.

 

After migratelp output of #lslv -m backuplv

 

Output:

 

#lslv  -m  backuplv

LP    PP1      PV1               PP2  PV2               PP3  PV3

0001   0001    hdisk0          0010   hdisk1

0002   0002    hdisk0     0011   hdisk1                 

0003   0003    hdisk0     0012   hdisk1                          

0004   0004    hdisk0     0013   hdisk1

 

Here we have moved 2^nd copy of backuplv to correct site, to preserve cross site mirroring.

 

So if in future any issue while accessing Disk-Mirror site 1, then same data will be accessible from other site disk hdisk1. If cross site mirroring not preserved in production environment it will cause application and database outage, if that file system having critical data on it.

 

 

 

 

Why PP from same disk mirror site got allocated to “backuplv”

 

Answer:

 

Some time admin just use “chfs” command to increase file system size, without extending LP from Disk-Mirror site 1 and Disk-Mirror site 2.

 

 

 

 

Good practice :

 

Identify disk from Disk-Mirror site 1 and Disk-Mirror site 2

Select those disk (hdisk0 and hdisk1)

Extend desired LP for backuplv

Increase file system size.

 

 

 

Thanks  !!!!

 

 

 

 

 

 

 

 

 

 

 

 

                   

 

How to use screen command in Linux

--> To open a new screen:

#screen -S "Screenname"
Eg.: 

#screen -S "testscreen"

--> To deattach the screen:

Cntrl+A then D   //it will run after detach also
--->    Ctrl-a followed by k: close the current windows (kill)

--> To list the screen:

screen -list

--> To reattach the screen:

screen -r ScreenID------- From screen -list



Example:

# screen -S "testscreen"   // create screen name testscreen
Agent pid 21981

Now we are running small test script in "testscreen".

#/test.sh >>data.txt &  //ran script test.sh to append hello and date/time in data.txt
[1] 4917

Cntrl+A then D   //it will detach

[detached]
# screen -list   //list currently running screen
There is a screen on:
        3015.testscreen (Detached)
1 Socket in /var/run/uscreens/S-alhatul.

Reattch screen using following command
screen -r ScreenID
#screen -r 3015   //reattach screen
Agent pid 21981

output of screen command

Output of test.sh
===data.txt ======

#cat data.txt
hello
Fri Aug 30 15:21:00 CEST 2019
hello
Fri Aug 30 15:21:02 CEST 2019
hello
Fri Aug 30 15:21:04 CEST 2019
hello

Ctrl-a followed by k:
It will terminate current screen session.

#screen -list
No Sockets found in /var/run/uscreens/S-alhatul.

Reduce XFS filesystem on Linux

Hello friend ,

today i am going to discuss on XFS file system Limitation and that limitation is Linux admin cant reduce XFS file system online. after knowing this limitation i am still thinking about what is that preventing XFS shrink ? also is there any alternative way we can reduce size of XFS file system.

Yes, we can reduce size of XFS , but for this downtime needed.

Step 1: Backup existing file system backup using "xfsdump" 

Step 2: Then umount XFS file system.

Step 3: Then remove LV .

Step 4: Create required size LV.

Step 5: Mount File system.

But here Linux admin require downtime.

.........

Copy pedbg zip file using SCP from HMC to NIM server.

How to copy pedbg file after pedbg file created on HMC

Pedbg collected using "hscpe" user.

#pedbg -r 

Above command will remove previous data

Now collect fresh pedbg logs using following command

#pedbg -q 4 -c

This will generate file in /dump with name like /dump/hmclogs.hmcabc.zip

Copy thes to remote NIM or AIX Lpar using SCP command.

Login to NIM , create folder in /TMP like "pedbglogs"

#cd /tmp

#mkdir pedbglogs

#cd pedbglogs

# scp hscpe@hmcIP:/dump/hmclogs.hmcabc.zip  .

Here . Means current directory on NIM server. (/tmp/pedbglogs)

** Make sure /tmp have enough space to hold this pedbg file.

Thanks !!!!!

AIX lpar not connecting ,ssh process issue troubleshoot

Got incident for AIX LPAR not connecting from outside using ssh.

To solve this issue followed following approach.

First did pre-check like tried to connect from AIX NIM server and CyberArk, so found that its not connecting from both NIM and CyberArk.

so finally decided to try from putty, that also failed.

so, decided to take console from HMC and took session. upon checking found that "ssh" process is in inoperative state. so tried to start that process but its not starting using following command. :(

#startstc -s sshd 

#refresh -s sshd

so decided to check which process is holding that port 22. checked this using lsof command

#lsof -i :22

this command showing process id where socket status is "closed_wait",

so decided that will go for kill that process.

killed that process by command #kill -9 PID

and started "sshd" process and finally it work.

AIX lpar we able access from cyberARK ,putty and NIM.

Finally we conclude that, ssh process was in "hung" state and sshd process refresh and stop, start also not worked, then finally after clearing PID associated with that process solve issue.

Thanks !!!!

AIX HACMP ,GPFS and veritas Cluster state check command

 AIX HACMP ,GPFS and veritas Cluster state check command

AIX HACMP resource group state check command

#clRGinfo

How to check cluster is in  stable state or not

#lssrc -ls clstrmgrES |grep -i state

if output is showing "ST_STABLE" ,then cluster is in stable state .

#lssrc -g cluster   // Show process status in cluster group

GPFS cluster information check command

#mmgetstate -aLs

following command will show information about gpfs cluster

#mmlscluster

Veritas Cluster state check command

#hastatus -summ

Thanks !!!!!

How to replace faulty disk on AIX VIO rootvg using DIAG HOT plug task

 How to replace faulty disk on AIX VIO rootvg using DIAG HOT plug task??

AIX admin can replace faulty hard drive using

"DIAG" procedure. here faulty disk must support 

hot swap operation.

Before replacement we need to do some pre-check,

1. IS this part of rootvg and its in mirror ?

Ans : if yes then ,identify that disk and unmirror 

that disk from rootvg using following command.

#umirrovg rootvg hdisk1

#reducevg rootvg hdisk1

2. Identify disk location using lscfg command

#lscfg -vpl hdisk1

3. make sure that disk is removed from rootvg.

Once all precheck done we can identify hdisk1 by following procdure,

PART I

DIAG--->Certify media Task---->Hot plug task--->scsi and scsi RAID hot plug RAID manager----->Identify disk

Select hdisk1 and make sure location and disk is current and press enter key.

Example . 

XXX.XXX.XX.P2-C9-D5    //disk location

once "Enter key pressed" LED will blink, which set disk to identify mode.

once disk identified, you can exit from this menu by pressing "enter" and come to previous menu by "ESC +3"

PART II

Next is Remove and replace hdisk1

===============================

DIAG--->Certify media Task---->Hot plug task--->scsi and scsi RAID hot plug RAID manager----->Identify disk-->Replace / Remove a Device Attached to An SCSI Hot Swap Enclosure Device

select hdisk1 and make sure location and disk is current and press enter key.

XXX.XXX.XX.P2-C9-D5 

now disk is ready for remove and replace.

now you can ask remote engineer to perform disk replacement.

PART III

Once disk replaced you need to detect that disk on AIX LPAR.

follow below procedure it will configure newly replaced disk on LPAR.

DIAG--->Certify media Task---->Hot plug task--->scsi and scsi RAID hot plug RAID manager-->Configure Added/Replaced Devices

identify new disk by lspv and lscfg command 

#lscfg -vpl hdisk    // it will have new serial number.

Thanks !!!!