Permission denied error while executing Perl script from remote jump server

While performing OS admin task, one of the application guy came to me ,was complaining about permission issue on one of the Linux server.

Error : "permission denied "to user while login from remote user.

after seeing this error first thought in mind was may be user account is locked, but here login method was password less and also i checked whether user account is locked or not.

#passwd -S username

user account wasn't locked

so what's next..........

After i checked /etc/passwd file and checked whether user is exist or not and found that user is not present in  /etc/passwd file ,so thought that this is the issue. But application guy was saying that he was able login using same command few days back. Now i  really confused, what is the issue and what denying access to user on Linux server. Again i checked which command user was executing and found that he was executing Perl script with some parameter, which was trying to login on Linux server to application user account. Perl script was trying to login to app user account .

again i thought that let check log in /var/log file ,so checked messages but not found anything, then i turned to /var/log/authlog file and asked app guy to execute that command again and kept authlog file in monitoring mode by executing following command and here i found some hint from error "Failed publickey"  that error is in authorized_keys file.

tail -f /var/log/authlog

Entry was like below :

Failed publickey for "appuser" from "XX.XX.XX.XXX"(IP address).

after reading this messages i decided to check authorized_keys  file .

I executed below sequence of command 

#su - appuser

#cat /apphome/.ssh/authorized_keys

Now here i get messages that "permission denied". I am not understanding why authorized_keys file in appuser home directory denying permission to owner itself. After that checked permission of authorized_keys file and found that file ownership is not correct ,owner of file is different, somebody had changed ownership to other user.

After proper change request i done ownership change and asked app team guy to check again?

Here finally user able to login on remote server.

from above error message, we conclude that first understand problem, then do proper analysis, like check logs on server and try to extract some indication from that, sometime appteam guy also dont know what is the issue, in this incident app guy was asking me to create new user and here this wasn't correct.

For above type of error please check following point.

1. Check log files which are related to error like, authlog,messages.

2. Find indication from that log file.

3. Check permission and ownership of authorized_keys,this file permission is always 600 and file is not world writable.

4. Check permission and ownership of home directory.

5. Also check required user exist or not.

6. Correct public key in authorized_keys.

Thanks!!!!!

Virtual Media Library on Virtual I/O server

In this post i am going to share how to create ISO image from AIX mksysb and restore to AIX LPAR

using Virtual media library from VIO. its also called as media repository .media repository contain ISO image.

Let see how to create ISO image from AIX mksysb.
mkcd -L -S -I /mksysb/AIX_ISO -m /mksysb/AIX_mksysb_bkp

-L create ISO image
-I Path to store ISO image
-m Previously created mksysb image
-S stop mkcd command from writing ISO image on CD/DVD.
a
once ISO image created next task is to make sure that following condition satisfied.
1.Version of VIOS 1.5 or Later.
2.Virtual SCSI adapter mapping between VIO server and AIX LPAR.
3.Healthy ISO image .here we already created (AIX_ISO).

so Lets see step by step process how to create media Library and map it to AIX LPAR.

1.First check that whether any media repository exist or not by following command.
$lsrep
if repository exist command output will show its size ,if not exist then proceed for step 2.
2.Create media repository using command "mkrep"

$mkrep -sp rootvg -size 10G
confirm that media repository is created by executing following command. 
$lsrep
Size(mb) Free(mb) Parent Pool Parent Size Parent Free
10198    10198    rootvg     139776     22528

If AIX admin want to store multiple AIX ISO then he might need to increase the size of media repository.command for increasing media repository is 

$chrep -size 6G

3.Create Virtual media disk by executing following command .
#mkvopt -name AIX_71_TL04_base_image -file /tmp/AIX_ISO

here we copied already created "AIX_ISO" /tmp on VIO server.

confirm that virtual media disk is created or not by executing command 
#lsrep

padmin$ lsrep
 Size(mb) Free(mb) Parent Pool Parent Size Parent Free
10198    10198    rootvg     139776     22528
 Name                                    File Size  Optical         Access

 AIX_ISO                                  3679    None             rw

4.Connect this Virtual optical media device to AIX LPAR using following command.

$mkvdev -fbo  -vadapter vhost6
vtopt0 Available 

vhost6 is virtual SCSI adapter connected to AIX LPAR.

5.Load ISO to virtual drive by executing command loadopt.

$ loadopt -disk AIX_ISO -vtd vtopt0

check mapping on VIO server by command lsmap

$lsmap -vadapter vhost6

$ lsmap -vadapter vhost3

6.Once ISO is loaded on Virtual drive next step is to boot AIX LPAR in SMS mode .

in SMS mode virtual drive will be same as CD/DVD

 drive,choose correct adapter and boot from CD/DVD which holding AIX_ISO.

7.once installation done unload image from drive by command unloadopt.

$unloadopt -release -vtd vtopt0

please share  your comment.

thanks

ASM disk error synchronous I/O operation to a disk failed oracle RAC on AIX.

Oracle RAC cluster on AIX 6.1 TL08 : Error while restoring RMAN backup on ASM disk.

ORA-15080: synchronous I/O operation to a disk failed

While restoring oracle RMAN backup, we faced error "synchronous I/O operation to a disk failed".

After checking at database end they confirmed that backup is ok and everything is fine from their end,but we thought that why should we take oracle RMAN backup one more time and try with new fresh backup.

we took backup once again and tried to restore on ASM oracle RAC setup.but no success :(.

So its OS team turn now,after doing analysis on OS end.found that ASM disk reserve policy is not correctly set and this is the main reason for  error "synchronous I/O operation to a disk failed".

so we changed ASM disk reserver_lock policy by using following command.

on server ASM disk was from EMC storage.

command : 

#chdev –l hdiskpowerX –a reserve_lock=no 

check attributes after change using command.

# lsattr –El hdiskpowerX |grep -i reserve_lock

After doing resreve_lock policy database team tried backup restore on oracle RAC and guess what this time its work perfect!!!!!!!!.

So error was at OS end it’s because of not done proper pre-requisite check.

Pre-requisite for oracle ASM disk

==========================

1. All ASM disk reserve_lock must be set to "no"

check attribute using commnad # lsattr –El hdiskpowerX.

2. check and confirm that all ASM disk having correct ownership set.

If ownership is not correctly set then at oracle RAC end they will not able to detect ASM disk

also i like to share how to assign RAW/ASM disk to Oracle RAC cluster ,there are some simple steps.

1.Ask storage guy to assign disk from storage to AIX LPAR.
2.detect that disk on AIX LPAR by executing "cfgmgr" command.

3.After detecting disk don't assign any PVID to DISK and don't create any file system on it.
4.simply change reserve_lock policy to no_reserve and change ownership to oracle user .
5.after this inform oracle admin then he will detect disk and assign to ASM group.

RAW/ASM disk not contain any file system.its purpose is for getting performance.

Thanks !!!

AIX command

Commonly used AIX command

AIX LVM command

Lsvg       List all VG on server

Lsvg –o   List active VG on server

Lsvg –l  vgname                                List all logical volume in volume group

example  of mkvg command.

mkvg -y vgname hdisk0 hdisk1

mkvg -y my_vg –s 128 hdisk0 hdisk1

mkvg -s 2 -t 2 vgname hdisk1

# mkvg -B -y vgname -s 128 -f -n -V 101

# mkvg -S -y vgname -s 128 -f -n -V 101

Flag of mkvg

-s                    specify the physical partition size
-y                    indicate the name of the new vg

-f                     forcefully create volume group

-t                    t factor

-V                 major number of volume group

-S                Create scalable volume group

LV create command

#mklv -t jfs2 -y testlv datavg 60

Flag

-t type of LV

-y LV name

File system creation command

crfs -v jfs2 -d testlv -m /datamnt -A yes

-v  File system Type

-d  device on which we creating file system

-m mount point

-A  automount

#Mount /mount-point  File system mount command.

AIX system resource controller command

lssrc –a                 show all system resource controller and their status

lssrc  –s sshd      show sshd status on server    

lssrc –g nfs            show service status on all services in nfs group

lssrc –g group name                        Command to see service status in specific group

lssrc –s service_name                                    Command to see specific service status

startsrc –g groupname                  To start all services in specific group

stopsrc –g groupname                   To stop all services in specific group

refresh –s service name                               To refresh specific service

refresh –g groupname                  to refresh services in specific group

Command for changing bootlist and showing bootlist on AIX

bootlist -m normal -o    It will  display bootlist for AIX server.
example

#bootlist -m normal -o
output :
hdisk0 blv=hd5 pathid=0

bootlist -b     Show last boot device from which AIX LPAR booted.

bootlist -m normal hdisk0 blv=hd5 hdisk1 blv=hd5    This command change bootlist to hdisk0 and hdisk1

AIX file system utilization command

df -gt     This command display all filesystem which are mounted on AIX server.
df -gt /fsname   Display specific filesysetm disk utilization.

-g  show File system utilization in GB unit
-m show File system utilization in MB unit
-k  show File system utilization in 1024-byte blocks unit


Command to take OS backup on AIX

mksysb -ieX /path_to_take_backup     This command will create AIX mksysb backup with excluded content.

mksysb -iX   This will create full backup of AIX OS.

-i mksz command called to create image.data file.
-e check /etc/exclude.rootvg and exclude required file from backup
-X expand /tmp dynamically if needed


Mirroring AIX rootvg to hdisk1
extendvg rootvg hdisk1
mirrorvg rootvg
bosboot -ad hdisk0
bosboot -ad hdisk1
bootlist -m normal hdisk0 hdisk1



Command To list devices on AIX
cfgmgr  Detect new device on AIX
lsdev  List all devices on AIX LPAR
lsdev -Cc disk    Show disk information on AIX
lsdev -Cc adapter |grep -i fcs  show only FCS adapter information
lsdev -Cc adapter |grep -i ent  show only ethernet adapter information
lscfg -vl fcs0 | grep Network  display wwn of fcs0 adapter


Performance Monitoring
For monitoring performance of AIX LPAR admin can use command like topas,nmon,svmon,ipcs ,vmstat and df .

topas    Monitor top process which are consuming system resources ,also can monitor paging space ,memory and CPU utilization

while topas is running if admin press ~ (tild) then topas switch to nmon mode.

nmon   This command also show information like ,CPU ,Memory and paging
 space.

lsps -s  Show paging space utilization.

# svmon -P -O summary=basic,unit=MB    Virtual memory stats in Mb

#vmstat 2 5   Show five output at interval of 2 second.it contains CPU and memory utilization.

ipcs     Report interprocess communication information like shared memory,semaphore and queue details.

-s semaphore 
-m memory
q- message queue

will update new command .............

How to shrink AIX file system

Scenario:

df -gt /oracle

Filesystem    GB blocks      Used      Free %Used   Mounted on

/dev/oralv       100.00      49         50   50%    /oracle

/dev/oraloglv    100.00      49         50   50% /  /oralog

shrink /oracle file system by 20 GB and add that space to /oralog filesysetm

solution

:

**make sure that both filesystem are in same volume group,if not then we unable to allocate space

which we shrink from /oracle File system.

then use following command to shrink /oracle

#chfs -a size=-20G /oracle

above command will take some time ....

Then check again size of File system it must be resized to 80 GB.

Then next step is add that 20 GB which we taken from /oracle to /oralog.

command to extend size of /oralog

#chfs -a size=+20G /oralog

new size of /oralog will be 120GB .

AIX JFS -online increase possible ,but decrease not supported.

AIX JFS2-online decrease and increase supported.

QUORUM in aix

what is quorom in aix ?


During my AIX learning day i always wondered about what is quorom and how it affect volume group on AIX ? But after doing google i found answer like below.

Here is the answer :

Number of disk must be present in volume group to successfully vary on volume group in AIX.

"51% is mandatory vgda."


for example, XYZ volume group having 6 disk so mandatory disk for 51% VGDA is 4.

if disk in volume group is 10 then mandatory disk for 51% VGDA is 6.

Its simple


51% VGDA=Number of disk in Volume group/2 +1


so what happen if that mandatory percentage (51%) not available ?

rootvg quorom always kept disabled.

QUORUM:         1 (Disabled)

Reason behind this is when AIX OS boot and at the time of boot if rootvg having 2 disk and 1 disk from rootvg crashed then OS will never came up,because it not satisfies 51% VGDA rule.so its always good idea to keep mirrored rootvg quorom disabled.



will update new idea and thought on AIX

Thanks !!

Create filesystem in AIX using INLINE and JFS2 log

How To create filesystem in AIX using INLINE and JFS2 log

Method 1: INLINE Log

Identify Volume group on which we going to create Logical volume "testlv".

Make sure that volume group having enough physical partition free to create logical volume(LV).

Here Volume group is datavg

 

mklv -y testlv -t jfs2 datavg

crfs -v jfs2 -d testlv -m /test -a logname=INLINE -A yes

mount /test

Method 2:

Using JFS2 log

mklv -y testlv1 -t jfs2 datavg

mklv -y testloglv -t jfs2log -a e -r n datavg 1

mount /test1

Difference between JFS2 log and INLINE

• JFS2 log support many Filesystems.
• every Filesystem have 1 INLINE log
• if we have several busy Filesystem and we having JFS2log which support these FS,then there is chances of performance bottleneck, to avoid this INLINE log is good option where every filesystem can have 1 inline.