Saturday 28 March 2020

Bad magic number in super-block while trying to open /dev/rhel/root

While running an oracle upgrade i ran into file system space issues, in Rhel 7 without any downtime, how we can extend the volumes please check here, There are some steps that we have to follow :

1. Present a new physical disk to the server

2. Adding the new physical disk to an exisiting Volume Group.

3. Expanding a logical volume within that Volume Group to use the new disk space.

4. Notify the operating system about the change in size.

[oracle@rhel7 ~]$ su -
Password:
Last login: Sat Mar 28 21:11:53 IST 2020 on pts/3
[root@rhel7 ~]# fdisk -l

Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b49f1

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2050047     1024000   83  Linux
/dev/sda2         2050048   100370431    49160192   8e  Linux LVM

Disk /dev/mapper/rhel-root: 41.9 GB, 41943040000 bytes, 81920000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/rhel-swap: 8388 MB, 8388608000 bytes, 16384000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Step 1: Present the new physical disk to the server.

And Step 2: Add the new physical disk to an existing Volume Group.

[root@rhel7 ~]# fdisk /dev/sda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): e
Partition number (3,4, default 3):
First sector (100370432-251658239, default 100370432):
Using default value 100370432
Last sector, +sectors or +size{K,M,G} (100370432-251658239, default 251658239): +40G
Partition 3 of type Extended and of size 40 GiB is set

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

# Now run the partprobe command to sync the changes

[root@rhel7 ~]# partprobe
[root@rhel7 ~]# fdisk -l

Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b49f1

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2050047     1024000   83  Linux
/dev/sda2         2050048   100370431    49160192   8e  Linux LVM
/dev/sda3       100370432   184256511    41943040    5  Extended

Disk /dev/mapper/rhel-root: 41.9 GB, 41943040000 bytes, 81920000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/rhel-swap: 8388 MB, 8388608000 bytes, 16384000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


[root@rhel7 ~]# vgdisplay
  --- Volume group ---
  VG Name               rhel
  System ID           
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               86.87 GiB
  PE Size               4.00 MiB
  Total PE              22239
  Alloc PE / Size       12000 / 46.88 GiB
  Free  PE / Size       10239 / 40.00 GiB
  VG UUID               NvVNw2-XR30-7oS3-eGkz-81ZP-YziW-G238Kw

Step 3: Expand the logical volume to use the new space
 
[root@rhel7 ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/rhel/root
  LV Name                root
  VG Name                rhel
  LV UUID                QibbKI-Hu0u-5I16-nYhe-VRXI-a14N-JgyX8N
  LV Write Access        read/write
  LV Creation host, time localhost, 2020-03-28 20:23:37 +0530
  LV Status              available
  # open                 1
  LV Size                39.06 GiB
  Current LE             10000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0
 
  --- Logical volume ---
  LV Path                /dev/rhel/swap
  LV Name                swap
  VG Name                rhel
  LV UUID                ATHN2H-ycQ6-Z0az-pkxp-jCvF-2JBd-OWs7KG
  LV Write Access        read/write
  LV Creation host, time localhost, 2020-03-28 20:23:37 +0530
  LV Status              available
  # open                 2
  LV Size                7.81 GiB
  Current LE             2000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1
 
[root@rhel7 ~]# lvextend -L+30G /dev/rhel/root
  Extending logical volume root to 69.06 GiB
  Logical volume root successfully resized


[root@rhel7 ~]# lvextend -L+8G /dev/rhel/root
  Extending logical volume root to 77.06 GiB
  Logical volume root successfully resized

# I ran into an issue with below command, the fix for that is given below,
# this is step 4 Step 4: Update the filesystem to use the new space

[root@rhel7 ~]# resize2fs /dev/rhel/root
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/rhel/root
Couldn't find valid filesystem superblock.

Check the space is still low :

[root@rhel7 ~]# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   40G   37G  2.4G  95% /
devtmpfs               6.0G     0  6.0G   0% /dev
tmpfs                  6.0G  148K  6.0G   1% /dev/shm
tmpfs                  6.0G  8.9M  6.0G   1% /run
tmpfs                  6.0G     0  6.0G   0% /sys/fs/cgroup
/dev/sda1              997M  126M  872M  13% /boot



[root@rhel7 ~]# xfs_growfs /dev/rhel/root
meta-data=/dev/mapper/rhel-root  isize=256    agcount=4, agsize=2560000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=10240000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=5000, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 10240000 to 20201472


# Now check the space, its 78 GB now

[root@rhel7 ~]# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   78G   37G   41G  48% /
devtmpfs               6.0G     0  6.0G   0% /dev
tmpfs                  6.0G  148K  6.0G   1% /dev/shm
tmpfs                  6.0G  8.9M  6.0G   1% /run
tmpfs                  6.0G     0  6.0G   0% /sys/fs/cgroup
/dev/sda1              997M  126M  872M  13% /boot


Monitoring the classic replicat

Sometimes we see huge lag in our goldengate configuration that is because there are pending long running transaction at the target side, lets see how we can monitor the replicat status in the target system


GGSCI (node2.example.com) 27> send replicat rep1 status

Sending STATUS request to REPLICAT REP1 ...
  Current status: Processing data
  Sequence #: 22
  RBA: 9315748
  87049 records in current transaction


GGSCI (node2.example.com) 28> send replicat rep1 status

Sending STATUS request to REPLICAT REP1 ...
  Current status: Processing data
  Sequence #: 22
  RBA: 10275752
  96021 records in current transaction


GGSCI (node2.example.com) 31> dblogin useridlias ogg
Password:
ERROR: Unrecognized parameter (USERIDLIAS), expected USERID/USERIDALIAS.

GGSCI (node2.example.com) 32> dblogin useridalias ogg
Successfully logged into database.

GGSCI (node2.example.com as ggadmin@east) 33> info replicat rep1, detail

REPLICAT   REP1      Last Started 2020-03-28 14:28   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:02:28 ago)
Process ID           7179
Log Read Checkpoint  File /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000020
                     First Record  RBA 55224002

Current Log BSN value: 1877190

Last Committed Transaction CSN value: 1897958


  Extract Source                          Begin             End           

  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000020  2020-03-28 13:33  First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000020  2020-03-24 23:48  2020-03-28 13:33
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000018  2020-03-13 23:07  2020-03-24 23:48
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000016  2020-03-13 23:07  First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000016  2019-11-22 13:10  2020-03-13 23:07
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000010  * Initialized *   2019-11-22 13:10
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000010  * Initialized *   First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000009  2019-11-14 22:52  2019-11-22 12:53
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000002  2019-11-14 22:49  2019-11-14 22:52
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000002  2019-11-14 22:49  First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000002  * Initialized *   2019-11-14 22:49
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000000  * Initialized *   First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000000  * Initialized *   First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000000  * Initialized *   First Record 
  /u01/app/oracle/product/12.1.2/ggate/dirdat/rt000000  * Initialized *   First Record 


Current directory    /u01/app/oracle/product/12.1.2/ggate

Report file          /u01/app/oracle/product/12.1.2/ggate/dirrpt/REP1.rpt
Parameter file       /u01/app/oracle/product/12.1.2/ggate/dirprm/rep1.prm
Checkpoint file      /u01/app/oracle/product/12.1.2/ggate/dirchk/REP1.cpr
Checkpoint table     ggadmin.ckpt
Process file         /u01/app/oracle/product/12.1.2/ggate/dirpcs/REP1.pcr
Error log            /u01/app/oracle/product/12.1.2/ggate/ggserr.log


ERROR OGG-00664 OCI Error beginning session status = 1017-ORA-01017

Fixing credential store issue in Goldengate, i faced this issue recently so thought to update here on the blog, usually when we refresh our environments there is an issue with credentials of goldengate administrator, in that case you have to check credential store and make sure to update in extract or replicat process parameter files.


GGSCI (node1.example.com) 2> start *

Sending START request to MANAGER ...
EXTRACT DPUMP1 starting

Sending START request to MANAGER ...
EXTRACT EXT1 starting


GGSCI (node1.example.com) 3> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                         
EXTRACT     STOPPED     DPUMP1      00:00:00      84:16:29 
EXTRACT     STOPPED     EXT1            00:00:00      84:16:28 

GGSCI (node1.example.com) 12> view report ext1


***********************************************************************
                 Oracle GoldenGate Capture for Oracle
   Version 12.1.2.1.0 OGGCORE_12.1.2.1.0_PLATFORMS_140727.2135.1_FBO
   Linux, x64, 64bit (optimized), Oracle 12c on Aug  7 2014 10:40:21

Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.


                    Starting at 2020-03-28 12:53:43
***********************************************************************

***********
***
                          : [/u01/app/oracle/product/12.1.2/ggate/extract(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x4f) [0x6ab4df]]
                          : [/u01/app/oracle/product/12.1.2/ggate/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)
+0x104) [0x6ab9e4]]
                          : [/u01/app/oracle/product/12.1.2/ggate/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x8b) [0x6abdab]]
                          : [/u01/app/oracle/product/12.1.2/ggate/extract(main+0x3f) [0x5bfa2f]]
                          : [/lib64/libc.so.6(__libc_start_main+0xf4) [0x3ed681d994]]
                          : [/u01/app/oracle/product/12.1.2/ggate/extract [0x534319]]

2020-03-28 12:53:45  ERROR   OGG-00664  OCI Error beginning session (status = 1017-ORA-01017: invalid username/password; logon denied).

2020-03-28 12:53:45  ERROR   OGG-01668  PROCESS ABENDING.


GGSCI (node1.example.com) 13>   info credentialstore

Reading from ./dircrd/:

Default domain: OracleGoldenGate

  Alias: ggadmin
  Userid: ggadmin

  Alias: ggus
  Userid: ggus

GGSCI (node1.example.com) 14> delete credentialstore

Credential store deleted from ./dircrd/.

GGSCI (node1.example.com) 15> info credentialstore

ERROR: Unable to open credential store in ./dircrd/.


# Here you need to know the password of ggadmin user (administrater user of goldengate) if you dont know you can reset as well and update the credential store.

GGSCI (node1.example.com) 16> alter credentialstore add useridalias ogg user ggadmin, password g

ERROR: Invalid argument 'useridalias'.

GGSCI (node1.example.com) 17> alter credentialstore add user ggadmin alias ogg
Password:

ERROR: Unable to open credential store in ./dircrd/.

GGSCI (node1.example.com) 18> ALTER CREDENTIALSTORE ADD USER ggadmin alias ogg password g

ERROR: Invalid argument 'password'.

GGSCI (node1.example.com) 19>  ALTER CREDENTIALSTORE ADD USER ggadmin alias ogg
Password:

ERROR: Unable to open credential store in ./dircrd/.

GGSCI (node1.example.com) 20> add credentialstore

Credential store created in ./dircrd/.

GGSCI (node1.example.com) 21>  ALTER CREDENTIALSTORE ADD USER ggadmin alias ogg
Password:

Credential store in ./dircrd/ altered.

GGSCI (node1.example.com) 22> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                         
EXTRACT      STOPPED     DPUMP1      00:00:00      84:25:11 
EXTRACT      STOPPED     EXT1            00:00:00      84:25:10 


## Update the parameter file with this alias

GGSCI (node1.example.com) 30> view params ext1

EXTRACT ext1
USERIDALIAS ogg

EXTTRAIL /u01/app/oracle/product/12.1.2/ggate/dirdat/lt
TABLE SCOTT.TAB1;
TABLE SCOTT.TAB2;


GGSCI (node1.example.com) 26> start *

Sending START request to MANAGER ...
EXTRACT DPUMP1 starting

Sending START request to MANAGER ...
EXTRACT EXT1 starting


GGSCI (node1.example.com) 27> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                         
EXTRACT       RUNNING     DPUMP1      00:00:00      84:25:43 
EXTRACT      RUNNING     EXT1        84:26:08      00:00:01 


GGSCI (node1.example.com) 28> info ext1

EXTRACT    EXT1      Last Started 2020-03-28 13:02   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:05 ago)
Process ID           5950
Log Read Checkpoint  Oracle Redo Logs
                     2020-03-28 13:10:24  Seqno 17, RBA 5702656
                     SCN 0.1896693 (1896693)