Configuring Exadata Cell Storage Servers from scratch

       


In this article we configure Exadata Cell Storage Servers from scratch, Storage interconnect, CELL Disks, Grid Disks, Flashcache, Flashlog, ASM Storage

Our Cell Server host interconnects as of now are these :

[root@cell ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:94:9C:E2  
          inet addr:10.10.1.11  Bcast:10.10.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:79481 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47935 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:101776703 (97.0 MiB)  TX bytes:9452946 (9.0 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:7642 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7642 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:509969 (498.0 KiB)  TX bytes:509969 (498.0 KiB)


We connect to ASM first to reconfigure Cell Storage
Connect to ASM Instance 

SQL> drop diskgroup data;
drop diskgroup data
*
ERROR at line 1:
ORA-15260: permission denied on ASM disk group

Connect as SYSASM 

SQL> conn / as sysasm
Connected.
SQL> drop diskgroup data;

Diskgroup dropped.

[root@cell ~]# cellcli
CellCLI: Release 12.1.1.1.1 - Production on Tue Oct 08 20:55:40 IST 2019

Copyright (c) 2007, 2013, Oracle.  All rights reserved.
Cell Efficiency Ratio: 941

Ensure that cell services are running, RS, CELLSRV and MS

CellCLI> alter cell startup services all 

Starting the RS, CELLSRV, and MS services...
CELL-01510: Cannot start a new Restart Server (RS) at port number: 9903. An instance of RS might be already running.
Getting the state of RS services...  running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.

Ensure from the linux cell OS that cellsrv process is running healthy and its parent CELLRSOMT

[root@cell ~]# ps -ef |grep cellrsomt
root      4536  3679  0 20:58 ?        00:00:00 /opt/oracle/cell12.1.1.1.1_LINUX.X64_140712/cellsrv/bin/cellrsomt -rs_conf /opt/oracle/cell12.1.1.1.1_LINUX.X64_140712/cellsrv/deploy/config/cellinit.ora -ms_conf /opt/oracle/cell12.1.1.1.1_LINUX.X64_140712/cellsrv/deploy/config/cellrsms.state -cellsrv_conf /opt/oracle/cell12.1.1.1.1_LINUX.X64_140712/cellsrv/deploy/config/cellrsos.state -debug 0
root      5023  4256  0 21:06 pts/0    00:00:00 grep cellrsomt


[root@cell ~]# ps -ef |grep "/cellsrv "
root      4538  4536  3 20:58 ?        00:00:12 /opt/oracle/cell12.1.1.1.1_LINUX.X64_140712/cellsrv/bin/cellsrv 100 5000 9 5042
root      4989  4256  0 21:05 pts/0    00:00:00 grep /cellsrv 



CellCLI> list griddisk
         data_CD_DATA01_cell     active
         data_CD_DATA02_cell     active

CellCLI> drop GRIDDISK ALL HARDDISK PREFIX=data
GridDisk data_CD_DATA01_cell successfully dropped
GridDisk data_CD_DATA02_cell successfully dropped

CellCLI> drop flashcache
Flash cache cell_FLASHCACHE successfully dropped

CellCLI> drop flashlog
Flash log cell_FLASHLOG successfully dropped

CellCLI> drop cell  
Cell cell successfully dropped
Stopping CELLSRV services... 
The SHUTDOWN of CELLSRV services was successful.

CellCLI> CREATE CELL interconnect1=eth0
Cell cell successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created...
CellDisk FD_00_cell successfully created
Flash log cell_FLASHLOG successfully created
Flash cache cell_FLASHCACHE successfully created

## In this case we will use eth0 since this is a test environment, In your machine it will be Infiband interconnect IBBOND0, and IBBOND1 or depends on your machine configuration and environment

## Ideally you should have two interconnects, but since this is a test environment we are using eth0 for test only

##  Ideally we use this :- CREATE CELL interconnect1=interface_name, interconnect2=interface_name

## But in our test environment we use below

CellCLI> CREATE CELL interconnect1=eth0
Cell cell successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created...
CellDisk FD_00_cell successfully created
Flash log cell_FLASHLOG successfully created
Flash cache cell_FLASHCACHE successfully created

CellCLI> CREATE CELLDISK ALL

CellDisk CD_DATA01_cell successfully created
CellDisk CD_DATA02_cell successfully created

CellCLI> list celldisk
         CD_DATA01_cell  normal
         CD_DATA02_cell  normal
         FD_00_cell      normal

CellCLI> CREATE GRIDDISK ALL HARDDISK PREFIX=data, size=20G

CELL-02646: CREATE GRIDDISK failed due to insufficient free space on cell disks: CD_DATA01_cell, CD_DATA02_cell.

## In this case we will use 10G since we are limited with space in the test environment

CellCLI> CREATE GRIDDISK ALL HARDDISK PREFIX=data, size=10G

GridDisk data_CD_DATA01_cell successfully created
GridDisk data_CD_DATA02_cell successfully created

CellCLI> list griddisk 
         data_CD_DATA01_cell     active
         data_CD_DATA02_cell     active

Connect to ASM Instance

SQL>   CREATE DISKGROUP data external REDUNDANCY disk 'o/10.10.1.11/data_CD_DATA02_cell'
   ATTRIBUTE 'AU_SIZE' = '4M',
              'compatible.rdbms'='11.2.0.4',
              'compatible.asm'='12.1.0.2',
   'cell.smart_scan_capable'='TRUE'
   /

Diskgroup created.

## Your ASM storage with Disk group is now ready for the database