Add Disks with Different Sizes to ASM Diskgroup ONLINE

Scenario:

As of a sudden, I came to know that an AWS EC2 instance has a limitation of maximum of 27 EBS volumes to be attached to it. One of my EC2 DB server was already having 26 volumes attached to it, most of the volumes sizes is 500 GB. I planned to replace all the 500 GB luns with 1 TB luns to free a room for adding more luns in the future.

Facts:

- My DB is using ASM.
- My ASM instance version is 19c.
- My ASM Diskgroups redundancy is EXTERNAL, which means I've no redundancy, which means losing one disk can bring the whole diskgroup offline.
- The ASM diskgroup can contain disks with different sizes at the same time (this will help me perform the disk replacement with bigger sizes online). Not recommended from the performance perspective, but it's OK to have it for disk replacement purpose.
- The ASM diskgroup can balance the data evenly between the disks with different sizes on the same diskgroup, which means if the diskgroup is 80% full, each disk inside the diskgroup will be 80% full regardless of its size.

Technical Steps:

For the purpose of keeping this post briefed, I'll show here an example of replacing two disks of 516 GB with 1 TB disk in the RECO disk group.

1- 1 TB volume is attached to the machine.

2- I've partitioned the disk into 1 primary partition using fdisk command:

3- Labeled the 1TB disk as an ASM disk:

# oracleasm createdisk PSS_RECO_DISK01 /dev/nvme17n1p1

4- login to the ASM instance as / as sysasm:

# export ORACLE_SID=+ASM
# $GRID_HOME/bin/sqlplus '/ as sysasm'

5- At the same time, add the new disk of 1 TB to the diskgroup and remove two disks of 516 GB from the same diskgroup:

Note: you don't have to wait between adding the new and removing the old disks, run "ADD DISK" command followed by "DROP DISK" command in sequence as once you get back the prompt, but you shouldn't remove the old disks from the machine until the REBALANCE completes (remember: my diskgroup redundancy is EXTERNAL, this is why I must wait for the REBALANCE activity to complete)

-- Increase the rebalance power to speed up the replacement of the disks:
SQL>  ALTER DISKGROUP PSS_RECO REBALANCE POWER 11;

-- Add the 1TB disk:
SQL> ALTER DISKGROUP PSS_RECO ADD DISK '/dev/oracleasm/disks/PSS_RECO_DISK01';

-- Remove two disks of 500G from the same diskgroup:
SQL> ALTER DISKGROUP PSS_RECO drop disk PSS_RECO_0000;
SQL> ALTER DISKGROUP PSS_RECO drop disk PSS_RECO_0001;

6- Wait until the REBALANCE activity complete:

-- Keep checking the status until it returns no rows:
SQL> select * from V$ASM_OPERATION;

-- Also you can use this query to check the size of Data which being rebalanced so far:
SQL>
col DISK_PATH        for a45
col diskgroup_name  for a15
col FAILGROUP       for a17
col disk_name            for a17
col TOTAL_GB         for 9999999.99
col FREE_GB            for 9999999.99
SELECT NVL(a.name, '[CANDIDATE]')diskgroup_name, b.path disk_path, b.name disk_name,
b.failgroup, b.total_MB/1024 TOTAL_GB, b.free_MB/1024 FREE_GB, b.state, b.MODE_STATUS
FROM v$asm_diskgroup a RIGHT OUTER JOIN v$asm_disk b USING (group_number)
ORDER BY a.name, b.path;

-- After approximately 49 minutes , it completed:

-- The dropped disks should now show as candidates:

SQL> SELECT NVL(a.name, '[CANDIDATE]')diskgroup_name, b.path disk_path, b.name disk_name,
b.failgroup, b.total_MB/1024 TOTAL_GB, b.free_MB/1024 FREE_GB, b.state, b.MODE_STATUS
FROM v$asm_diskgroup a RIGHT OUTER JOIN v$asm_disk b USING (group_number)
ORDER BY a.name, b.path;

7- After the dropped disks show as candidates, now you can un-provision them from the server:

# oracleasm deletedisk PSS_RECO_0000
# oracleasm deletedisk PSS_RECO_0001

# oracleasm scandisks

8- Don't forget to return the REBALANCE power on the same diskgroup to its default:

SQL>  ALTER DISKGROUP PSS_RECO REBALANCE POWER 1;