The appliance_console_cli on secondary tries to init disks twice, leading to failure

[jarovo]

Version: cfme-5.11.7.3-1.el8cf.x86_64
reproducibility 100% with our test suite.

The command
appliance_console_cli --internal --username root --password smartvm --replication standby --primary-host $SOME_IP --standby-host $SOME_OTHER_IP --cluster-node-number 2 --auto-failover --dbname vmdb_production --verbose --dbdisk /dev/vdb --standalone

Fails as follows (note the doubled Initialize postgresql disk starting):

create encryption key
configuring internal database
Initialize postgresql disk starting                    # Note! First time!
Initialize postgresql disk complete
Initialize postgresql starting
Initialize postgresql complete
Configuring Server as Standby
Configuring Replication Standby Server...
Initialize postgresql disk starting                    # Note! Second time!
/opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logging.rb:41:in `say_error': ManageIQ::ApplianceConsole::MiqSignalError (ManageIQ::ApplianceConsole::MiqSignalError)
Initialize postgresql disk failed with error - /sbin/parted exit code: 1 error was: Error: Partition(s) 1 on /dev/vdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a resu....
See /var/www/miq/vmdb/log/appliance_console.log for details.
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logging.rb:82:in `log_and_feedback_exception'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logging.rb:57:in `rescue in log_and_feedback'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logging.rb:54:in `log_and_feedback'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/database_replication_standby.rb:159:in `initialize_postgresql_disk'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/database_replication_standby.rb:63:in `activate'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/cli.rb:295:in `set_replication'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/cli.rb:197:in `run'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/cli.rb:427:in `parse'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/bin/appliance_console_cli:7:in `<top (required)>'
	from /opt/rh/cfme-gemset/bin/appliance_console_cli:23:in `load'
	from /opt/rh/cfme-gemset/bin/appliance_console_cli:23:in `<main>'

This is in the logs, but it appears to be unrelated:

I, [2020-08-18T12:29:23.116465 ManageIQ/manageiq#2178]  INFO -- : MIQ(ManageIQ::ApplianceConsole::DatabaseReplicationStandby#initialize_postgresql_disk) : starting
E, [2020-08-18T12:29:24.167927 ManageIQ/manageiq#2178] ERROR -- : MIQ(ManageIQ::ApplianceConsole::DatabaseReplicationStandby#initialize_postgresql_disk)  Command failed: /sbin/parted exit code: 1 error was: Error: Partition(s) 1 on /dev/vdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.
. Error: Error: Partition(s) 1 on /dev/vdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.
. Output: . At: /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logical_volume_management.rb:51:in `create_partition_to_fill_disk'

After untangling the logging by removing the log_and_feedback i get this output:

TERM environment variable not set.
create encryption key
configuring internal database
Initialize postgresql disk starting
Initialize postgresql disk complete
Initialize postgresql starting
Initialize postgresql complete
Configuring Server as Standby
Configuring Replication Standby Server...
/opt/rh/cfme-gemset/gems/awesome_spawn-1.5.0/lib/awesome_spawn.rb:108:in `run!': /sbin/parted exit code: 1 error was: Error: Partition(s) 1 on /dev/vdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes. (AwesomeSpawn::CommandResultError)

Error: Partition(s) 1 on /dev/vdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.

	from /opt/rh/cfme-gemset/gems/linux_admin-2.0.1/lib/linux_admin/common.rb:24:in `run!'
	from /opt/rh/cfme-gemset/gems/linux_admin-2.0.1/lib/linux_admin/disk.rb:52:in `create_partition_table'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logical_volume_management.rb:51:in `create_partition_to_fill_disk'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/logical_volume_management.rb:32:in `setup'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/database_replication_standby.rb:164:in `initialize_postgresql_disk'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/database_replication_standby.rb:63:in `activate'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/cli.rb:295:in `set_replication'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/cli.rb:197:in `run'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/lib/manageiq/appliance_console/cli.rb:427:in `parse'
	from /opt/rh/cfme-gemset/gems/manageiq-appliance_console-5.3.3/bin/appliance_console_cli:7:in `<top (required)>'
	from /opt/rh/cfme-gemset/bin/appliance_console_cli:23:in `load'
	from /opt/rh/cfme-gemset/bin/appliance_console_cli:23:in `<main>

Executing manually: parted -s /dev/vdb mklabel gpt shows

Error: Partition(s) 1 on /dev/vdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.

This tells me that the disk indeed was inited already:

[root@dhcp-8-... ~]# lvs
  LV               VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home          VG-CFME -wi-ao----   1.00g                                                    
  lv_log           VG-CFME -wi-ao----  10.00g                                                    
  lv_os            VG-CFME -wi-ao----   4.50g                                                    
  lv_swap          VG-CFME -wi-ao----  <9.00g                                                    
  lv_tmp           VG-CFME -wi-ao----   1.00g                                                    
  lv_var           VG-CFME -wi-ao----  12.00g                                                    
  lv_var_log       VG-CFME -wi-ao----   1.00g                                                    
  lv_var_log_audit VG-CFME -wi-ao---- 512.00m                                                    
  lv_pg            vg_pg   -wi-ao----   4.65g                                                    
[root@dhcp-8-... ~]# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/vda2  VG-CFME lvm2 a--  <39.00g    0 
  /dev/vdb1  vg_pg   lvm2 a--    4.65g    0