Error with Ubuntu HA+LVM2+DRBD: VG creation refers to wrong device
Currently I am working on a system with 2 servers  and have to set up iSCSI target service over LVM volume over DRBD. These are basic configuration details:

 - Architecture: i386 32-bit
 - Disk: VM Disk over SATA RAID-1
 - OS: Ubuntu 12.04
 - DRBD version: 8.3.11
 - DRBD base file system: ext4
 - LVM2 version: 2.02.66
 - cLVM version: 2.02.66
 - Cluster infrastructure: pacemaker+heartbeat
 - Pacemaker version: 1.1.6
 - Heartbeat version: 1:3.0.5

I created DRBD disk, created crm resources to control master-slave DRBD activation between 2 servers, synced the data, and tried setting up PV, VG and LV over it from the Primary node. This is the result:


    piing@piing-node-2:~$ sudo lvcreate -L 186G -n lun0 iscsivg01
     device-mapper: reload ioctl failed: Invalid argument
      Aborting. Failed to activate new LV to wipe the start of it.
      Node /dev/mapper/iscsivg01-lun0 was not removed by udev. Falling back to direct node removal.
      
    piing@piing-node-2:~$ sudo lvcreate -vvvv -L 186G -n lun0 iscsivg01
    #lvmcmdline.c:1035         Processing: lvcreate -vvvv -L 186G -n lun0 iscsivg01
    #lvmcmdline.c:1038         O_DIRECT will be used
    #config/config.c:987       Setting global/locking_type to 1
    #config/config.c:987       Setting global/wait_for_locks to 1
    #locking/locking.c:240       File-based locking selected.
    #config/config.c:964       Setting global/locking_dir to /var/lock/lvm
    #activate/activate.c:361       Getting target version for linear
    #ioctl/libdm-iface.c:1821         dm version   OF   [16384]
    #ioctl/libdm-iface.c:1821         dm versions   OF   [16384]
    #activate/activate.c:361       Getting target version for striped
    #ioctl/libdm-iface.c:1821         dm versions   OF   [16384]
    ..................................................................................
    #device/dev-io.c:533         Closed /dev/ram8
    #device/dev-io.c:487         Opened /dev/sda8 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    **#label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2)
    #format_text/format-text.c:1137         /dev/sda8: Found metadata at 6656 size 678 (in area at 4096 size 192512) for iscsivg01 (JuQJwx-pH4y-vuGj-uo82-viH0-B0DH-0Do7dx)
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG iscsivg01 with 1 mdas**
    #cache/lvmcache.c:923         lvmcache: /dev/sda8: setting iscsivg01 VGID to JuQJwxpH4yvuGjuo82viH0B0DH0Do7dx
    #cache/lvmcache.c:1173         lvmcache: /dev/sda8: VG iscsivg01: Set creation host to piing-node-2.
    #device/dev-io.c:487         Opened /dev/ram9 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram9: block size is 4096 bytes
    #label/label.c:184       /dev/ram9: No label detected
    #device/dev-io.c:533         Closed /dev/ram9
    ...................................................................................
    #label/label.c:270         Using cached label for /dev/sda8
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (3) from /dev/sda8 at 6656 size 678
    #metadata/pv_manip.c:296         /dev/sda8 0:      0  47683: NULL(0:0)
    #format_text/archiver.c:127     Archiving volume group "iscsivg01" metadata (seqno 3).
    #metadata/lv_manip.c:2042     Creating logical volume lun0
    #metadata/pv_map.c:55         Allowing allocation on /dev/sda8 start PE 0 length 47683
    #metadata/lv_manip.c:1366         Trying allocation using contiguous policy.  Need 47616 extents for 1 parallel areas and 0 log areas of 0 extents. (Total 47616 extents.)
    #metadata/lv_manip.c:1230         Trying allocation area 0 on /dev/sda8 start PE 0 length 47616 leaving 67.
    #metadata/lv_manip.c:795         Allocating parallel area 0 on /dev/sda8 start PE 0 length 47616.
    #metadata/pv_manip.c:296         /dev/sda8 0:      0  47616: lun0(0:0)
    #metadata/pv_manip.c:296         /dev/sda8 1:  47616     67: NULL(0:0)
    #device/dev-io.c:533         Closed /dev/sda8
    #device/dev-io.c:487         Opened /dev/sda8 RW O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    #format_text/format-text.c:590         Writing iscsivg01 metadata to /dev/sda8 at 7680 len 941
    #format_text/format-text.c:695         Pre-Committing iscsivg01 metadata (4) to /dev/sda8 header at 4096
    #format_text/format-text.c:695         Committing iscsivg01 metadata (4) to /dev/sda8 header at 4096
    #format_text/archiver.c:390     Creating volume group backup "/etc/lvm/backup/iscsivg01" (seqno 4).
    #format_text/format-text.c:888         Writing iscsivg01 metadata to /etc/lvm/backup/.lvm_piing-node-2_14473_2072241393
    #format_text/format-text.c:932         Committing iscsivg01 metadata (4)
    #format_text/format-text.c:933         Renaming /etc/lvm/backup/iscsivg01.tmp to /etc/lvm/backup/iscsivg01
    #locking/file_locking.c:291       Locking LV JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy (R)
    #metadata/metadata.c:3091       Finding volume group for uuid JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2) with 1 mdas
    #format_text/format-text.c:1137         /dev/sda8: Found metadata at 7680 size 941 (in area at 4096 size 192512) for iscsivg01 (JuQJwx-pH4y-vuGj-uo82-viH0-B0DH-0Do7dx)
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG iscsivg01 with 1 mdas
    #cache/lvmcache.c:923         lvmcache: /dev/sda8: setting iscsivg01 VGID to JuQJwxpH4yvuGjuo82viH0B0DH0Do7dx
    #cache/lvmcache.c:1173         lvmcache: /dev/sda8: VG iscsivg01: Set creation host to piing-node-2.
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (4) from /dev/sda8 at 7680 size 941
    #metadata/metadata.c:3097     Found volume group "iscsivg01"
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #device/dev-io.c:291       /dev/sda8: read_ahead is 256 sectors
    #mm/memlock.c:265       Locking memory
    #mm/memlock.c:176         mlock        764KiB      8048000 -      8107000 r-xp  00000000 08:05 265000     /sbin/lvm
    ..................................................................................
    #mm/memlock.c:232         Locked 14680064 bytes
    #mm/memlock.c:318         memlock_count inc to 1
    #activate/dev_manager.c:827         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #activate/dev_manager.c:827         Getting device info for iscsivg01-lun0-real [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-real]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-real OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-real OF   [16384]
    #activate/dev_manager.c:827         Getting device info for iscsivg01-lun0-cow [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-cow]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-cow OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-cow OF   [16384]
    #activate/dev_manager.c:1135         Checking kernel supports striped segment type for lun0
    #metadata/metadata.c:2160         Calculated readahead of LV lun0 is 256
    #libdm-deptree.c:1268     Creating iscsivg01-lun0
    #ioctl/libdm-iface.c:1821         dm create iscsivg01-lun0 LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #libdm-common.c:799         iscsivg01-lun0: Stacking NODE_ADD (252,1) 0:6 0660
    #libdm-deptree.c:1601     Loading iscsivg01-lun0 table (252:1)
    **#libdm-deptree.c:1547         Adding target to (252:1): 0 390070272 linear 8:8 384
    #ioctl/libdm-iface.c:1821         dm table   (252:1) OF   [16384]
    #ioctl/libdm-iface.c:1821         dm reload   (252:1) NF   [16384]
    #ioctl/libdm-iface.c:1838   device-mapper: reload ioctl failed: Invalid argument**
    #libdm-deptree.c:1686         <backtrace>
    #activate/dev_manager.c:1471         <backtrace>
    #activate/dev_manager.c:1506         <backtrace>
    #activate/activate.c:571         <backtrace>
    #activate/activate.c:1128         <backtrace>
    #mm/memlock.c:282       Unlocking memory
    #mm/memlock.c:176         munlock        764KiB      8048000 -      8107000 r-xp  00000000 08:05 265000     /sbin/lvm
    ..................................................................................
    #mm/memlock.c:176         munlock        268KiB     bfb20000 -     bfb63000 rw-p  00000000 00:00 0          [stack]
    #mm/memlock.c:232         Unlocked 14405632 bytes
    #mm/memlock.c:327         memlock_count dec to 0
    #libdm-common.c:475         Created /dev/mapper/iscsivg01-lun0
    #activate/activate.c:1155         <backtrace>
    #locking/locking.c:389         <backtrace>
    #metadata/lv_manip.c:3230   Aborting. Failed to activate new LV to wipe the start of it.
    #locking/file_locking.c:286       Locking LV JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy (NL)
    #metadata/metadata.c:3091       Finding volume group for uuid JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #label/label.c:270         Using cached label for /dev/sda8
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (4) from /dev/sda8 at 7680 size 941
    #metadata/metadata.c:3097     Found volume group "iscsivg01"
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #mm/memlock.c:265       Locking memory
    #mm/memlock.c:176         mlock        764KiB      8048000 -      8107000 r-xp  00000000 08:05 265000     /sbin/lvm
    #mm/memlock.c:176         mlock          8KiB      8107000 -      8109000 r--p  000bf000 08:05 265000     /sbin/lvm
    #mm/memlock.c:176         mlock         12KiB      8109000 -      810c000 rw-p  000c1000 08:05 265000     /sbin/lvm
    #mm/memlock.c:176         mlock         32KiB      810c000 -      8114000 rw-p  00000000 00:00 0
    #mm/memlock.c:176         mlock      10500KiB      8ece000 -      990f000 rw-p  00000000 00:00 0          [heap]
    #mm/memlock.c:158         mlock default filter 'locale/locale-archive' matches 'b7282000-b7482000 r--p 00000000 08:05 400424     /usr/lib/locale/locale-archive': Skipping.
    ..................................................................................
    #mm/memlock.c:232         Unlocked 14405632 bytes
    #mm/memlock.c:327         memlock_count dec to 0
    [COLOR="Red"]  Node /dev/mapper/iscsivg01-lun0 was not removed by udev. Falling back to direct node removal.[/COLOR]
    #libdm-common.c:503         Removed /dev/mapper/iscsivg01-lun0
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #locking/file_locking.c:281       Unlocking LV JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #metadata/metadata.c:3091       Finding volume group for uuid JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #label/label.c:270         Using cached label for /dev/sda8
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (4) from /dev/sda8 at 7680 size 941
    #metadata/metadata.c:3097     Found volume group "iscsivg01"
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #activate/activate.c:938         <backtrace>
    #metadata/pv_manip.c:296         /dev/sda8 0:      0  47616: NULL(0:0)
    #metadata/pv_manip.c:296         /dev/sda8 1:  47616     67: NULL(0:0)
    #format_text/format-text.c:590         Writing iscsivg01 metadata to /dev/sda8 at 8704 len 678
    #format_text/format-text.c:695         Pre-Committing iscsivg01 metadata (5) to /dev/sda8 header at 4096
    #format_text/format-text.c:695         Committing iscsivg01 metadata (5) to /dev/sda8 header at 4096
    #format_text/archiver.c:390     Creating volume group backup "/etc/lvm/backup/iscsivg01" (seqno 5).
    #format_text/format-text.c:888         Writing iscsivg01 metadata to /etc/lvm/backup/.lvm_piing-node-2_14473_712565374
    #format_text/format-text.c:932         Committing iscsivg01 metadata (5)
    #format_text/format-text.c:933         Renaming /etc/lvm/backup/iscsivg01.tmp to /etc/lvm/backup/iscsivg01
    #lvcreate.c:540         <backtrace>
    #locking/file_locking.c:74       Unlocking /var/lock/lvm/V_iscsivg01
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/V_iscsivg01
    #device/dev-io.c:533         Closed /dev/sda8


I've searched for the solution and tried many things: tweaking filter in lvm.conf, partitioning the drbd device, etc.. but none worked. Moreover, I've actually set up such system successfully just a week before. But now when the servers had to be re-configured with RAID-1 and re-installed, the old settings stopped working. 

This is the filter line I'm using:

	filter = [ "a|/dev/lvm/drbd|", "a|/dev/drbd|", "r/.*/" ]


While investigating further, I found that when vgcreate command was used to create Volume Group over Physical Volume already sitting on DRBD device ( /dev/drbd0, /dev/lvm/drbd0 ), the device reference from pvs fell to ( /dev/sda8 ), the device underlying DRBD. Even weirder is that result from pvscan still showed as DRBD device.

    piing@piing-node-2:~$ sudo vgs
      No volume groups found
      
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      No volume groups found
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/drbd0      lvm2 a-   186.26g 186.26g
      
    piing@piing-node-2:~$ sudo pvscan -v
        Wiping cache of LVM-capable devices
        Wiping internal VG cache
        Walking through all physical volumes
      PV /dev/drbd0   VG iscsivg01   lvm2 [186.26 GiB / 186.26 GiB free]
      Total: 1 [186.26 GiB] / in use: 1 [186.26 GiB] / in no VG: 0 [0   ]
      ]
      
    piing@piing-node-2:~$ sudo vgcreate iscsivg01 /dev/drbd0
      Volume group "iscsivg01" successfully created
      
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      Found volume group "iscsivg01" using metadata type lvm2
      
    piing@piing-node-2:~$ sudo vgs
      VG        #PV #LV #SN Attr   VSize   VFree
      iscsivg01   1   0   0 wz--n- 186.26g 186.26g
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG        Fmt  Attr PSize   PFree
      **/dev/sda8  iscsivg01 lvm2 a-   186.26g 186.26g**
      
    piing@piing-node-2:~$ sudo pvscan -v
        Wiping cache of LVM-capable devices
        Wiping internal VG cache
        Walking through all physical volumes
      **PV /dev/drbd0   VG iscsivg01   lvm2 [186.26 GiB / 186.26 GiB free]**
      Total: 1 [186.26 GiB] / in use: 1 [186.26 GiB] / in no VG: 0 [0   ]
      ]
      

If I then removed the VG, result from pvs went back to showing DRBD device. Then I created VG again, and it becomes /dev/sda8 again.

    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed
      
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      No volume groups found
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/drbd0      lvm2 a-   186.26g 186.26g
      
    piing@piing-node-2:~$ sudo vgcreate iscsivg01 /dev/drbd0
      Volume group "iscsivg01" successfully created
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG        Fmt  Attr PSize   PFree
      **/dev/sda8  iscsivg01 lvm2 a-   186.26g 186.26g**
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      Found volume group "iscsivg01" using metadata type lvm2
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG        Fmt  Attr PSize   PFree
      **/dev/sda8  iscsivg01 lvm2 a-   186.26g 186.26g**
      
    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/drbd0      lvm2 a-   186.26g 186.26g
      

And the system doesn't even recognize ( /dev/drbd0 ) or ( /dev/lvm/drbd0 ) to be in volume group ( iscsivg01 ) when directly addressed.

    piing@piing-node-2:~$ sudo pvs -v /dev/drbd0
        Using physical volume(s) on command line
        Wiping cache of LVM-capable devices
      PV             VG   Fmt  Attr PSize   PFree   DevSize PV UUID
      /dev/drbd0      lvm2 a-   186.26g 186.26g 186.26g IeZ1rK-UMhp-bLqH-waSM-PZ3t-afJU-Wbvy2G
      
    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed
      
    piing@piing-node-2:~$ sudo vgcreate -v iscsivg01 /dev/drbd0
        Wiping cache of LVM-capable devices
        Wiping cache of LVM-capable devices
        Adding physical volume '/dev/drbd0' to volume group 'iscsivg01'
        Archiving volume group "iscsivg01" metadata (seqno 0).
        Creating volume group backup "/etc/lvm/backup/iscsivg01" (seqno 1).
      Volume group "iscsivg01" successfully created
      
    piing@piing-node-2:~$ sudo pvs -v
        Scanning for physical volume names
        Wiping cache of LVM-capable devices
      PV         VG   Fmt  Attr PSize   PFree   DevSize PV UUID
      **/dev/sda8       lvm2 a-   186.26g 186.26g 186.26g IeZ1rK-UMhp-bLqH-waSM-PZ3t-afJU-Wbvy2G**
      
    piing@piing-node-2:~$ sudo lvcreate -v -L 186G -n lun0 iscsivg01 /dev/drbd0
        Setting logging type to disk
        Finding volume group "iscsivg01"
      **Physical Volume "/dev/drbd0" not found in Volume Group "iscsivg01"**
      

Finally, I tried re-creating it as cluster-aware volume. Which left me unable to even do anything with it, until I had to zero out the DRBD disk and start over. :(


    piing@piing-node-2:~$ sudo vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1035         Processing: vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1038         O_DIRECT will be used
    #config/config.c:987       Setting global/locking_type to 1
    #config/config.c:987       Setting global/wait_for_locks to 1
    #locking/locking.c:240       File-based locking selected.
    #config/config.c:964       Setting global/locking_dir to /var/lock/lvm
    #config/config.c:992       metadata/pvmetadatasize not found in config: defaulting to 255
    #config/config.c:992       metadata/pvmetadatacopies not found in config: defaulting to 1
    #locking/file_locking.c:235       Locking /var/lock/lvm/V_iscsivg01 WB
    #locking/file_locking.c:141         _do_flock /var/lock/lvm/V_iscsivg01:aux WB
    #locking/file_locking.c:141         _do_flock /var/lock/lvm/V_iscsivg01 WB
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/V_iscsivg01:aux
    #device/dev-io.c:487         Opened /dev/ram0 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram0: block size is 4096 bytes
    #label/label.c:184       /dev/ram0: No label detected
    #device/dev-io.c:533         Closed /dev/ram0
    #device/dev-io.c:487         Opened /dev/ram1 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram1: block size is 4096 bytes
    #label/label.c:184       /dev/ram1: No label detected
    #device/dev-io.c:533         Closed /dev/ram1
    #device/dev-io.c:487         Opened /dev/sda1 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda1: block size is 4096 bytes
    #label/label.c:184       /dev/sda1: No label detected
    #device/dev-io.c:533         Closed /dev/sda1
    #device/dev-io.c:487         Opened /dev/ram2 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram2: block size is 4096 bytes
    #label/label.c:184       /dev/ram2: No label detected
    #device/dev-io.c:533         Closed /dev/ram2
    #device/dev-io.c:487         Opened /dev/ram3 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram3: block size is 4096 bytes
    #label/label.c:184       /dev/ram3: No label detected
    #device/dev-io.c:533         Closed /dev/ram3
    #device/dev-io.c:487         Opened /dev/ram4 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram4: block size is 4096 bytes
    #label/label.c:184       /dev/ram4: No label detected
    #device/dev-io.c:533         Closed /dev/ram4
    #device/dev-io.c:487         Opened /dev/ram5 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram5: block size is 4096 bytes
    #label/label.c:184       /dev/ram5: No label detected
    #device/dev-io.c:533         Closed /dev/ram5
    #device/dev-io.c:487         Opened /dev/sda5 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda5: block size is 4096 bytes
    #label/label.c:184       /dev/sda5: No label detected
    #device/dev-io.c:533         Closed /dev/sda5
    #device/dev-io.c:487         Opened /dev/ram6 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram6: block size is 4096 bytes
    #label/label.c:184       /dev/ram6: No label detected
    #device/dev-io.c:533         Closed /dev/ram6
    #device/dev-io.c:487         Opened /dev/sda6 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda6: block size is 4096 bytes
    #label/label.c:184       /dev/sda6: No label detected
    #device/dev-io.c:533         Closed /dev/sda6
    #device/dev-io.c:487         Opened /dev/ram7 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram7: block size is 4096 bytes
    #label/label.c:184       /dev/ram7: No label detected
    #device/dev-io.c:533         Closed /dev/ram7
    #device/dev-io.c:487         Opened /dev/sda7 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda7: block size is 4096 bytes
    #label/label.c:184       /dev/sda7: No label detected
    #device/dev-io.c:533         Closed /dev/sda7
    #device/dev-io.c:487         Opened /dev/ram8 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram8: block size is 4096 bytes
    #label/label.c:184       /dev/ram8: No label detected
    #device/dev-io.c:533         Closed /dev/ram8
    #device/dev-io.c:487         Opened /dev/sda8 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    #label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2)
    #format_text/format-text.c:1137         /dev/sda8: Found metadata at 4608 size 683 (in area at 4096 size 192512) for iscsivg01 (ebjnVL-uTLR-ixTo-9ySB-fY5d-0uhF-CTYvyu)
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG iscsivg01 with 1 mdas
    #cache/lvmcache.c:923         lvmcache: /dev/sda8: setting iscsivg01 VGID to ebjnVLuTLRixTo9ySBfY5d0uhFCTYvyu
    #cache/lvmcache.c:1173         lvmcache: /dev/sda8: VG iscsivg01: Set creation host to piing-node-2.
    #device/dev-io.c:487         Opened /dev/ram9 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram9: block size is 4096 bytes
    #label/label.c:184       /dev/ram9: No label detected
    #device/dev-io.c:533         Closed /dev/ram9
    #device/dev-io.c:487         Opened /dev/ram10 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram10: block size is 4096 bytes
    #label/label.c:184       /dev/ram10: No label detected
    #device/dev-io.c:533         Closed /dev/ram10
    #device/dev-io.c:487         Opened /dev/ram11 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram11: block size is 4096 bytes
    #label/label.c:184       /dev/ram11: No label detected
    #device/dev-io.c:533         Closed /dev/ram11
    #device/dev-io.c:487         Opened /dev/ram12 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram12: block size is 4096 bytes
    #label/label.c:184       /dev/ram12: No label detected
    #device/dev-io.c:533         Closed /dev/ram12
    #device/dev-io.c:487         Opened /dev/ram13 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram13: block size is 4096 bytes
    #label/label.c:184       /dev/ram13: No label detected
    #device/dev-io.c:533         Closed /dev/ram13
    #device/dev-io.c:487         Opened /dev/ram14 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram14: block size is 4096 bytes
    #label/label.c:184       /dev/ram14: No label detected
    #device/dev-io.c:533         Closed /dev/ram14
    #device/dev-io.c:487         Opened /dev/ram15 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram15: block size is 4096 bytes
    #label/label.c:184       /dev/ram15: No label detected
    #device/dev-io.c:533         Closed /dev/ram15
    #label/label.c:270         Using cached label for /dev/sda8
    #locking/file_locking.c:74       Unlocking /var/lock/lvm/V_iscsivg01
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/V_iscsivg01
    #device/dev-io.c:533         Closed /dev/sda8
    #vgcreate.c:60   A volume group called iscsivg01 already exists.
    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed

    piing@piing-node-2:~$ sudo vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1035         Processing: vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1038         O_DIRECT will be used
    ..................................................................................
    #device/dev-io.c:533         Closed /dev/ram8
    #device/dev-io.c:487         Opened /dev/sda8 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    #label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2)
    #device/dev-io.c:533         Closed /dev/sda8
    #device/dev-io.c:487         Opened /dev/ram9 RO O_DIRECT
    ..................................................................................
    #filters/filter-regex.c:172         /dev/ram8: Skipping (regex)
    #filters/filter-regex.c:172         /dev/sda8: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram9: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram10: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram11: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram12: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram13: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram14: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram15: Skipping (regex)
    #metadata/metadata.c:177     Adding physical volume '/dev/lvm/drbd0' to volume group 'iscsivg01'
    #format_text/archiver.c:127     Archiving volume group "iscsivg01" metadata (seqno 0).
    #metadata/pv_manip.c:296         /dev/lvm/drbd0 0:      0  47683: NULL(0:0)
    #device/dev-io.c:533         Closed /dev/lvm/drbd0
    #device/dev-io.c:487         Opened /dev/lvm/drbd0 RW O_DIRECT
    #device/dev-io.c:134         /dev/lvm/drbd0: block size is 4096 bytes
    #format_text/format-text.c:590         Writing iscsivg01 metadata to /dev/lvm/drbd0 at 4608 len 696
    #format_text/format-text.c:695         Pre-Committing iscsivg01 metadata (1) to /dev/lvm/drbd0 header at 4096
    #format_text/format-text.c:695         Committing iscsivg01 metadata (1) to /dev/lvm/drbd0 header at 4096
    #cache/lvmcache.c:1136         lvmcache: /dev/lvm/drbd0: now in VG iscsivg01 with 1 mdas
    #cache/lvmcache.c:923         lvmcache: /dev/lvm/drbd0: setting iscsivg01 VGID to B9aqVzzZD3t57nCZt9JEB9jXN5D2eB22
    #locking/file_locking.c:74       Unlocking /var/lock/lvm/P_orphans
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/P_orphans
    #locking/file_locking.c:74       Unlo
Error with Ubuntu HA+LVM2+DRBD: VG creation refers to wrong device
===

Currently I am working on a system with 2 servers  and have to set up iSCSI target service over LVM volume over DRBD. These are basic configuration details:

 - Architecture: i386 32-bit
 - Disk: VM Disk over SATA RAID-1
 - OS: Ubuntu 12.04
 - DRBD version: 8.3.11
 - DRBD base file system: ext4
 - LVM2 version: 2.02.66
 - cLVM version: 2.02.66
 - Cluster infrastructure: pacemaker+heartbeat
 - Pacemaker version: 1.1.6
 - Heartbeat version: 1:3.0.5

I created DRBD disk, created crm resources to control master-slave DRBD activation between 2 servers, synced the data, and tried setting up PV, VG and LV over it from the Primary node. This is the result:


    piing@piing-node-2:~$ sudo lvcreate -L 186G -n lun0 iscsivg01
     device-mapper: reload ioctl failed: Invalid argument
      Aborting. Failed to activate new LV to wipe the start of it.
      Node /dev/mapper/iscsivg01-lun0 was not removed by udev. Falling back to direct node removal.
      
    piing@piing-node-2:~$ sudo lvcreate -vvvv -L 186G -n lun0 iscsivg01
    #lvmcmdline.c:1035         Processing: lvcreate -vvvv -L 186G -n lun0 iscsivg01
    #lvmcmdline.c:1038         O_DIRECT will be used
    #config/config.c:987       Setting global/locking_type to 1
    #config/config.c:987       Setting global/wait_for_locks to 1
    #locking/locking.c:240       File-based locking selected.
    #config/config.c:964       Setting global/locking_dir to /var/lock/lvm
    #activate/activate.c:361       Getting target version for linear
    #ioctl/libdm-iface.c:1821         dm version   OF   [16384]
    #ioctl/libdm-iface.c:1821         dm versions   OF   [16384]
    #activate/activate.c:361       Getting target version for striped
    #ioctl/libdm-iface.c:1821         dm versions   OF   [16384]
    ..................................................................................
    #device/dev-io.c:533         Closed /dev/ram8
    #device/dev-io.c:487         Opened /dev/sda8 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    **#label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2)
    #format_text/format-text.c:1137         /dev/sda8: Found metadata at 6656 size 678 (in area at 4096 size 192512) for iscsivg01 (JuQJwx-pH4y-vuGj-uo82-viH0-B0DH-0Do7dx)
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG iscsivg01 with 1 mdas**
    #cache/lvmcache.c:923         lvmcache: /dev/sda8: setting iscsivg01 VGID to JuQJwxpH4yvuGjuo82viH0B0DH0Do7dx
    #cache/lvmcache.c:1173         lvmcache: /dev/sda8: VG iscsivg01: Set creation host to piing-node-2.
    #device/dev-io.c:487         Opened /dev/ram9 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram9: block size is 4096 bytes
    #label/label.c:184       /dev/ram9: No label detected
    #device/dev-io.c:533         Closed /dev/ram9
    ...................................................................................
    #label/label.c:270         Using cached label for /dev/sda8
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (3) from /dev/sda8 at 6656 size 678
    #metadata/pv_manip.c:296         /dev/sda8 0:      0  47683: NULL(0:0)
    #format_text/archiver.c:127     Archiving volume group "iscsivg01" metadata (seqno 3).
    #metadata/lv_manip.c:2042     Creating logical volume lun0
    #metadata/pv_map.c:55         Allowing allocation on /dev/sda8 start PE 0 length 47683
    #metadata/lv_manip.c:1366         Trying allocation using contiguous policy.  Need 47616 extents for 1 parallel areas and 0 log areas of 0 extents. (Total 47616 extents.)
    #metadata/lv_manip.c:1230         Trying allocation area 0 on /dev/sda8 start PE 0 length 47616 leaving 67.
    #metadata/lv_manip.c:795         Allocating parallel area 0 on /dev/sda8 start PE 0 length 47616.
    #metadata/pv_manip.c:296         /dev/sda8 0:      0  47616: lun0(0:0)
    #metadata/pv_manip.c:296         /dev/sda8 1:  47616     67: NULL(0:0)
    #device/dev-io.c:533         Closed /dev/sda8
    #device/dev-io.c:487         Opened /dev/sda8 RW O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    #format_text/format-text.c:590         Writing iscsivg01 metadata to /dev/sda8 at 7680 len 941
    #format_text/format-text.c:695         Pre-Committing iscsivg01 metadata (4) to /dev/sda8 header at 4096
    #format_text/format-text.c:695         Committing iscsivg01 metadata (4) to /dev/sda8 header at 4096
    #format_text/archiver.c:390     Creating volume group backup "/etc/lvm/backup/iscsivg01" (seqno 4).
    #format_text/format-text.c:888         Writing iscsivg01 metadata to /etc/lvm/backup/.lvm_piing-node-2_14473_2072241393
    #format_text/format-text.c:932         Committing iscsivg01 metadata (4)
    #format_text/format-text.c:933         Renaming /etc/lvm/backup/iscsivg01.tmp to /etc/lvm/backup/iscsivg01
    #locking/file_locking.c:291       Locking LV JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy (R)
    #metadata/metadata.c:3091       Finding volume group for uuid JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2) with 1 mdas
    #format_text/format-text.c:1137         /dev/sda8: Found metadata at 7680 size 941 (in area at 4096 size 192512) for iscsivg01 (JuQJwx-pH4y-vuGj-uo82-viH0-B0DH-0Do7dx)
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG iscsivg01 with 1 mdas
    #cache/lvmcache.c:923         lvmcache: /dev/sda8: setting iscsivg01 VGID to JuQJwxpH4yvuGjuo82viH0B0DH0Do7dx
    #cache/lvmcache.c:1173         lvmcache: /dev/sda8: VG iscsivg01: Set creation host to piing-node-2.
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (4) from /dev/sda8 at 7680 size 941
    #metadata/metadata.c:3097     Found volume group "iscsivg01"
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #device/dev-io.c:291       /dev/sda8: read_ahead is 256 sectors
    #mm/memlock.c:265       Locking memory
    #mm/memlock.c:176         mlock        764KiB      8048000 -      8107000 r-xp  00000000 08:05 265000     /sbin/lvm
    ..................................................................................
    #mm/memlock.c:232         Locked 14680064 bytes
    #mm/memlock.c:318         memlock_count inc to 1
    #activate/dev_manager.c:827         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #activate/dev_manager.c:827         Getting device info for iscsivg01-lun0-real [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-real]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-real OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-real OF   [16384]
    #activate/dev_manager.c:827         Getting device info for iscsivg01-lun0-cow [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-cow]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-cow OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy-cow OF   [16384]
    #activate/dev_manager.c:1135         Checking kernel supports striped segment type for lun0
    #metadata/metadata.c:2160         Calculated readahead of LV lun0 is 256
    #libdm-deptree.c:1268     Creating iscsivg01-lun0
    #ioctl/libdm-iface.c:1821         dm create iscsivg01-lun0 LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #libdm-common.c:799         iscsivg01-lun0: Stacking NODE_ADD (252,1) 0:6 0660
    #libdm-deptree.c:1601     Loading iscsivg01-lun0 table (252:1)
    **#libdm-deptree.c:1547         Adding target to (252:1): 0 390070272 linear 8:8 384
    #ioctl/libdm-iface.c:1821         dm table   (252:1) OF   [16384]
    #ioctl/libdm-iface.c:1821         dm reload   (252:1) NF   [16384]
    #ioctl/libdm-iface.c:1838   device-mapper: reload ioctl failed: Invalid argument**
    #libdm-deptree.c:1686         <backtrace>
    #activate/dev_manager.c:1471         <backtrace>
    #activate/dev_manager.c:1506         <backtrace>
    #activate/activate.c:571         <backtrace>
    #activate/activate.c:1128         <backtrace>
    #mm/memlock.c:282       Unlocking memory
    #mm/memlock.c:176         munlock        764KiB      8048000 -      8107000 r-xp  00000000 08:05 265000     /sbin/lvm
    ..................................................................................
    #mm/memlock.c:176         munlock        268KiB     bfb20000 -     bfb63000 rw-p  00000000 00:00 0          [stack]
    #mm/memlock.c:232         Unlocked 14405632 bytes
    #mm/memlock.c:327         memlock_count dec to 0
    #libdm-common.c:475         Created /dev/mapper/iscsivg01-lun0
    #activate/activate.c:1155         <backtrace>
    #locking/locking.c:389         <backtrace>
    #metadata/lv_manip.c:3230   Aborting. Failed to activate new LV to wipe the start of it.
    #locking/file_locking.c:286       Locking LV JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy (NL)
    #metadata/metadata.c:3091       Finding volume group for uuid JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #label/label.c:270         Using cached label for /dev/sda8
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (4) from /dev/sda8 at 7680 size 941
    #metadata/metadata.c:3097     Found volume group "iscsivg01"
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #mm/memlock.c:265       Locking memory
    #mm/memlock.c:176         mlock        764KiB      8048000 -      8107000 r-xp  00000000 08:05 265000     /sbin/lvm
    #mm/memlock.c:176         mlock          8KiB      8107000 -      8109000 r--p  000bf000 08:05 265000     /sbin/lvm
    #mm/memlock.c:176         mlock         12KiB      8109000 -      810c000 rw-p  000c1000 08:05 265000     /sbin/lvm
    #mm/memlock.c:176         mlock         32KiB      810c000 -      8114000 rw-p  00000000 00:00 0
    #mm/memlock.c:176         mlock      10500KiB      8ece000 -      990f000 rw-p  00000000 00:00 0          [heap]
    #mm/memlock.c:158         mlock default filter 'locale/locale-archive' matches 'b7282000-b7482000 r--p 00000000 08:05 400424     /usr/lib/locale/locale-archive': Skipping.
    ..................................................................................
    #mm/memlock.c:232         Unlocked 14405632 bytes
    #mm/memlock.c:327         memlock_count dec to 0
    [COLOR="Red"]  Node /dev/mapper/iscsivg01-lun0 was not removed by udev. Falling back to direct node removal.[/COLOR]
    #libdm-common.c:503         Removed /dev/mapper/iscsivg01-lun0
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy OF   [16384]
    #locking/file_locking.c:281       Unlocking LV JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #metadata/metadata.c:3091       Finding volume group for uuid JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy
    #label/label.c:270         Using cached label for /dev/sda8
    #label/label.c:270         Using cached label for /dev/sda8
    #format_text/format-text.c:498         Read iscsivg01 metadata (4) from /dev/sda8 at 7680 size 941
    #metadata/metadata.c:3097     Found volume group "iscsivg01"
    #activate/dev_manager.c:236         Getting device info for iscsivg01-lun0 [LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy]
    #ioctl/libdm-iface.c:1821         dm info  LVM-JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #ioctl/libdm-iface.c:1821         dm info  JuQJwxpH4yvuGjuo82viH0B0DH0Do7dxpiezeL2gMmflbxUbpOYw8sD2KPSIraMy NF   [16384]
    #activate/activate.c:938         <backtrace>
    #metadata/pv_manip.c:296         /dev/sda8 0:      0  47616: NULL(0:0)
    #metadata/pv_manip.c:296         /dev/sda8 1:  47616     67: NULL(0:0)
    #format_text/format-text.c:590         Writing iscsivg01 metadata to /dev/sda8 at 8704 len 678
    #format_text/format-text.c:695         Pre-Committing iscsivg01 metadata (5) to /dev/sda8 header at 4096
    #format_text/format-text.c:695         Committing iscsivg01 metadata (5) to /dev/sda8 header at 4096
    #format_text/archiver.c:390     Creating volume group backup "/etc/lvm/backup/iscsivg01" (seqno 5).
    #format_text/format-text.c:888         Writing iscsivg01 metadata to /etc/lvm/backup/.lvm_piing-node-2_14473_712565374
    #format_text/format-text.c:932         Committing iscsivg01 metadata (5)
    #format_text/format-text.c:933         Renaming /etc/lvm/backup/iscsivg01.tmp to /etc/lvm/backup/iscsivg01
    #lvcreate.c:540         <backtrace>
    #locking/file_locking.c:74       Unlocking /var/lock/lvm/V_iscsivg01
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/V_iscsivg01
    #device/dev-io.c:533         Closed /dev/sda8


I've searched for the solution and tried many things: tweaking filter in lvm.conf, partitioning the drbd device, etc.. but none worked. Moreover, I've actually set up such system successfully just a week before. But now when the servers had to be re-configured with RAID-1 and re-installed, the old settings stopped working. 

This is the filter line I'm using:

	filter = [ "a|/dev/lvm/drbd|", "a|/dev/drbd|", "r/.*/" ]


While investigating further, I found that when vgcreate command was used to create Volume Group over Physical Volume already sitting on DRBD device ( /dev/drbd0, /dev/lvm/drbd0 ), the device reference from pvs fell to ( /dev/sda8 ), the device underlying DRBD. Even weirder is that result from pvscan still showed as DRBD device.

    piing@piing-node-2:~$ sudo vgs
      No volume groups found
      
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      No volume groups found
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/drbd0      lvm2 a-   186.26g 186.26g
      
    piing@piing-node-2:~$ sudo pvscan -v
        Wiping cache of LVM-capable devices
        Wiping internal VG cache
        Walking through all physical volumes
      PV /dev/drbd0   VG iscsivg01   lvm2 [186.26 GiB / 186.26 GiB free]
      Total: 1 [186.26 GiB] / in use: 1 [186.26 GiB] / in no VG: 0 [0   ]
      ]
      
    piing@piing-node-2:~$ sudo vgcreate iscsivg01 /dev/drbd0
      Volume group "iscsivg01" successfully created
      
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      Found volume group "iscsivg01" using metadata type lvm2
      
    piing@piing-node-2:~$ sudo vgs
      VG        #PV #LV #SN Attr   VSize   VFree
      iscsivg01   1   0   0 wz--n- 186.26g 186.26g
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG        Fmt  Attr PSize   PFree
      **/dev/sda8  iscsivg01 lvm2 a-   186.26g 186.26g**
      
    piing@piing-node-2:~$ sudo pvscan -v
        Wiping cache of LVM-capable devices
        Wiping internal VG cache
        Walking through all physical volumes
      **PV /dev/drbd0   VG iscsivg01   lvm2 [186.26 GiB / 186.26 GiB free]**
      Total: 1 [186.26 GiB] / in use: 1 [186.26 GiB] / in no VG: 0 [0   ]
      ]
      

If I then removed the VG, result from pvs went back to showing DRBD device. Then I created VG again, and it becomes /dev/sda8 again.

    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed
      
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      No volume groups found
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/drbd0      lvm2 a-   186.26g 186.26g
      
    piing@piing-node-2:~$ sudo vgcreate iscsivg01 /dev/drbd0
      Volume group "iscsivg01" successfully created
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG        Fmt  Attr PSize   PFree
      **/dev/sda8  iscsivg01 lvm2 a-   186.26g 186.26g**
    piing@piing-node-2:~$ sudo vgscan
      Reading all physical volumes.  This may take a while...
      Found volume group "iscsivg01" using metadata type lvm2
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG        Fmt  Attr PSize   PFree
      **/dev/sda8  iscsivg01 lvm2 a-   186.26g 186.26g**
      
    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed
      
    piing@piing-node-2:~$ sudo pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/drbd0      lvm2 a-   186.26g 186.26g
      

And the system doesn't even recognize ( /dev/drbd0 ) or ( /dev/lvm/drbd0 ) to be in volume group ( iscsivg01 ) when directly addressed.

    piing@piing-node-2:~$ sudo pvs -v /dev/drbd0
        Using physical volume(s) on command line
        Wiping cache of LVM-capable devices
      PV             VG   Fmt  Attr PSize   PFree   DevSize PV UUID
      /dev/drbd0      lvm2 a-   186.26g 186.26g 186.26g IeZ1rK-UMhp-bLqH-waSM-PZ3t-afJU-Wbvy2G
      
    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed
      
    piing@piing-node-2:~$ sudo vgcreate -v iscsivg01 /dev/drbd0
        Wiping cache of LVM-capable devices
        Wiping cache of LVM-capable devices
        Adding physical volume '/dev/drbd0' to volume group 'iscsivg01'
        Archiving volume group "iscsivg01" metadata (seqno 0).
        Creating volume group backup "/etc/lvm/backup/iscsivg01" (seqno 1).
      Volume group "iscsivg01" successfully created
      
    piing@piing-node-2:~$ sudo pvs -v
        Scanning for physical volume names
        Wiping cache of LVM-capable devices
      PV         VG   Fmt  Attr PSize   PFree   DevSize PV UUID
      **/dev/sda8       lvm2 a-   186.26g 186.26g 186.26g IeZ1rK-UMhp-bLqH-waSM-PZ3t-afJU-Wbvy2G**
      
    piing@piing-node-2:~$ sudo lvcreate -v -L 186G -n lun0 iscsivg01 /dev/drbd0
        Setting logging type to disk
        Finding volume group "iscsivg01"
      **Physical Volume "/dev/drbd0" not found in Volume Group "iscsivg01"**
      

Finally, I tried re-creating it as cluster-aware volume. Which left me unable to even do anything with it, until I had to zero out the DRBD disk and start over. :(


    piing@piing-node-2:~$ sudo vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1035         Processing: vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1038         O_DIRECT will be used
    #config/config.c:987       Setting global/locking_type to 1
    #config/config.c:987       Setting global/wait_for_locks to 1
    #locking/locking.c:240       File-based locking selected.
    #config/config.c:964       Setting global/locking_dir to /var/lock/lvm
    #config/config.c:992       metadata/pvmetadatasize not found in config: defaulting to 255
    #config/config.c:992       metadata/pvmetadatacopies not found in config: defaulting to 1
    #locking/file_locking.c:235       Locking /var/lock/lvm/V_iscsivg01 WB
    #locking/file_locking.c:141         _do_flock /var/lock/lvm/V_iscsivg01:aux WB
    #locking/file_locking.c:141         _do_flock /var/lock/lvm/V_iscsivg01 WB
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/V_iscsivg01:aux
    #device/dev-io.c:487         Opened /dev/ram0 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram0: block size is 4096 bytes
    #label/label.c:184       /dev/ram0: No label detected
    #device/dev-io.c:533         Closed /dev/ram0
    #device/dev-io.c:487         Opened /dev/ram1 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram1: block size is 4096 bytes
    #label/label.c:184       /dev/ram1: No label detected
    #device/dev-io.c:533         Closed /dev/ram1
    #device/dev-io.c:487         Opened /dev/sda1 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda1: block size is 4096 bytes
    #label/label.c:184       /dev/sda1: No label detected
    #device/dev-io.c:533         Closed /dev/sda1
    #device/dev-io.c:487         Opened /dev/ram2 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram2: block size is 4096 bytes
    #label/label.c:184       /dev/ram2: No label detected
    #device/dev-io.c:533         Closed /dev/ram2
    #device/dev-io.c:487         Opened /dev/ram3 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram3: block size is 4096 bytes
    #label/label.c:184       /dev/ram3: No label detected
    #device/dev-io.c:533         Closed /dev/ram3
    #device/dev-io.c:487         Opened /dev/ram4 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram4: block size is 4096 bytes
    #label/label.c:184       /dev/ram4: No label detected
    #device/dev-io.c:533         Closed /dev/ram4
    #device/dev-io.c:487         Opened /dev/ram5 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram5: block size is 4096 bytes
    #label/label.c:184       /dev/ram5: No label detected
    #device/dev-io.c:533         Closed /dev/ram5
    #device/dev-io.c:487         Opened /dev/sda5 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda5: block size is 4096 bytes
    #label/label.c:184       /dev/sda5: No label detected
    #device/dev-io.c:533         Closed /dev/sda5
    #device/dev-io.c:487         Opened /dev/ram6 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram6: block size is 4096 bytes
    #label/label.c:184       /dev/ram6: No label detected
    #device/dev-io.c:533         Closed /dev/ram6
    #device/dev-io.c:487         Opened /dev/sda6 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda6: block size is 4096 bytes
    #label/label.c:184       /dev/sda6: No label detected
    #device/dev-io.c:533         Closed /dev/sda6
    #device/dev-io.c:487         Opened /dev/ram7 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram7: block size is 4096 bytes
    #label/label.c:184       /dev/ram7: No label detected
    #device/dev-io.c:533         Closed /dev/ram7
    #device/dev-io.c:487         Opened /dev/sda7 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda7: block size is 4096 bytes
    #label/label.c:184       /dev/sda7: No label detected
    #device/dev-io.c:533         Closed /dev/sda7
    #device/dev-io.c:487         Opened /dev/ram8 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram8: block size is 4096 bytes
    #label/label.c:184       /dev/ram8: No label detected
    #device/dev-io.c:533         Closed /dev/ram8
    #device/dev-io.c:487         Opened /dev/sda8 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    #label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2)
    #format_text/format-text.c:1137         /dev/sda8: Found metadata at 4608 size 683 (in area at 4096 size 192512) for iscsivg01 (ebjnVL-uTLR-ixTo-9ySB-fY5d-0uhF-CTYvyu)
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG iscsivg01 with 1 mdas
    #cache/lvmcache.c:923         lvmcache: /dev/sda8: setting iscsivg01 VGID to ebjnVLuTLRixTo9ySBfY5d0uhFCTYvyu
    #cache/lvmcache.c:1173         lvmcache: /dev/sda8: VG iscsivg01: Set creation host to piing-node-2.
    #device/dev-io.c:487         Opened /dev/ram9 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram9: block size is 4096 bytes
    #label/label.c:184       /dev/ram9: No label detected
    #device/dev-io.c:533         Closed /dev/ram9
    #device/dev-io.c:487         Opened /dev/ram10 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram10: block size is 4096 bytes
    #label/label.c:184       /dev/ram10: No label detected
    #device/dev-io.c:533         Closed /dev/ram10
    #device/dev-io.c:487         Opened /dev/ram11 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram11: block size is 4096 bytes
    #label/label.c:184       /dev/ram11: No label detected
    #device/dev-io.c:533         Closed /dev/ram11
    #device/dev-io.c:487         Opened /dev/ram12 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram12: block size is 4096 bytes
    #label/label.c:184       /dev/ram12: No label detected
    #device/dev-io.c:533         Closed /dev/ram12
    #device/dev-io.c:487         Opened /dev/ram13 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram13: block size is 4096 bytes
    #label/label.c:184       /dev/ram13: No label detected
    #device/dev-io.c:533         Closed /dev/ram13
    #device/dev-io.c:487         Opened /dev/ram14 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram14: block size is 4096 bytes
    #label/label.c:184       /dev/ram14: No label detected
    #device/dev-io.c:533         Closed /dev/ram14
    #device/dev-io.c:487         Opened /dev/ram15 RO O_DIRECT
    #device/dev-io.c:134         /dev/ram15: block size is 4096 bytes
    #label/label.c:184       /dev/ram15: No label detected
    #device/dev-io.c:533         Closed /dev/ram15
    #label/label.c:270         Using cached label for /dev/sda8
    #locking/file_locking.c:74       Unlocking /var/lock/lvm/V_iscsivg01
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/V_iscsivg01
    #device/dev-io.c:533         Closed /dev/sda8
    #vgcreate.c:60   A volume group called iscsivg01 already exists.
    piing@piing-node-2:~$ sudo vgremove iscsivg01
      Volume group "iscsivg01" successfully removed

    piing@piing-node-2:~$ sudo vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1035         Processing: vgcreate -vvvv -c y iscsivg01 /dev/lvm/drbd0
    #lvmcmdline.c:1038         O_DIRECT will be used
    ..................................................................................
    #device/dev-io.c:533         Closed /dev/ram8
    #device/dev-io.c:487         Opened /dev/sda8 RO O_DIRECT
    #device/dev-io.c:134         /dev/sda8: block size is 4096 bytes
    #label/label.c:160       /dev/sda8: lvm2 label detected
    #cache/lvmcache.c:1136         lvmcache: /dev/sda8: now in VG #orphans_lvm2 (#orphans_lvm2)
    #device/dev-io.c:533         Closed /dev/sda8
    #device/dev-io.c:487         Opened /dev/ram9 RO O_DIRECT
    ..................................................................................
    #filters/filter-regex.c:172         /dev/ram8: Skipping (regex)
    #filters/filter-regex.c:172         /dev/sda8: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram9: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram10: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram11: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram12: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram13: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram14: Skipping (regex)
    #filters/filter-regex.c:172         /dev/ram15: Skipping (regex)
    #metadata/metadata.c:177     Adding physical volume '/dev/lvm/drbd0' to volume group 'iscsivg01'
    #format_text/archiver.c:127     Archiving volume group "iscsivg01" metadata (seqno 0).
    #metadata/pv_manip.c:296         /dev/lvm/drbd0 0:      0  47683: NULL(0:0)
    #device/dev-io.c:533         Closed /dev/lvm/drbd0
    #device/dev-io.c:487         Opened /dev/lvm/drbd0 RW O_DIRECT
    #device/dev-io.c:134         /dev/lvm/drbd0: block size is 4096 bytes
    #format_text/format-text.c:590         Writing iscsivg01 metadata to /dev/lvm/drbd0 at 4608 len 696
    #format_text/format-text.c:695         Pre-Committing iscsivg01 metadata (1) to /dev/lvm/drbd0 header at 4096
    #format_text/format-text.c:695         Committing iscsivg01 metadata (1) to /dev/lvm/drbd0 header at 4096
    #cache/lvmcache.c:1136         lvmcache: /dev/lvm/drbd0: now in VG iscsivg01 with 1 mdas
    #cache/lvmcache.c:923         lvmcache: /dev/lvm/drbd0: setting iscsivg01 VGID to B9aqVzzZD3t57nCZt9JEB9jXN5D2eB22
    #locking/file_locking.c:74       Unlocking /var/lock/lvm/P_orphans
    #locking/file_locking.c:51         _undo_flock /var/lock/lvm/P_orphans
    #locking/file_locking.c:74       Unlo