Knowledge Base Article: 000544154

Kernel 3.10.0-1127.8.2.el7.x86_64( or above) panic on RHEL 7.8 with PowerPath 7.1 (000544154)
Primary Product : PowerPath for Linux
Product : PowerPath for Linux

Version: 5 Article Type: Break Fix Audience: Level 30 = Customers Last Published: Sun Aug 23 23:46:52 GMT 2020

Summary:
Issue: Environment:

OS: RHEL v7.8

Kernel: 3.10.0-1127.8.2.el7 or above.
DELL EMC SW: PowerPath v7.1 (build 75)

Issue:

Host unable to boot and crashes while booting (or starting PowerPath) with kernel 3.10.0-1127.8.2.el7 (or above) and PowerPath v7.1 but works fine
with the lower kernel 3.10.0-1127.el7.
Call Trace generated during the crash:

===========================================================================================
[ 497.393570] Modules linked in: emcpdm(POE) emcpgpx(POE) emcpmpx(POE) emcp(POE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs
lockd grace fscache sunrpc dm_mirror dm_region_hash dm_log dm_mod btrfs ext4 mbcache jbd2 raid6_pq xor intel_powerclamp coretemp kvm_intel
kvm irqbypass crc32_pclmul ipmi_ssif ghash_clmulni_intel iTCO_wdt aesni_intel iTCO_vendor_support gpio_ich lrw gf128mul glue_helper ablk_helper
ioatdma cryptd joydev pcspkr ipmi_si ipmi_devintf lpc_ich i7core_edac acpi_power_meter ipmi_msghandler wmi acpi_cpufreq sg sch_fq_codel
ip_tables xfs libcrc32c uas usb_storage sr_mod cdrom sd_mod ata_generic pata_acpi lpfc mgag200 qla2xxx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet e1000e libata crc_t10dif igb crct10dif_generic nvme_fc crct10dif_pclmul
[ 497.393959] nvme_fabrics crc32c_intel nvme_core megaraid_sas scsi_transport_fc dca qlge drm_panel_orientation_quirks i2c_algo_bit ptp scsi_tgt
crct10dif_common pps_core [last unloaded: emcpioc]
[ 497.394042] CPU: 1 PID: 0 Comm: swapper/1 Kdump: loaded Tainted: P OE ------------ 3.10.0-1127.8.2.el7.x86_64 #1
[ 497.394079] Hardware name: Cisco Systems Inc R210-2121605W/R210-2121605W, BIOS C200.1.4.3n.0.032320181054 03/23/2018
[ 497.394114] task: ffff8d27ddefe2a0 ti: ffff8d27ddf24000 task.ti: ffff8d27ddf24000
[ 497.394139] RIP: 0010:[<ffffffff8517da10>] [<ffffffff8517da10>] __blk_add_trace+0x20/0x340
[ 497.394176] RSP: 0018:ffff8d333f203b70 EFLAGS: 00010282
[ 497.394195] RAX: 5c74726f70722d30 RBX: ffff8d333a162ff8 RCX: 0000000000000000
[ 497.394219] RDX: 000000003e77f600 RSI: ffff8d333a163058 RDI: 5c74726f70722d30
[ 497.394243] RBP: ffff8d333f203bd0 R08: 0000000001800008 R09: 000000000000000f
[ 497.394268] R10: 5c74726f70722d30 R11: ffff8d333a163058 R12: ffff8d333e77f600
[ 497.394291] R13: ffff8d27ddefe2a0 R14: 0000000000000000 R15: 0000000000000000
[ 497.394316] FS: 0000000000000000(0000) GS:ffff8d333f200000(0000) knlGS:0000000000000000
[ 497.394343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 497.394363] CR2: 0000000002be3188 CR3: 000000183664e000 CR4: 00000000000207e0
[ 497.394387] Call Trace:
[ 497.394398] <IRQ>
[ 497.394412] [<ffffffff852266d2>] ? kmem_cache_free+0x1e2/0x200
[ 497.394436] [<ffffffff8517dd84>] blk_add_trace_rq.isra.10+0x54/0x90
[ 497.394478] [<ffffffffc0c40d3e>] PowerPlatformTopIodone+0x1ae/0x2a0 [emcp]
[ 497.394507] [<ffffffffc0c40e51>] PowerTopIodone+0x21/0x140 [emcp]
[ 497.394535] [<ffffffffc0c410e4>] PowerProcessTopIodonePirps+0x64/0xe0 [emcp]
[ 497.394565] [<ffffffffc0c4140c>] PowerBottomIodoneNew+0x2ac/0x5d0 [emcp]
[ 497.394593] [<ffffffffc0c41b52>] PowerPlatformBottomIodone+0xf2/0x220 [emcp]
[ 497.394622] [<ffffffff8528cbec>] bio_endio+0x8c/0x130
[ 497.394643] [<ffffffff85354b10>] blk_update_request+0x90/0x370
[ 497.394667] [<ffffffff854ec274>] scsi_end_request+0x34/0x1e0
[ 497.394688] [<ffffffff854ec5e8>] scsi_io_completion+0x168/0x720
[ 497.394714] [<ffffffff850a599e>] ? irq_exit+0x8e/0x110
[ 497.394738] [<ffffffff854e18dc>] scsi_finish_command+0xdc/0x140
[ 497.394759] [<ffffffff854ebb30>] scsi_softirq_done+0x130/0x160
[ 497.394784] [<ffffffff8535c496>] blk_done_softirq+0x96/0xc0
[ 497.394804] [<ffffffff850a5695>] __do_softirq+0xf5/0x280
[ 497.394830] [<ffffffff8579642c>] call_softirq+0x1c/0x30
[ 497.394853] [<ffffffff8502f715>] do_softirq+0x65/0xa0
[ 497.394871] [<ffffffff850a5a15>] irq_exit+0x105/0x110
[ 497.394891] [<ffffffff85797876>] do_IRQ+0x56/0xf0
[ 497.395735] [<ffffffff8578936a>] common_interrupt+0x16a/0x16a
[ 497.396568] <EOI>
[ 497.396581] [<ffffffff855c6174>] ? cpuidle_enter_state+0x54/0xd0
[ 497.398234] [<ffffffff855c62ce>] cpuidle_idle_call+0xde/0x230
[ 497.399066] [<ffffffff85037c6e>] arch_cpu_idle+0xe/0xc0
[ 497.399875] [<ffffffff85101c2a>] cpu_startup_entry+0x14a/0x1e0
[ 497.400666] [<ffffffff8505a517>] start_secondary+0x1f7/0x270
[ 497.401431] [<ffffffff850000d5>] start_cpu+0x5/0x14
[ 497.402167] Code: fe ff ff 0f 1f 84 00 00 00 00 00 55 49 89 fa 49 89 f3 48 89 e5 41 57 41 56 41 55 65 4c 8b 2c 25 c0 0e 01 00 41 54 53 48 83 ec 38
<83> 3f 02 89 55 d0 44 89 4d cc 44 0f b6 3d 36 2c be 00 0f 85 7d
[ 497.403771] RIP [<ffffffff8517da10>] __blk_add_trace+0x20/0x340
[ 497.404506] RSP <ffff8d333f203b70>
===========================================================================================

Cause: As per the Engineering ticket PPLI-2068, the RCA details mentions this:
 blk_add_trace_rq signature is changed

Change: Installing/Starting PowerPath v7.1 on kernel 3.10.0-1127.8.2.el7 or above

Resolution:
The Issue will be fixed in PowerPath v7.2 (build 65) (Expected to be GA in the Q3 2020. With PowerPath 7.1, do not go above kernel Kernel
3.10.0-1127.el7.x86_64.

This KB will be updated when PowerPath 7.2 for Linux is released.

NOTE: This issue has also been resolved by upgrading to PP 7.1 P02 for linux. ( Available now).

Primary Product: PowerPath for Linux

Product: PowerPath for Linux