igmpproxy crashs IPFire kernel since core-update 183

[tscholz]

Hi there

Firstly, thanks for the app. It's been my favorite for watching IPTV behind my firewall for years, no problems.

While I understand that this platform may not serve as a bug tracker for other appliances, I believe there is an issue between igmpproxy and certain versions of the Linux kernel. That's why I am reporting the issue here. Please feel free to close this issue if I'm wrong.

I recently upgraded my IPFire installation to core-update 183, 184 and now 185, and since 183, igmpproxy has been crashing the kernel. Here is a snippet of the error log:

Apr 18 00:55:58 ipfire kernel: virtio_net virtio0 green0: entered allmulticast mode
Apr 18 00:55:58 ipfire kernel: virtio_net virtio2 red0: entered allmulticast mode

Apr 18 00:56:26 ipfire kernel: BUG: kernel NULL pointer dereference, address: 000000000000009a
Apr 18 00:56:26 ipfire kernel: #PF: supervisor read access in kernel mode
Apr 18 00:56:26 ipfire kernel: #PF: error_code(0x0000) - not-present page
Apr 18 00:56:26 ipfire kernel: PGD 1048d1067 P4D 1048d1067 PUD 10f1a7067 PMD 0
Apr 18 00:56:26 ipfire kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
Apr 18 00:56:26 ipfire kernel: CPU: 0 PID: 12795 Comm: igmpproxy Not tainted 6.6.15-ipfire #1
Apr 18 00:56:26 ipfire kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 18 00:56:26 ipfire kernel: RIP: 0010:ip_mr_forward+0xde/0x3a0
Apr 18 00:56:26 ipfire kernel: Code: 48 8d 7e 01 48 8d 04 fd 00 00 00 00 48 29 f8 48 c1 e0 04 48 01 e8 48 8b 00 49 39 c5 0f 84 d9 00 00 00 49 8b 46 58 48 83 e0 fe <80> b8 9a 00 00 00 00 75 2c 45 85 ff 0f 84 a4 00 00 00 48 83 c4 18
Apr 18 00:56:26 ipfire kernel: RSP: 0018:ffffad418287fbe0 EFLAGS: 00010246
Apr 18 00:56:26 ipfire kernel: RAX: 0000000000000000 RBX: ffff99fc42a9bcc0 RCX: 0000000000000001
Apr 18 00:56:26 ipfire kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002
Apr 18 00:56:26 ipfire kernel: RBP: ffff99fc4156e000 R08: 0000000000000000 R09: 0000000000000001
Apr 18 00:56:26 ipfire kernel: R10: ffff99fc42a9bcc0 R11: 0000000000000000 R12: ffffffffaa5d1f80
Apr 18 00:56:26 ipfire kernel: R13: ffff99fc47412000 R14: ffff99fc42da8b00 R15: 0000000000000000
Apr 18 00:56:26 ipfire kernel: FS: 00007863f4dbc740(0000) GS:ffff99fd57c00000(0000) knlGS:0000000000000000
Apr 18 00:56:26 ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 18 00:56:26 ipfire kernel: CR2: 000000000000009a CR3: 000000010be38005 CR4: 0000000000770ef0
Apr 18 00:56:26 ipfire kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 18 00:56:26 ipfire kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 18 00:56:26 ipfire kernel: PKRU: 55555554
Apr 18 00:56:26 ipfire kernel: Call Trace:
Apr 18 00:56:26 ipfire kernel:
Apr 18 00:56:26 ipfire kernel: ? __die+0x23/0x80
Apr 18 00:56:26 ipfire kernel: ? page_fault_oops+0x171/0x4e0
Apr 18 00:56:26 ipfire kernel: ? exc_page_fault+0x42c/0x730
Apr 18 00:56:26 ipfire kernel: ? asm_exc_page_fault+0x26/0x30
Apr 18 00:56:26 ipfire kernel: ? ip_mr_forward+0xde/0x3a0
Apr 18 00:56:26 ipfire kernel: ? _raw_spin_unlock_irqrestore+0x24/0x60
Apr 18 00:56:26 ipfire kernel: ipmr_mfc_add+0x656/0x900
Apr 18 00:56:26 ipfire kernel: ? __skb_datagram_iter+0x7d/0x350
Apr 18 00:56:26 ipfire kernel: ip_mroute_setsockopt+0x46c/0x680
Apr 18 00:56:26 ipfire kernel: do_ip_setsockopt+0x1bb/0x12b0
Apr 18 00:56:26 ipfire kernel: ip_setsockopt+0x33/0xb0
Apr 18 00:56:26 ipfire kernel: __sys_setsockopt+0x87/0x130
Apr 18 00:56:26 ipfire kernel: __x64_sys_setsockopt+0x1f/0x40
Apr 18 00:56:26 ipfire kernel: do_syscall_64+0x5d/0x90
Apr 18 00:56:26 ipfire kernel: ? exit_to_user_mode_prepare+0x1a/0x130
Apr 18 00:56:26 ipfire kernel: ? syscall_exit_to_user_mode+0x37/0x50
Apr 18 00:56:26 ipfire kernel: ? do_syscall_64+0x6c/0x90
Apr 18 00:56:26 ipfire kernel: ? do_syscall_64+0x6c/0x90
Apr 18 00:56:26 ipfire kernel: ? syscall_exit_to_user_mode+0x37/0x50
Apr 18 00:56:26 ipfire kernel: ? do_sync_core+0x2c/0x40
Apr 18 00:56:26 ipfire kernel: ? __flush_smp_call_function_queue+0xa9/0x430
Apr 18 00:56:26 ipfire kernel: ? exit_to_user_mode_prepare+0x1a/0x130
Apr 18 00:56:26 ipfire kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Apr 18 00:56:26 ipfire kernel: RIP: 0033:0x7863f4ecd6de
Apr 18 00:56:26 ipfire kernel: Code: 0f 1f 40 00 48 8b 15 39 c7 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 49 89 ca b8 36 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 0a c3 66 0f 1f 84 00 00 00 00 00 48 8b 15 01
Apr 18 00:56:26 ipfire kernel: RSP: 002b:00007fff0dddfa48 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
Apr 18 00:56:26 ipfire kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007863f4ecd6de
Apr 18 00:56:26 ipfire kernel: RDX: 00000000000000cc RSI: 0000000000000000 RDI: 0000000000000003
Apr 18 00:56:26 ipfire kernel: RBP: 0000000001f9f8f4 R08: 000000000000003c R09: 0000000000000000
Apr 18 00:56:26 ipfire kernel: R10: 00007fff0dddfa50 R11: 0000000000000246 R12: 00007fff0dddfab0
Apr 18 00:56:26 ipfire kernel: R13: 0000000000409ac7 R14: 0000000000000005 R15: 0000000000000000
Apr 18 00:56:26 ipfire kernel:
Apr 18 00:56:26 ipfire kernel: Modules linked in: tun nfnetlink_queue xt_NFQUEUE xt_MASQUERADE cfg80211 rfkill 8021q garp xt_set ip_set_hash_net ip_set xt_hashlimit xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_mangle iptable_filter vfat fat sch_cake intel_rapl_common nfit libnvdimm kvm_intel kvm virtio_net net_failover virtio_balloon failover psmouse i2c_piix4 irqbypass i2c_core pcspkr lp parport_pc parport video wmi crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 floppy sha1_ssse3 ata_generic virtio_blk serio_raw pata_acpi dm_mirror dm_region_hash dm_log dm_mod
Apr 18 00:56:26 ipfire kernel: CR2: 000000000000009a
Apr 18 00:56:26 ipfire kernel: ---[ end trace 0000000000000000 ]---
Apr 18 00:56:26 ipfire kernel: RIP: 0010:ip_mr_forward+0xde/0x3a0
Apr 18 00:56:26 ipfire kernel: Code: 48 8d 7e 01 48 8d 04 fd 00 00 00 00 48 29 f8 48 c1 e0 04 48 01 e8 48 8b 00 49 39 c5 0f 84 d9 00 00 00 49 8b 46 58 48 83 e0 fe <80> b8 9a 00 00 00 00 75 2c 45 85 ff 0f 84 a4 00 00 00 48 83 c4 18
Apr 18 00:56:26 ipfire kernel: RSP: 0018:ffffad418287fbe0 EFLAGS: 00010246
Apr 18 00:56:26 ipfire kernel: RAX: 0000000000000000 RBX: ffff99fc42a9bcc0 RCX: 0000000000000001
Apr 18 00:56:26 ipfire kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002
Apr 18 00:56:26 ipfire kernel: RBP: ffff99fc4156e000 R08: 0000000000000000 R09: 0000000000000001
Apr 18 00:56:26 ipfire kernel: R10: ffff99fc42a9bcc0 R11: 0000000000000000 R12: ffffffffaa5d1f80
Apr 18 00:56:26 ipfire kernel: R13: ffff99fc47412000 R14: ffff99fc42da8b00 R15: 0000000000000000
Apr 18 00:56:26 ipfire kernel: FS: 00007863f4dbc740(0000) GS:ffff99fd57c00000(0000) knlGS:0000000000000000
Apr 18 00:56:26 ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 18 00:56:26 ipfire kernel: CR2: 000000000000009a CR3: 000000010be38005 CR4: 0000000000770ef0
Apr 18 00:56:26 ipfire kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 18 00:56:26 ipfire kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 18 00:56:26 ipfire kernel: PKRU: 55555554
Apr 18 00:56:26 ipfire kernel: note: igmpproxy[12795] exited with irqs disabled
Apr 18 00:56:26 ipfire kernel: ------------[ cut here ]------------
Apr 18 00:56:26 ipfire kernel: Voluntary context switch within RCU read-side critical section!
Apr 18 00:56:26 ipfire kernel: WARNING: CPU: 0 PID: 12795 at kernel/rcu/tree_plugin.h:320 rcu_note_context_switch+0x5c0/0x620
Apr 18 00:56:26 ipfire kernel: Modules linked in: tun nfnetlink_queue xt_NFQUEUE xt_MASQUERADE cfg80211 rfkill 8021q garp xt_set ip_set_hash_net ip_set xt_hashlimit xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_mangle iptable_filter vfat fat sch_cake intel_rapl_common nfit libnvdimm kvm_intel kvm virtio_net net_failover virtio_balloon failover psmouse i2c_piix4 irqbypass i2c_core pcspkr lp parport_pc parport video wmi crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 floppy sha1_ssse3 ata_generic virtio_blk serio_raw pata_acpi dm_mirror dm_region_hash dm_log dm_mod
Apr 18 00:56:26 ipfire kernel: CPU: 0 PID: 12795 Comm: igmpproxy Tainted: G D 6.6.15-ipfire #1
Apr 18 00:56:26 ipfire kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 18 00:56:26 ipfire kernel: RIP: 0010:rcu_note_context_switch+0x5c0/0x620
Apr 18 00:56:26 ipfire kernel: Code: 00 00 00 00 0f 85 75 fd ff ff 49 89 8c 24 a0 00 00 00 e9 68 fd ff ff 48 c7 c7 f0 9a a3 a9 c6 05 ab 13 49 01 01 e8 90 a7 f6 ff <0f> 0b e9 a2 fa ff ff 49 83 bc 24 98 00 00 00 00 49 8b 84 24 a0 00
Apr 18 00:56:26 ipfire kernel: RSP: 0018:ffffad418287fc70 EFLAGS: 00010046
Apr 18 00:56:26 ipfire kernel: RAX: 0000000000000000 RBX: ffff99fd57c2db80 RCX: 0000000000000000
Apr 18 00:56:26 ipfire kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Apr 18 00:56:26 ipfire kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Apr 18 00:56:26 ipfire kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff99fd57c2ce00
Apr 18 00:56:26 ipfire kernel: R13: ffff99fc45871700 R14: ffffffffaa5d1f80 R15: ffff99fc45871f40
Apr 18 00:56:26 ipfire kernel: FS: 0000000000000000(0000) GS:ffff99fd57c00000(0000) knlGS:0000000000000000
Apr 18 00:56:26 ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 18 00:56:26 ipfire kernel: CR2: 000000000000009a CR3: 0000000004030002 CR4: 0000000000770ef0
Apr 18 00:56:26 ipfire kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 18 00:56:26 ipfire kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 18 00:56:26 ipfire kernel: PKRU: 55555554
Apr 18 00:56:26 ipfire kernel: Call Trace:
Apr 18 00:56:26 ipfire kernel:
Apr 18 00:56:26 ipfire kernel: ? rcu_note_context_switch+0x5c0/0x620
Apr 18 00:56:26 ipfire kernel: ? __warn+0x81/0x130
Apr 18 00:56:26 ipfire kernel: ? rcu_note_context_switch+0x5c0/0x620
Apr 18 00:56:26 ipfire kernel: ? report_bug+0x1a2/0x1d0
Apr 18 00:56:26 ipfire kernel: ? handle_bug+0x49/0xa0
Apr 18 00:56:26 ipfire kernel: ? exc_invalid_op+0x17/0x80
Apr 18 00:56:26 ipfire kernel: ? asm_exc_invalid_op+0x1a/0x20
Apr 18 00:56:26 ipfire kernel: ? rcu_note_context_switch+0x5c0/0x620
Apr 18 00:56:26 ipfire kernel: __schedule+0x82/0x12e0
Apr 18 00:56:26 ipfire kernel: ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
Apr 18 00:56:26 ipfire kernel: schedule+0x56/0xa0
Apr 18 00:56:26 ipfire kernel: schedule_preempt_disabled+0x18/0x30
Apr 18 00:56:26 ipfire kernel: __mutex_lock.constprop.0+0x680/0x6b0
Apr 18 00:56:26 ipfire kernel: ip_mc_drop_socket+0x2b/0xc0
Apr 18 00:56:26 ipfire kernel: inet_release+0x1f/0x70
Apr 18 00:56:26 ipfire kernel: __sock_release+0x3a/0xc0
Apr 18 00:56:26 ipfire kernel: sock_close+0x15/0x20
Apr 18 00:56:26 ipfire kernel: __fput+0xea/0x290
Apr 18 00:56:26 ipfire kernel: task_work_run+0x5a/0x90
Apr 18 00:56:26 ipfire kernel: do_exit+0x329/0xaa0
Apr 18 00:56:26 ipfire kernel: make_task_dead+0x81/0x170
Apr 18 00:56:26 ipfire kernel: rewind_stack_and_make_dead+0x17/0x20
Apr 18 00:56:26 ipfire kernel: RIP: 0033:0x7863f4ecd6de
Apr 18 00:56:26 ipfire kernel: Code: Unable to access opcode bytes at 0x7863f4ecd6b4.
Apr 18 00:56:26 ipfire kernel: RSP: 002b:00007fff0dddfa48 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
Apr 18 00:56:26 ipfire kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007863f4ecd6de
Apr 18 00:56:26 ipfire kernel: RDX: 00000000000000cc RSI: 0000000000000000 RDI: 0000000000000003
Apr 18 00:56:26 ipfire kernel: RBP: 0000000001f9f8f4 R08: 000000000000003c R09: 0000000000000000
Apr 18 00:56:26 ipfire kernel: R10: 00007fff0dddfa50 R11: 0000000000000246 R12: 00007fff0dddfab0
Apr 18 00:56:26 ipfire kernel: R13: 0000000000409ac7 R14: 0000000000000005 R15: 0000000000000000
Apr 18 00:56:26 ipfire kernel:
Apr 18 00:56:26 ipfire kernel: ---[ end trace 0000000000000000 ]---
Apr 18 00:57:26 ipfire kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Apr 18 00:57:26 ipfire kernel: rcu: ^ITasks blocked on level-0 rcu_node (CPUs 0-1): P12795/1:b..l
Apr 18 00:57:26 ipfire kernel: rcu: ^I(detected by 0, t=60002 jiffies, g=161269, q=13017 ncpus=2)
Apr 18 00:57:26 ipfire kernel: task:igmpproxy state:D stack:0 pid:12795 ppid:1 flags:0x00004002
Apr 18 00:57:26 ipfire kernel: Call Trace:
Apr 18 00:57:26 ipfire kernel:
Apr 18 00:57:26 ipfire kernel: __schedule+0x318/0x12e0
Apr 18 00:57:26 ipfire kernel: schedule+0x56/0xa0
Apr 18 00:57:26 ipfire kernel: schedule_preempt_disabled+0x18/0x30
Apr 18 00:57:26 ipfire kernel: __mutex_lock.constprop.0+0x392/0x6b0
Apr 18 00:57:26 ipfire kernel: ip_mc_drop_socket+0x2b/0xc0
Apr 18 00:57:26 ipfire kernel: inet_release+0x1f/0x70
Apr 18 00:57:26 ipfire kernel: __sock_release+0x3a/0xc0
Apr 18 00:57:26 ipfire kernel: sock_close+0x15/0x20
Apr 18 00:57:26 ipfire kernel: __fput+0xea/0x290
Apr 18 00:57:26 ipfire kernel: task_work_run+0x5a/0x90
Apr 18 00:57:26 ipfire kernel: do_exit+0x329/0xaa0
Apr 18 00:57:26 ipfire kernel: make_task_dead+0x81/0x170
Apr 18 00:57:26 ipfire kernel: rewind_stack_and_make_dead+0x17/0x20
Apr 18 00:57:26 ipfire kernel: RIP: 0033:0x7863f4ecd6de
Apr 18 00:57:26 ipfire kernel: RSP: 002b:00007fff0dddfa48 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
Apr 18 00:57:26 ipfire kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007863f4ecd6de
Apr 18 00:57:26 ipfire kernel: RDX: 00000000000000cc RSI: 0000000000000000 RDI: 0000000000000003
Apr 18 00:57:26 ipfire kernel: RBP: 0000000001f9f8f4 R08: 000000000000003c R09: 0000000000000000
Apr 18 00:57:26 ipfire kernel: R10: 00007fff0dddfa50 R11: 0000000000000246 R12: 00007fff0dddfab0
Apr 18 00:57:26 ipfire kernel: R13: 0000000000409ac7 R14: 0000000000000005 R15: 0000000000000000
Apr 18 00:57:26 ipfire kernel:

When this occurs, many applications get stuck in "D" (uninterruptible sleep), including pakfire, ping (when pinging hostnames in the internet, not ip addresses), whois, iptables -L, and so on.

Everything works fine when igmpproxy is not running. However, once it is started, the system misbehaves and needs to be rebooted.

Thank you for looking into this. :)

[mpaland]

I can confirm this. It's a bad problem and stops IPFire updates for IGMP users completely.

Additional info:
Before core 183, IPFire used longterm kernel 6.1.xx.
Core 183 and up use longterm kernel 6.6.xx, which causes the described problem above. Something in the kernel changed and is incompatible now.

Unfortunately I'm not very familiar with the code of the Linux kernel, so I can't investigate the problem myself.

[tscholz]

Just for information: I've switched to the IPFire development build 186. IGMPProxy is working again. I didn't see any crashes or other strange behaviour of the kernel so far.