博客

watchdog一般是一个硬件模块（其实可以当做是一个定时器），其作用是，在嵌入式操作系统中，很多应用情况是系统长期运行且无人看守，导致程序跑飞，所以难免怕万一出现系统死机，那就悲剧了，这时，watchdog就会自动帮你重启系统。

那么其是如何实现此功能的呢？简单解释一下其实现原理：

 watchdog硬件的逻辑就是，其硬件上有个记录超时功能，然后要求用户需要每隔一段时间（此时间可以根据自己需求而配置）去对其进行一定操作，比如往里面写一些固定的值，俗称“喂狗”，那么看门狗发现超时了，即过了返么长时间你还不给看门狗喂食，那么看门狗就认为你系统是死机了，出问题了，看门狗就帮你reset重启系统。

  为何在要系统初始化的时候关闭watchdog

了解了watchdog的原理后，此问题就很容易理解了。如果不禁用watchdog，那么就要单独写程序去定期“喂狗”，那多麻烦，多无聊啊。毕竟咱此处叧是去用uboot初始化必要的硬件资源和系统资源而已，完全用丌到返个watchdog的机制。需要用到，那也是你linux 内核跑起来了，是你系统关心的事情，和我uboot没啥关系的，所以肯定此处要去关闭watchdog（的reset功能）了。

看门狗（watchdog）包括一个4分频的预分频器和一个32位的计数器，时钟通过预分频器输入定时器。定时器递减计数，递减的最小值为0XFF。如果设置一个小于0XFF的值，系统会将0XFF装入计数器，因此最小看门狗间隔为t(pclk)X256X4。

门狗的用途是使微控制器在进入错误状态后的一定时间内复位。当看门狗使能时，如果用户程序没有在周期时间内喂狗（重装），看门狗会产生一个系统复位。

看门狗的特性如下：

1.如果没有周期性重装，则产生片内复位。

2.调试模式。

3.由软件使能，但要求禁止硬件复位或看门狗复位/中断~。

4.错误/不完整的喂狗时序会导致复位/中断（如果使能）。

5.指示看门狗复位的标志。

6.带内部预分频器的可编程32位定时器。

7.可选择t(pclk)X4的倍数的时间周期。

基本操作：看门狗应当根据下面的方法来使用

1.在WDTC寄存器中设置看门狗定时器的固定装载值；

2.在WDMOD寄存器中设置模式；

3.通过向WDFEED寄存器顺序写入0XAA和0X55启动看门狗；

4.在看门狗向下溢出之前应当再次喂狗，以防止复位/中断~！

当看门狗计数器想下溢出时，程序计数器将从0x00000000开始，和外部复位一样。可以检查看门狗超时标志WDTOF来确定看门狗是否产生复位条件，WDTOF标志必须由软件清零。

MTK6737平台看门狗相关


文档：

MT6737_LTE_Smartphone_Application_Processor_Functional_Specification_V1.1应用处理器功能规格书 重要.pdf

如下内容：

top reset generate unit顶级重启发生单元

a watchdog timer is also included in this module看门狗定时器也包含在这个模块

类似的看门狗相关的数据手册 电路图

MTK 看门狗详细解析

目录

WDT硬件设计 2

0 WDT原理 2

1 MediatekWDT硬件设计 2

WDT软件设计 2

2设计原理 2

3 WDT驱动 2

4 WDK驱动 2

5 超时触发流程 2

分析方法 2

6 HWT 2

7 HW REBOOT 2

案例分析 2

8 HWT-kernel初始化时间长（1） 2

9 HWT-kernel初始化时间长（2） 2

10 HWT-lk执行时间长 2

11 HWT中断频繁触发 2

12 HWT-大量DEVAPC violation 2

13 HWT-SWT卡死 2

14 HWT-死锁卡死 2

15 HWT- 关机死锁卡死 2

16 HWT-死循环卡死 3

17 HWT-CPU卡死 3

18 HWT-rt throttle引起卡死 3

19 HWT-hotplug失败 3

20 HWT-在TEE OS里卡死 3

21 HWT-spinlock嵌套死锁 3

22 HWT-两把不同的spinlock造成死锁 3

23 HW REBOOT-HW故障 3

附录 3

24 相关FAQ 3

0 WDT原理

1. 原理

在手机Soc Chip中，里面的AP跑着linux操作系统软件，而任何软件都可能存在各种问题，如果遇到了这些异常，软件可能陷入死循环，导致手机变成“砖头”，如果没有其他硬件辅助，那么只能断电（拔电池）然后重新开机才行。为了避免出现这种情况，芯片内部增加了一个看门狗模块，这个模块专门检测CPU运行状态，只要出现卡死就复位系统。

WDT全称是watchdog timer，就是看门狗模块，看门狗其实就是一个可以在一定时间内被复位的计数器。当看门狗启动后，计数器开始自动计数，经过一定时间，如果计数没有被复位，计数器达到指定数值就会发出复位信号，很多设备包括CPU接到这个信号而复位重启（俗称“被狗咬”）。为了保证看门狗不发出复位信号，就需要在看门狗允许的时间间隔内对看门狗计数器清零（俗称“喂狗”），计数器重新计数。如果系统正常并保证按时“喂狗”，那么就相安无事。一旦程序故障卡死，没有“喂狗”，系统“被咬”复位。

2. 种类

嵌入式系统中主要可以分为两种类型的看门狗：

• 
  
  Soc芯片内部集成WDT，这是Soc常用的设计。当然PC上可能用独立的看门狗芯片。
  
  
• 
  
  软件模拟看门狗，只要有个timer就可以模拟。
  
  

1 MediatekWDT硬件设计

在现有Soc芯片中，只有1个看门狗硬件模块，这个模块被包含在一个叫RGU的模块里。在早期平台中，比如MT6577，MT6575除了RGU里的WDT外，每个CPU都有一个local WDT，可以保证每个CPU不卡死。而MT6577之后就只有RGU里的WDT了，如图：

可以看到WDT在RGU里只是其中一个模块，还有其他模块可以产生复位信号，比如thermal，当温度过高会触发IRQ或直接复位，这也是一种硬件保护措施。复位信号经过Mixer后会分出2个信号，grst_b会复位芯片内部模块，还有1个信号通过芯片管脚WATCHDOG复位外部芯片。有时为了debug，也可以测量WATCHDOG pin脚（正常是高电平，有复位信号时低电平），看这次重启是否是WDT等触发的。

2. dual mode

WDT功能，大家可以看datasheet里的寄存器介绍，比如包含了

• 
  
  WDT_MODE：功能开关
  
  
• 
  
  WDT_LENGTH：超时计数
  
  
• 
  
  WDT_RESTART：写该寄存器触发计数清0
  
  
• 
  
  WDT_STA：状态
  
  
• 
  
  WDT_INTAVEL：reset信号长度
  
  
• 
  
  WDT_SWRST：写该寄存器直接触发复位
  
  

这里介绍一个dual mode功能，通常CPU发生卡死，我们需要知道卡死位置，分析原因，然后改进，而不希望直接复位。设计的方法是：WDT超时后先不发复位信号，而是先送出IRQ（如上面的框图），WDT IRQ通常配置成CPU的FIQ，如果CPU只是软件卡死（内核死锁，中断过多等），会响应该FIQ，然后我们在FIQ里收集异常信息和CPU寄存器信息，然后再走正常的panic流程。重启后我们就有信息分析此次WDT timeout的原因了。为了保证IRQ发出后，CPU不卡死，WDT再次计时，如果在panic流程又卡死了，就会由WDT发出复位信号复位。

简单讲：WDT超时先发出IRQ，WDT重新计时，如果再次超时WDT直接发出复位信号。可以看到WDT超时后分2个阶段，第1阶段发出IRQ用于收集异常信息，第2阶段直接复位。通常我们在第1阶段就完成异常信息收集并通过WDT_SWRST寄存器完成复位，而不用等到第2阶段的。这种情况我们称为


HWT


（Hardware Watchdog Timeout）

事实上任何事情总有意外，比如CPU因为PMIC提供的电压低，或硬件故障等原因导致WDT超时发出的IRQ在CPU端得不到响应，WDT只能通过第2阶段复位芯片。这种情况我们称为


HW reboot


。HW reboot通常和硬件有较大关系。比如：

• 
  
  硬件故障，电压或频率异常
  
  
• 
  
  总线卡死
  
  

2设计原理

• 
  
  
  bit位代表CPU的喂狗状态：CPU核数每个平台都不太一样，我们用bit位代表1个CPU，比如4个CPU就用4个bit，我们用kick_bit变量表示。
  
  
  
• 
  
  
  喂狗进程负责喂狗：在喂狗模块初始化时针对每个CPU创建的对应的内核实时进程（优先级最高）并绑定到对应的CPU上（也就是说这个进程只运行在指定CPU上，不会发生迁移），这些进程喂狗（其实就是将kick_bit对应的bit位置1而已，并没有真正喂狗）后进入睡眠，喂狗间隔是
  
  20s
  
  ，如此循环反复。
  
  
  
• 
  
  
  真正喂外部WDT狗：任何一个喂狗进程在喂狗之后，会检查kick_bit变量，如果该值等于cpus_kick_bit变量（这个变量的每个bit代表对应CPU online情况，如果CPU online，对应bit置1），表示所有CPU都已喂狗，这时才真正喂外部WDT狗，同时将kick_bit清0，进入下一轮喂狗流程，如此循环反复。
  
  
  

用一张框图总结上面的流程（以4核CPU为例）：

这里的wdtk-x就是对应CPU的喂狗进程。以上的一套设计可以保证各个CPU卡死都可以通过看门狗复位。

喂狗间隔是20s，而超时时间是30s，也就是说最长能容忍卡住的时间是10s（卡一小会还是可以的），超过这个时间，系统就会复位了。这里还有问题，由于喂狗进程之间没有同步，是否有可能存在刚开始一起喂狗，之后渐渐出现先后呢？误差肯定有的，但在任何30s时间里，喂狗进程都会喂1次狗的（因为喂狗间隔20s，每个CPU肯至少会喂1次狗）。而只要CPU卡死超过10s就复位了。

多核CPU不可能一直都是online，系统会根据负载做hotplug，在某颗CPU power down/power up时，会更新cpus_kick_bit变量并将kick_bit变量清0，同时喂外部WDT狗。

当只剩1个CPU时，并且该CPU要进入睡眠，此时kernel进入suspend状态，WDT模块当然也要关闭的，唤醒时再重启开启。那睡眠后的流程卡死怎么办？那已超出WDT管辖的范围了，需要用其他硬件保证，不在该文章讨论范围。

3 WDT驱动

流程是：




module_init



(





mtk_wdt_init



) =>



mtk_wdt_init



() =>



mtk_wdt_probe



()。

在




mtk_wdt_probe



()函数里，做了如下几件事：

这样就完成WDT的初始化了，WDT初始化是比较早的，基本上3s内就完成了（kernel时间）。以下是72KK版本的uart log中搜索wdt结果：

睡眠/唤醒

在系统进入睡眠时，WDT要被关闭，不过不是在WDT suspend里关闭WDT，因为suspend不是睡眠最后的步骤。这里会在sleep模块通过调用WDK的wd_suspend_notify()来间接调用mtk_wd_suspend()来关闭WDT的。同理唤醒时也是早于resume流程就被sleep模块通过WDK的resume_notify()调用mtk_wd_resume()重新启动WDT。

重启

系统重启（比如adb reboot）是通过WDT来完成复位的，在驱动里提供了




wdt_arch_reset



()函数，这个函数通过写WDT_SWRST寄存器完成软件复位。基本上重启的log都长这样：

红线以上是正常的log，写完WDT_SWRST寄存器基本就复位了，但有些平台并不是立即生效，需要过几百毫秒才完成复位，因此有可能看到红线以下的log，这是正常的。

喂狗函数

WDT启动后就开始计时了，如果没人喂狗就会触发FIQ，然后在硬复位。因此需要提供了




mtk_wdt_restart



()函数，调用一次就将计数清0。

如果有驱动需要长时间关中断运行，比如开机时的TP固件升级，就需要在里面添加喂狗动作，防止HWT。

开关WDT

有时为了调试，我们需要关闭看门狗，驱动里也提供了开启/关闭函数：




mtk_wdt_enable



()。如果你要关闭WDT，可以直接在代码里调用这个函数，那么WDT就永久关闭了。

在WDK驱动里还提供了用户接口，可以很方便在adb shell里关闭/打开WDT。

结语

WDT驱动比较简单，基本都是控制WDT硬件寄存器达到所需效果。重点还在于WDK驱动(喂狗模块)。

4 WDK驱动

• 
  通过
  
  
  
  
  mtk_rgu_sysfs
  
  
  
  ()创建了sysfs文件系统节点。
  
  
• 
  通过
  
  
  
  
  start_kicker
  
  
  
  ()创建per CPU的喂狗进程wdtk-x，这些进程都是
  
  系统中优先级最高的进程
  
  ，执行的核心函数是
  
  
  
  kwdt_thread
  
  
  
  ()，里面实现了前面提到的喂狗框架。
  
  
• 
  通过
  
  
  
  
  wk_proc_init
  
  
  
  ()创建了proc文件系统节点/proc/wdk。我们可以通过它来设定超时时间、喂狗时间、打开/关闭WDT。格式如下（如关闭WDT）：
  
  echo 0 20 30 0 0 > /proc/wdk
  
  
  
  • 
    参数1：目前没用到，可以忽略，直接写0。
    
  • 
    参数2：喂狗间隔，默认20，单位秒。
    
  • 
    参数3：超时时间，默认30，单位秒。
    
  • 
    参数4：目前没用到，可以忽略。
    
  • 
    参数5：打开/关闭WDT。
    
• 
  根据当前online CPU状况更新cpus_kick_bit并注册CPU hotplug回调函数，因为CPU会根据负载上电/下电，因此需要及时更新cpus_kick_bit、kick_bit。
  

需要注意的是：

• 
  WDT初始化和WDK初始化是处于kernel初始化的不同阶段，WDT通过module_init()完成，而WDK通过late_initcall()，中间隔着各种驱动，因此如果有些驱动执行时间太长，会导致开机30s HWT的问题，这个在后面章节提到。
  
• 
  在
  
  KK1.MP11以前的版本
  
  启动喂狗进程时，会直接调用到kwdt_thread()，里面会设置一遍kick_bit，最后等于cpus_kick_bit然后喂WDT，但实际上第1次kick并没有累加起来，查看log：
  

• 
  看到所有wdk进程起来后，都会设置kick_bit(就是local_bit)，但是没有累加起来，无法等于0xf，也就无法喂WDT，真正喂狗是在20s之后，从时间轴来看：
  

• 
  这是为什么呢？原因是
  
  
  
  
  start_kicker
  
  
  
  ()里没创建完进程，就会调用
  
  
  
  wk_cpu_update_bit_flag
  
  
  
  ()将kick_bit清0，因此无法累加。
  
  KK1.MP11及之后的版本就没有再调用
  
  
  
  
  wk_cpu_update_bit_flag
  
  
  
  ()，因此wdk初始化完就会完成第1次喂狗。
  
  

喂狗

kwdt_thread



()完成喂狗动作，里面就是一个循环：

• 
  如果需要更新超时时间，则在这里调用wdt函数更新。
  
• 
  更新对应CPU的kick_bit，如果等于cpus_kick_bit则清除kick_bit并喂WDT。
  
• 
  显示UTC时间，主要用于kernel和android时间同步，这条log非常重要，如果要看android log当时对应的kernel log，就可以通过这条log找到大致的位置。由于WDK是每20s跑一次，因此这条log也是每20s输出一次，如下：
  
  [   25.876493]<1>[thread:187][RT:25867046616] 2015-01-01 00:10:38.35106 UTC; android time 2015-01-01 00:10:38.35106
  
  
• 
  睡眠20s
  

各个CPU的




kwdt_thread



()没有同步，因此各自喂狗的时间不定，但只要喂狗间隔不超过30s就没有事。

CPU hotplug

由于有注册CPU hotplug回调函数，因此在某颗CPU上电时，会将cpus_kick_bit对应的bit置位并同时喂狗。下电时，会清除cpus_kick_bit对应bit位并同时喂狗。

5 超时触发流程

• 
  
  
  __vectors_start + 0x1C：这是异常向量表FIQ入口，所有FIQ中断都会导向这里。
  
  
  
• 
  
  
  
  
  fiq_glue
  
  
  
  ()：FIQ入口直接调用了这个函数，此时处于FIQ mode，用独立的栈，只要还在这个独立的栈里就无法调用printk输出log，原因是printk使用了current()，会从栈底取出thread_info结构体，而该独立的栈咩有thread_info结构体。
  
  
  
  
• 
  
  
  
  
  fiq_isr
  
  
  
  ()：这个函数更新了fiq_step。后面会继续讲fiq_step内容。
  
  
  
  
• 
  
  
  
  
  wdt_fiq
  
  
  
  ()：到了WDT注册的中断函数了，这里还是无法使用printk，因此只能调用
  
  
  
  aee_wdt_printf
  
  
  
  ()打印到buffer里。这里印出了最重要的信息kick和check(对应的kick_bit和cpus_kick_bit)，比如：
  
  kick=0x00000007,check=0x0000000f
  
  。得到kick和check就知道是哪些CPU没有喂狗了，接下来重点关注没喂狗的CPU调用栈等信息。
  
  
  
  
• 
  
  
  
  
  aee_wdt_fiq_info
  
  
  
  ()：这个函数首先将栈切换为当前进程栈，就可以用printk了。另外当前CPU的寄存器打印到per CPU buffer里。最后只允许一个CPU继续往下走，其他CPU直接死循环等待重启了。
  
  
  
  
• 
  
  
  
  
  aee_wdt_irq_info
  
  
  
  ()：这个函数将输出重要log
  
  
  
  
  • 
    
    
    喂狗，防止超时。
    
    
    
  • 
    
    
    停止其他CPU。
    
    
    
  • 
    
    
    将per CPU buffer输出到last_kmsg。注意：只输出到last_kmsg，kernel log是看不到的。last_kmsg里的log如下：
    
    
    

• • • 
      
      
      如果后面流程卡住了，那么能参考的信息只有last_kmsg了，因此这个信息尤为重要，里面包含CPU寄存器和调用栈。
      
      
      
    • 
      
      
      如果没有卡住，那么在kernel log也可以看到这些信息，类似如下：
      
      
      

• • • 
      
      
      我们重点看CPU Backtrace部分，这些都是函数地址，需要将地址转换为函数名和所在文件、行号。则需要用到arm-linux-android-addr2line（用于32位）/aarch64-linux-android-addr2line（用于64位，详情请查看quick start => Native Exception问题深入解析 => 进阶篇: coredump分析 => GNU tools）和vmlinux（必须是当时一起编译出来的，如果后面有更新过，addr2line得出的结果可能错误）。比如：
      
      
      

      
      
      
      
      ./prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-addr2line -Cfe vmlinux 0xffffffc00014e68c => ring_buffer_unlock_commit()
      
      
      
      

      
      
      
      全部转换后就可以看到完整的调用栈了，有助于我们分析哪里卡住。
      
      
      

  • 
    
    
    调用mt_aee_dump_sched_traces()输出中断信息。这个信息只有eng版本才打开(user版本打开方法请参考[FAQ15102]如何调试IRQ引起的HWT？)。这个有什么用呢？HWT有一个原因是IRQ太过频繁导致，因此IRQ信息能直接看出原因
    
    
    
    • 
      
      
      当前CPU中断信息，可以看到上次IRQ/SoftIRQ/tasklet/hrtimer/SoftTimer信息，如果HWT时调用栈在IRQ里，还可以看到该IRQ的信息，可以
      
      判断执行是否太久
      
      。
      
      
      
    • 
      
      
      IRQ Status会印出所有中断在Dur时间段里触发次数，
      
      如果太多就有问题了
      
      （比如>1000次每秒）。
      
      
      

• • 
    
    
    将aee wdt buffer打印到last_kmsg，之前是per CPU buffer，这次是aee wdt buffer：
    
    
    

，这里才能从last_kmsg看到kick/check信息。

• 
  
  
  
  
  BUG
  
  
  
  ()，之后就是正常的die/panic流程了，所以kernel log总能看到PC is at aee_wdt_irq_info。看到这个信息就可以判断是HWT了。
  
  
  
  

6 HWT

开机30s超时

kernel初始化流程：




start_kernel



() =>



rest_init



()，这个过程都是0号进程完成，在rest_init()就创建了1号进程执行



kernel_init



()，接着=>



kernel_init_freeable



() =>



do_basic_setup



() =>



do_initcalls



()。这个过程是单线程的，都是1号线程完成。

初始化时间长的问题，可以通过uart log检索

[1:swapper/0]

关键字分析就行了。后面有详细例子讲解。

分析所需材料：

• 
  db、vmlinux和uart log
  
• 
  db在sd卡下的mtklog/aee_exp，或/data/aee_exp目录下
  
• 
  vmlinux在codebase里的out/target/product/$proj/obj/kernel_obj/vmlinux
  

kernel运行过程中30s超时

此时WDK已初始化好，而喂狗进程都是优先级最高的实时进程，其他任何进程都无法卡到喂狗进程。那谁可以卡到喂狗进程呢？换句话说，谁可以影响到系统调度？答案有：

• 
  长时间关闭中断。这个直接无法调度了，还可以细分为：
  
  • 
    关中断的CPU死锁
    
  • 
    关中断后陷入循环
    
• 
  中断频繁触发。则导致CPU超载，进程基本没有机会被执行到。
  
• 
  长时间关抢断。
  
• 
  其他HW故障，如总线卡死或HW不稳定。
  

分析方法

：

• 
  获取kick/check bit得知哪些CPU没喂狗，查看last_kmsg/kernel log看这些CPU的调用栈（《超时触发流程》有讲如何转换CPU Backtrace），看是否卡在锁里。
  
• 
  如果怀疑中断过多，则需要eng版本复现抓取db，看isr monitor显示的各个中断触发的频率，判断是哪个中断引起，再找对应的驱动解决。
  
• 
  如果发现不符合逻辑引起的卡死，可能是HW不稳定，需要按其他方式处理。
  

分析所需材料

• • 
    db、vmlinux，必要时需要uart log或mobile log。
    

• 
  
  db在sd卡下的mtklog/aee_exp，或/data/aee_exp目录下
  
  
• 
  
  vmlinux在codebase里的out/target/product/$proj/obj/kernel_obj/vmlinux
  
  

7 HW REBOOT

#define CPU_ONLINE 0x0002 /* CPU (unsigned)v is up */
#define CPU_UP_PREPARE 0x0003 /* CPU (unsigned)v coming up */
#define CPU_UP_CANCELED 0x0004 /* CPU (unsigned)v NOT coming up */
#define CPU_DOWN_PREPARE 0x0005 /* CPU (unsigned)v going down */
#define CPU_DOWN_FAILED 0x0006 /* CPU (unsigned)v NOT going down */
#define CPU_DEAD 0x0007 /* CPU (unsigned)v dead */
#define CPU_DYING 0x0008 /* CPU (unsigned)v not running any task, not handling interrupts, soon dead. Called on the dying cpu, interrupts are already disabled. Must not sleep, must not fail */
#define CPU_POST_DEAD 0x0009 /* CPU (unsigned)v dead, cpu_hotplug lock is dropped */
#define CPU_STARTING 0x000A /* CPU (unsigned)v soon running. Called on the new cpu, just before enabling interrupts. Must not sleep, must not fail */

8 HWT-kernel初始化时间长（1）

基本信息

分析过程

抓取uart log，如下：

• 
  
  
  
  
  内核中很多驱动通过module_init()，late_init()等方法初始化，在内核初始化中在do_one_initcal()调用这些注册的函数的。
  
  
  
  
  
• 
  
  
  
  kernel_init() -> do_basic_setup() -> do_initcalls() -> do_initcall_level() -> do_initcall_level() -> do_one_initcal() -> xxx()
  
  
  
  

这些函数都是在1号进程里执行（单进程），因此我们只要查看[1:swapper/0]的log(为何是1:swapper/0呢？因为init还没被加载，所以fork出来的名字和父进程0:swapper/0一样)，就可以知道卡在哪里了：

解决方法



结语






解决这类问题，首先要对kernel初始化流程有所了解。否则即使知道HWT，也无法找到问题点。




9 HWT-kernel初始化时间长（2）

基本信息

问题：在recovery mode中，会出异常并自动进入normal mode

版本：ALPS.JB2.MP.V1.2

平台：MT6589

分析过程

抓取uart log，如下：

这是典型的开机30s WDT timeout的case，查看CPU0的backtrace，发现已经进入idle了。按照解题思路，需要查看WDK初始化情况，在log中搜索WDK关键字：

可以看到kicker thread在20s左右才创建的，有问题，因此搜索[1:swapper/0]的log：

看到TSP初始化时间间隔是很长的(16s)，检查TSP驱动，发现在kernel启动之后都会对TP做更新rom的动作，导致花费很长时间。

至于为什么normal mode下无问题呢，我们再抓取normal log来看，kiker thread也是20s左右创建的，但是之后CPU有plug off的动作，这个动作伴随着kick external WDT，所以才不会再30s时timeout。

解决方法

修正TP的驱动，仅在必要时才更新rom，并且更新时主动调用喂狗函数喂狗，避免超时。

10 HWT-lk执行时间长

基本信息

问题：项目刚开始，遇到板子下载软件后重启，几片板子都是这样，没有装外设（屏、TP都不装）

版本：ALPS.L0.MP8.V2

平台：MT6735

分析过程

抓取uart log，如下：

这是典型的刚进kenrel就发生WDT timeout的case，为什么会这样？

我们看到log：wdt_kernel_cb_addr=0, error before WDT successfully initialized

这个log从哪里来的？代码如下：

vendor/arm/atf-1.0/plat/common/aarch64/plat_common.c

void aee_wdt_dump()
{
    ......
    printf("=> aee_wdt_dump, cpu %d \n", (int)linear_id);

    ns_cpu_context = cm_get_context_by_mpidr(mpidr, NON_SECURE);
    atf_arg_t_ptr teearg = (atf_arg_t_ptr)(uintptr_t)TEE_BOOT_INFO_ADDR;
     regs = (void *)(teearg->atf_aee_debug_buf_start + (linear_id * sizeof(struct atf_aee_regs)));

    regs->pstate = SMC_GET_EL3(ns_cpu_context, CTX_SPSR_EL3)
    ......
    if(0 == wdt_kernel_cb_addr) {
        set_uart_flag();
        printf("=> wdt_kernel_cb_addr=0, error before WDT successfully initialized. cpu %d\n", (int)linear_id);
        printf("=> regs->pc : 0x%lx\n", regs->pc);
        printf("=> regs->lr : 0x%lx\n", regs->regs[30]);
        printf("=> regs->sp : 0x%lx\n", regs->sp);
        ......
    }
    ......
}

ATF是跑在kernel下面的固件，WDT timeout FIQ直接有ATF管理，所以ATF需要将其路由到kernel。通常情况下kernel初始化wdk驱动时会告诉ATF wdt timeout的handler，具体代码在：

kernel-3.10/drivers/misc/mediatek/aee/common/wdt-atf.c

static int __init aee_wdt_init(void)
{
    ......

    /* send SMC to ATF to register call back function */
#ifdef CONFIG_ARM64
    atf_aee_debug_phy_addr = (phys_addr_t)(0x00000000FFFFFFFF&mt_secure_call(MTK_SIP_KERNEL_WDT, (u64)&aee_wdt_atf_entry, 0, 0));
#else
    atf_aee_debug_phy_addr = (phys_addr_t)
    mt_secure_call(MTK_SIP_KERNEL_WDT, (u32)&aee_wdt_atf_entry, 0, 0);
#endif
    LOGD("\n MTK_SIP_KERNEL_WDT - 0x%p \n", &aee_wdt_atf_entry);

    if ((atf_aee_debug_phy_addr == 0) || (atf_aee_debug_phy_addr == 0xFFFFFFFF)) {
        LOGE("\n invalid atf_aee_debug_phy_addr \n");
    } else {
        /* use the last 16KB in ATF log buffer */
        atf_aee_debug_virt_addr = ioremap(atf_aee_debug_phy_addr, ATF_AEE_DEBUG_BUF_LENGTH);
        LOGD("\n atf_aee_debug_virt_addr = 0x%p \n", atf_aee_debug_virt_addr);
        if (atf_aee_debug_virt_addr)
            memset(atf_aee_debug_virt_addr, 0, ATF_AEE_DEBUG_BUF_LENGTH);
    }
    return 0;
}

通过SMC call将aee_wdt_atf_entry地址送给ATF，如果发生HWT，ATF就会回调aee_wdt_atf_entry()，不过看到wdt_kernel_cb_addr=0，表明kernel还没跑到这里。也表明kernel连wdt都还没初始化。那为什么还会timeout？

首先要了解：preloader、lk、kernel都会设定WDT timeout时间，preloader是16s，lk是10s，kernel是30s，都是在初始化时就设定好的。

我们观察到kernel才跑到0.360194s，那时还没跑到kernel初始化wdt的代码，也就是说，这时WDT的timeout时间还是用的是lk的设定时间10s，如果lk设定wdt时到kernel设定wdt前超过10s，就会发生WDT timeout，这个timeout可能在lk阶段，也可能在kernel阶段（kernel 设定wdt前，也表明lk执行时间过长，接近超时），我们看下是否是lk执行时间接近超时呢？

eng版本有lk profiling，查看log：

可以看到lk花的时间接近10s了，给kernel到wdt设定的时间不到0.5s，很容易超时的。

这题的原因是：lk执行时间过长，导致刚到kernel时（还没开始初始化wdt）就发生wdt timeout。需要看lk为何执行时间过久（eng有开lk profiling，搜索PROFILE关键字）：

就可以看到哪个阶段花的时间长了，如果不够细致，还可以自己添加log分析。

解决方法

通过profile解决异常花时间的模块问题。

这个case主要知识点是ATF接管WDT FIQ的流程，以及如何识别和分析timeout后印uart log。

11 HWT中断频繁触发

#ifdef __aarch64__
#define BUG() *((unsigned *)0xdead) = 0x0aee
#define HAVE_ARCH_BUG
#endif

12 HWT-大量DEVAPC violation

13 HWT-SWT卡死

404static int hang_detect_thread(void *arg)
405{
406
407	/* unsigned long flags; */
408	struct sched_param param = {.sched_priority = RTPM_PRIO_WDT };
409
410	LOGE("[Hang_Detect] hang_detect thread starts.\n");
411
412	sched_setscheduler(current, SCHED_FIFO, &param);
413
414	while (1) {
415		if ((1 == hd_detect_enabled) && (FindTaskByName("system_server") != -1)) {
416			LOGE("[Hang_Detect] hang_detect thread counts down %d:%d.\n",
417			     hang_detect_counter, hd_timeout);
418			if (hang_detect_counter <= 0) {
419				ShowStatus();
420			}
421
422
423			if (hang_detect_counter == 0) {
424				LOGE("[Hang_Detect] we should triger	HWT	...\n");
425				if (aee_mode != AEE_MODE_CUSTOMER_USER) {
426					aee_kernel_warning_api(__FILE__, __LINE__,
427								DB_OPT_NE_JBT_TRACES|DB_OPT_DISPLAY_HANG_DUMP,
428								"\nCRDISPATCH_KEY:SS Hang\n", "we triger HWT ");
429					msleep(30 * 1000);
430				} else {
431					/* aee_kernel_exception_api(__FILE__, __LINE__,
432							DB_OPT_NE_JBT_TRACES|DB_OPT_DISPLAY_HANG_DUMP,
433							"\nCRDISPATCH_KEY:SS Hang\n","we triger HWT "); */
434					msleep(30 * 1000);
435					local_irq_disable();
436					while (1);
437					BUG(); /* in order to get Hang info ASAP */
438				}
439			}
440
441	......
454}

14 HWT-死锁卡死

int mt_disable_clock(enum cg_clk_id id, char *name)
{
    int err;
    unsigned long flags;
    struct cg_clk *clk = id_to_clk(id);

    ......
    clkmgr_lock(flags);
    err = clk_disable_internal(clk, name);
    clkmgr_unlock(flags);

    return err;
}

1312int spm_mtcmos_ctrl_disp(int state)
1313{
1314    ......
1321    if (state == STA_POWER_DOWN) {
1322        spm_write(TOPAXI_PROT_EN, spm_read(TOPAXI_PROT_EN) | DISP_PROT_MASK);
1323        while ((spm_read(TOPAXI_PROT_STA1) & DISP_PROT_MASK) != DISP_PROT_MASK) { /* 卡在这里了！！！ */
1324        }

15 HWT- 关机死锁卡死

== 异常报告v1.5(仅供参考) ==
报告解读: MediaTek On-Line> Quick Start> E-Consulter之NE/KE分析报告解读> KE分析报告
详细描述: 断言失败{主动调用BUG()/BUG_ON()}，请结合崩溃进程调用栈检查相关代码
平台 : MT6757
版本 : alps-mp-m1.mp3/eng build
异常时间: 0.000000秒, Tue Sep 20 08:16:57 CST 2016

== CPU信息 ==
崩溃CPU信息:
 CPU0: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux __spin_lock_debug() + 496 <kernel/locking/spinlock_debug.c:218>
 vmlinux do_raw_spin_lock(lock=pmic_access_spin) + 736 <kernel/locking/spinlock_debug.c:254>
 vmlinux __raw_spin_lock() + 48 <include/linux/spinlock_api_smp.h:143>
 vmlinux _raw_spin_lock(lock=pmic_access_spin) + 64 <kernel/locking/spinlock.c:151>
 vmlinux spin_lock() + 44 <include/linux/spinlock.h:311>
 vmlinux pmic_config_interface_nolock(RegNum=516, val=1, SHIFT=4) + 68 <drivers/misc/mediatek/power/mt6757/pmic.c:839>
 vmlinux rtc_enable_k_eosc() + 60 <drivers/misc/mediatek/rtc/mt6351/mtk_rtc_hal.c:191>
 vmlinux hal_rtc_bbpu_pwdn(charger_status=false) + 32 <drivers/misc/mediatek/rtc/mt6351/mtk_rtc_hal.c:244>
 vmlinux rtc_bbpu_power_down() + 88 <drivers/misc/mediatek/rtc/mtk_rtc_common.c:400>
 vmlinux mt_power_off() + 76 <drivers/misc/mediatek/rtc/mtk_rtc_common.c:414>
 vmlinux machine_power_off() + 64 <arch/arm64/kernel/process.c:135>
 vmlinux kernel_power_off() + 96 <kernel/reboot.c:266>
 vmlinux SYSC_reboot() + 384 <kernel/reboot.c:334>
 vmlinux SyS_reboot(magic1=4276215469, magic2=672274793, cmd=1126301404, arg=0) + 392 <kernel/reboot.c:280>
 ...... ......
 vmlinux ____fput() + 8 <fs/file_table.c:271>
 vmlinux task_work_run() + 212 <kernel/task_work.c:125>
 vmlinux tracehook_notify_resume() + 24 <include/linux/tracehook.h:190>
 vmlinux do_notify_resume(thread_flags=4) + 84 <arch/arm64/kernel/signal.c:417>
 == 栈结束 ==
 对应汇编指令:
 行号 地址 指令 提示
 kernel/locking/spinlock_debug.c
 218 : FFFFFFC0001126B8: MOV X0, #-0x8000000001
 FFFFFFC0001126C0: MOVK X0, X0, #0xDEAD
 FFFFFFC0001126C4: STR W1, [X0] ; 进程停止在这里
 当时的寄存器值:
 X0: FFFFFF7FFFFFDEAD, X1: 0000000000000AEE, X2: CB88537FDC8CB01F, X3: 0000000000000003
 X4: 0000000000000000, X5: FFFFFFC0B49D8000, X6: FFFFFFC000106B3C, X7: 0000000000000001
 X8: FFFFFFC001CA7EA4, X9: 676E696B636F6C2F, X10: 636F6C6E6970732F, X11: 2E67756265645F6B
 X12: 5F5F2F3831323A63, X13: 636F6C5F6E697073, X14: 2867756265645F6B, X15: 0000000000000020
 X16: 0000000000000020, X17: 0000000000000000, X18: 0000000000000047, X19: FFFFFFC000F41448
 X20: 0000000000C65D40, X21: FFFFFFC001C6BB34, X22: 0000000000000005, X23: FFFFFFC000F41638
 X24: 00000021E56225BE, X25: 0000001EE37912F6, X26: FFFFFFC0012AD000, X27: FFFFFFC0AA8B2000
 X28: FFFFFFC001320BA8, X29: FFFFFFC0B49DFB40, X30: FFFFFFC0001126B8, SP: FFFFFFC0B49DFB40
 PC: FFFFFFC0001126C4

其他CPU信息:
 CPU4: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux ipi_cpu_stop() + 44 <arch/arm64/kernel/smp.c:677>
 vmlinux handle_IPI(ipinr=0) + 544 <arch/arm64/kernel/smp.c:727>
 ...... ......
 vmlinux mt_i2c_transfer(adap=0xFFFFFFC0AF819018, msgs=0xFFFFFFC0AA8B7818) + 68 <drivers/i2c/busses/i2c-mtk.c:970>
 vmlinux __i2c_transfer(adap=0xFFFFFFC0AF819018, msgs=0xFFFFFFC0AA8B7818) + 488 <drivers/i2c/i2c-core.c:2091>
 vmlinux i2c_transfer(adap=0xFFFFFFC0AF819018) + 96 <drivers/i2c/i2c-core.c:2162>
 ...... ......
 vmlinux i2c_smbus_xfer(read_write=1, protocol=2, data=0xFFFFFFC0AA8B7920) + 428 <drivers/i2c/i2c-core.c:2961>
 vmlinux i2c_smbus_read_byte_data(command=11) + 56 <drivers/i2c/i2c-core.c:2569>
 vmlinux bq2589x_read_byte(data=0xFFFFFFC0AA8B79C0, reg=11) + 64 <drivers/misc/mediatek/power/mt6757/bq2589x_charger.c:64>
 vmlinux bq2589x_get_vbus_status() + 36 <drivers/misc/mediatek/power/mt6757/bq2589x_charger.c:792>
 vmlinux upmu_is_chr_det() + 36 <drivers/power/mediatek/mtk_common.c:58>
 vmlinux bat_is_charger_exist() + 36 <drivers/power/mediatek/mtk_common.c:75>
 ...... ......
 vmlinux skb_recv_datagram() + 56 <net/core/datagram.c:272>
 ...... ......
 == 栈结束 ==

 CPU5: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux ipi_cpu_stop() + 44 <arch/arm64/kernel/smp.c:677>
 vmlinux handle_IPI(ipinr=0) + 544 <arch/arm64/kernel/smp.c:727>
 vmlinux __switch_to(prev=0xFFFFFFC0B4AE8000, next=0) + 96 <arch/arm64/kernel/process.c:395>
 vmlinux context_switch() + 788 <kernel/sched/core.c:2485>
 vmlinux __schedule() + 1164 <kernel/sched/core.c:3044>
 vmlinux schedule() + 36 <kernel/sched/core.c:3083>
 vmlinux schedule_preempt_disabled() + 16 <kernel/sched/core.c:3114>
 vmlinux cpu_idle_loop() + 264 <kernel/sched/idle.c:242>
 vmlinux cpu_startup_entry(state=CPUHP_ONLINE) + 296 <kernel/sched/idle.c:264>
 vmlinux secondary_start_kernel() + 540 <arch/arm64/kernel/smp.c:270>
 ...... 0x00000000400817E0()
 == 栈结束 ==

 CPU6: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux ipi_cpu_stop() + 44 <arch/arm64/kernel/smp.c:677>
 vmlinux handle_IPI(ipinr=0) + 544 <arch/arm64/kernel/smp.c:727>
 ...... ......
 vmlinux enqueue_task() + 68 <kernel/sched/core.c:919>
 ...... ......
 vmlinux __wake_up(mode=3, nr_exclusive=-1424458584, key=0) + 56 <kernel/sched/wait.c:94>
 vmlinux wake_up_bat() + 60 <drivers/power/mediatek/charger-manager.c:211>
 ...... ......
 vmlinux notifier_call_chain(nl=fb_notifier_list + 128, val=9, v=0xFFFFFFC0AB187CA8, nr_to_call=-1, nr_calls=0) + 132 <kernel/notifier.c:127>
 vmlinux __blocking_notifier_call_chain(nh=fb_notifier_list, nr_to_call=-1, nr_calls=0) + 64 <kernel/notifier.c:373>
 vmlinux blocking_notifier_call_chain() + 16 <kernel/notifier.c:384>
 vmlinux fb_notifier_call_chain(val=9, v=0xFFFFFFC0AB187CA8) + 24 <drivers/video/fbdev/core/fb_notify.c:45>
 vmlinux fb_blank(info=0xFFFFFFC0B1F71000, blank=4) + 176 <drivers/video/fbdev/core/fbmem.c:1073>
 vmlinux do_fb_ioctl(info=0xFFFFFFC0B1F71000, arg=4) + 748 <drivers/video/fbdev/core/fbmem.c:1209>
 vmlinux fb_ioctl(file=0xFFFFFFC02D05C000, arg=4) + 48 <drivers/video/fbdev/core/fbmem.c:1233>
 vmlinux vfs_ioctl() + 12 <fs/ioctl.c:43>
 vmlinux do_vfs_ioctl(filp=0xFFFFFFC02D05C000, fd=37, cmd=17937, arg=4) + 1060 <fs/ioctl.c:598>
 vmlinux SYSC_ioctl() + 96 <fs/ioctl.c:613>
 vmlinux SyS_ioctl(fd=37, cmd=17937, arg=4) + 128 <fs/ioctl.c:604>
 == 栈结束 ==

 CPU7: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux ipi_cpu_stop() + 44 <arch/arm64/kernel/smp.c:677>
 vmlinux handle_IPI(ipinr=0) + 544 <arch/arm64/kernel/smp.c:727>
 ...... ......
 vmlinux __local_bh_enable() + 68 <kernel/softirq.c:136>
 ...... ......
 vmlinux trace_hardirqs_off() + 12 <kernel/locking/lockdep.c:2713>
 vmlinux rcu_irq_exit() + 256 <kernel/rcu/tree.c:645>
 vmlinux irq_exit() + 88 <kernel/softirq.c:403>
 ...... ......
 vmlinux __this_cpu_preempt_check() + 20 <lib/smp_processor_id.c:68>
 vmlinux __trace_hardirqs_on_caller() + 68 <kernel/locking/lockdep.c:2626>
 vmlinux trace_hardirqs_on_caller() + 196 <kernel/locking/lockdep.c:2668>
 vmlinux trace_hardirqs_on() + 12 <kernel/locking/lockdep.c:2675>
 == 栈结束 ==

void machine_power_off(void)
{
    local_irq_disable();
    smp_send_stop();
    pr_emerg("machine_power_off, pm_power_off(%p)\n", pm_power_off);
    dump_stack();
    if (pm_power_off)
        pm_power_off();
}

static void ipi_cpu_stop(unsigned int cpu)
{
    if (system_state == SYSTEM_BOOTING || system_state == SYSTEM_RUNNING) {
        raw_spin_lock(&stop_lock);
        printk(KERN_CRIT "CPU%u: stopping\n", cpu);
        dump_stack();
        raw_spin_unlock(&stop_lock);
    }

    set_cpu_online(cpu, false);

    local_fiq_disable();
    local_irq_disable();

    while (1)
        cpu_relax();
}

16 HWT-死循环卡死

== 异常报告v1.4(仅供参考) ==
详细描述: 看门狗复位, 其中CPU0,1,2没有喂狗, 请检查对应调用栈是否存在死锁或长时间关中断
参考信息: MediaTek On-Line> Quick Start> 深入分析看门狗框架
异常时间: 3380.654356秒, Sun Mar 20 09:00:57 CST 2016


== CPU信息 ==
无喂狗CPU信息:
 CPU0: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux __iter_div_u64_rem() + 56 <include/linux/math64.h:122>
 vmlinux timespec_add_ns(ns=-6022794434679698269) + 56 <include/linux/time.h:224>
 vmlinux __getnstimeofday64(ts=0x000000003B9AC9FF) + 164 <kernel/time/timekeeping.c:508>
 vmlinux getnstimeofday64(ts=init_thread_union + 16000) + 8 <kernel/time/timekeeping.c:528>
 vmlinux do_gettimeofday(tv=init_thread_union + 16032) + 20 <kernel/time/timekeeping.c:699>
 vmlinux idle_get_current_time_ms() + 12 <drivers/misc/mediatek/base/power/spm_v2/mt_idle.c:474>
 vmlinux idle_ratio_calc_stop() <drivers/misc/mediatek/base/power/spm_v2/mt_idle.c:2014>
 vmlinux rgidle_enter(cpu=0) + 72 <drivers/misc/mediatek/base/power/spm_v2/mt_idle.c:2213>
 vmlinux mt_rgidle_enter(dev=0xFFFFFFC0FFF05BD8, drv=mt67xx_v2_cpuidle_driver, index=5) + 12 <drivers/cpuidle/cpuidle-mt67xx_v2.c:84>
 vmlinux cpuidle_enter_state(dev=0xFFFFFFC0FFF05BD8, index=5) + 128 <drivers/cpuidle/cpuidle.c:135>
 vmlinux cpuidle_enter(dev=0xFFFFFFC0FFF05BD8, index=5) + 20 <drivers/cpuidle/cpuidle.c:207>
 vmlinux cpuidle_idle_call() + 552 <kernel/sched/idle.c:145>
 vmlinux cpu_idle_loop() + 568 <kernel/sched/idle.c:215>
 vmlinux cpu_startup_entry(state=CPUHP_ONLINE) + 600 <kernel/sched/idle.c:263>
 vmlinux secondary_start_kernel() + 528 <arch/arm64/kernel/smp.c:250>
 == 栈结束 ==

 CPU1: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux multi_cpu_stop(data=0xFFFFFFC074467A08) + 88 <kernel/stop_machine.c:192>
 vmlinux cpu_stopper_thread(cpu=1) + 180 <kernel/stop_machine.c:458>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0FB91DA40) + 508 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0FB91DA80) + 212 <kernel/kthread.c:207>
 == 栈结束 ==

 CPU2: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux multi_cpu_stop(data=0xFFFFFFC074467A08) + 88 <kernel/stop_machine.c:192>
 vmlinux cpu_stopper_thread(cpu=2) + 180 <kernel/stop_machine.c:458>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0FB91DE40) + 508 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0FB91DE80) + 212 <kernel/kthread.c:207>
 == 栈结束 ==

[ 3380.668761] -(1)[11:migration/1][name:debug&]cpu#0: Offline
[ 3380.668770] -(1)[11:migration/1][name:debug&] .nr_running : 71

 CPU0: 进程名: (null), 进程标识符(pid): 0, 中断: 关

118__iter_div_u64_rem(u64 dividend, u32 divisor, u64 *remainder)
119{
120         u32 ret = 0;
121
122         while (dividend >= divisor) {
123                         /* The following asm() prevents the compiler from
124                            optimising this loop into a modulo operation.  */
125                         asm("" : "+rm"(dividend));
126
127                         dividend -= divisor;
128                         ret++;
129         }
130
131         *remainder = dividend;
132
133         return ret;
134}

493int __getnstimeofday64(struct timespec64 *ts)
494{
495         struct timekeeper *tk = &tk_core.timekeeper;
496         unsigned long seq;
497         s64 nsecs = 0;
498
499         do {
500                         seq = read_seqcount_begin(&tk_core.seq);
501
502                         ts->tv_sec = tk->xtime_sec;
503                         nsecs = timekeeping_get_ns(&tk->tkr);
505         } while (read_seqcount_retry(&tk_core.seq, seq));
506
507         ts->tv_nsec = 0;
508         timespec64_add_ns(ts, nsecs);

194static inline s64 timekeeping_get_ns(struct tk_read_base *tkr)
195{
196         cycle_t cycle_now, delta;
197         s64 nsec;
198
199         /* read clocksource: */
200         cycle_now = tkr->read(tkr->clock);
201
202         /* calculate the delta since the last update_wall_time: */
203         delta = clocksource_delta(cycle_now, tkr->cycle_last, tkr->mask);
204
205         nsec = delta * tkr->mult + tkr->xtime_nsec;
206         nsec >>= tkr->shift;
207
208         /* If arch requires, add in get_arch_timeoffset() */
209         return nsec + arch_gettimeoffset();
210}

tk_core = (
 seq = (sequence = 0x0009A6D0),
 timekeeper = (
 tkr = (
 clock = 0xFFFFFFC000FFE080,
 read = 0xFFFFFFC0007FA810,
 mask = 0x00FFFFFFFFFFFFFF,
 cycle_last = 0x0000000A286F2CF1,
 mult = 0x4CEC4EC5,
 shift = 0x18,
 xtime_nsec = 0x002D1EE8033F8540,
 base_mono = (tv64 = 0x0000030BC3A56B03)),
 xtime_sec = 0x56EDF615,
 wall_to_monotonic = (tv_sec = 0xFFFFFFFFA9121700, tv_nsec = 0x03B0D903),
 offs_real = (tv64 = 0x143D6468E02926FD),
 offs_boot = (tv64 = 0x0),
 offs_tai = (tv64 = 0x143D6468E02926FD),
 tai_offset = 0x0,
 base_raw = (tv64 = 0x0000030BF0C4523F),
 raw_time = (tv_sec = 0x0D15, tv_nsec = 0x30CFC03F),
 cycle_interval = 0x0001FBD0,
 xtime_interval = 0x0000989680002710,
 xtime_remainder = 0xFFFFFFFFFFFFD8F0,
 raw_interval = 0x00989680,
 ntp_tick = 0x0098968000000000,
 ntp_error = 0x0,
 ntp_error_shift = 0x8,

17 HWT-CPU卡死

18 HWT-rt throttle引起卡死

19 HWT-hotplug失败

== 异常报告v1.4(仅供参考) ==
详细描述: 看门狗复位, 其中CPU4,5,6没有喂狗, 请检查对应调用栈是否存在死锁或长时间关中断
参考信息: MediaTek On-Line> Quick Start> 深入分析看门狗框架
异常时间: 80085.401764秒, Mon May 2 10:33:18 CST 2016


== CPU信息 ==
无喂狗CPU信息:
 CPU4: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux multi_cpu_stop(data=0xFFFFFFC07EF23A08) + 92 <kernel/stop_machine.c:192>
 vmlinux cpu_stopper_thread() + 180 <kernel/stop_machine.c:458>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0B8115880) + 508 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0B81158C0) + 212 <kernel/kthread.c:207>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:808>
 == 栈结束 ==

 CPU5: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux multi_cpu_stop(data=0xFFFFFFC07EF23A08) + 88 <kernel/stop_machine.c:192>
 vmlinux cpu_stopper_thread() + 180 <kernel/stop_machine.c:458>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0B81E8000) + 508 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0B81E8040) + 212 <kernel/kthread.c:207>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:808>
 == 栈结束 ==

 CPU6: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux multi_cpu_stop(data=0xFFFFFFC07EF23A08) + 92 <kernel/stop_machine.c:192>
 vmlinux cpu_stopper_thread() + 180 <kernel/stop_machine.c:458>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0B81E8340) + 508 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0B81E8380) + 212 <kernel/kthread.c:207>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:808>
 == 栈结束 ==

其他CPU信息:
 CPU0: 进程名: (null), 进程标识符(pid): 0
 寄存器异常: SP(0xFFFFFFC00798FFF0)/FP(0x000048D66FC9A8E4)不在同一内核栈
 本地调用栈:
 ...... 0x0000000073E104F4()
 == 栈结束 ==

 CPU1: 空闲

 CPU2: 空闲

 CPU7: 进程名: kworker/u17:3, 进程标识符(pid): 24521
 本地调用栈:
 vmlinux pm_callback_power_off(kbdev=0xFFFFFFC0B16F0000) + 84 <drivers/misc/mediatek/gpu/gpu_mali/mali_midgard/mali-r7p0/drivers/gpu/arm/midgard/platform/mt6755/mtk_config_platform.c:166>
 vmlinux kbase_pm_clock_off(kbdev=0xFFFFFFC0B16F0000, is_suspend=false) + 148 <drivers/misc/mediatek/gpu/gpu_mali/mali_midgard/mali-r7p0/drivers/gpu/arm/midgard/backend/gpu/mali_kbase_pm_driver.c:1010>
 vmlinux kbase_pm_do_poweroff(kbdev=0xFFFFFFC0B16F0000, is_suspend=false) + 84 <drivers/misc/mediatek/gpu/gpu_mali/mali_midgard/mali-r7p0/drivers/gpu/arm/midgard/backend/gpu/mali_kbase_pm_backend.c:201>
 vmlinux kbasep_pm_do_gpu_poweroff_wq(data=0xFFFFFFC0B16F02E0) + 172 <drivers/misc/mediatek/gpu/gpu_mali/mali_midgard/mali-r7p0/drivers/gpu/arm/midgard/backend/gpu/mali_kbase_pm_policy.c:268>
 vmlinux process_one_work(worker=0xFFFFFFC0686E6B40, work=0xFFFFFFC0B16F02E0) + 340 <kernel/workqueue.c:2014>
 vmlinux worker_thread(__worker=0xFFFFFFC0686E6B40) + 316 <kernel/workqueue.c:2146>
 vmlinux kthread(_create=0xFFFFFFC02AB4BC00) + 212 <kernel/kthread.c:207>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:808>
 == 栈结束 ==

kick=0x00000087,check=0x00000070

 Line 375346: [80038.302561] .(2)[215:wdtk-2][name:wd_common_drv&][WDK],local_bit:0x4,cpu:2,check bit0x:ff,3,1,80038291783408,RT[80038302537870]
 Line 375347: [80038.322654] .(1)[214:wdtk-1][name:wd_common_drv&][WDK],local_bit:0x6,cpu:1,check bit0x:ff,3,1,80038291783408,RT[80038322580562]
 Line 378956: [80058.002641] .(0)[213:wdtk-0][name:wd_common_drv&][WDK],local_bit:0x1,cpu:0,check bit0x:70,2,0,80054385300139,RT[80058002570917]
 Line 378959: [80058.222523] .(7)[220:wdtk-7][name:wd_common_drv&][WDK],local_bit:0x81,cpu:7,check bit0x:70,2,0,80054385300139,RT[80058222499764]
 Line 378985: [80058.312706] .(2)[215:wdtk-2][name:wd_common_drv&][WDK],local_bit:0x85,cpu:2,check bit0x:70,2,0,80054385300139,RT[80058312634302]
 Line 378987: [80058.332677] .(1)[214:wdtk-1][name:wd_common_drv&][WDK],local_bit:0x87,cpu:1,check bit0x:70,2,0,80054385300139,RT[80058332619072]

 Line 4: [80038.335538] .(4)[107:hps_main]CPU3: shutdown
 Line 9: [80038.381523] .(1)[107:hps_main]CPU2: shutdown
 Line 10: [80038.386723] .(4)[107:hps_main]CPU1: shutdown
 Line 26: [80038.512832] .(0)[107:hps_main]CPU7: shutdown
 Line 28: [80038.519628] .(6)[107:hps_main]CPU0: shutdown
 Line 70: [80038.901728] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 123: [80039.284082] .(5)[1178:PerfServiceMana]CPU7: shutdown
 Line 155: [80039.570198] .(4)[107:hps_main]CPU6: shutdown
 Line 171: [80040.027143] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 184: [80040.110076] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 191: [80040.127737] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 194: [80040.128942] -(2)[0:swapper/2]CPU2: Booted secondary processor
 Line 197: [80040.130156] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 202: [80040.137509] .(5)[1178:PerfServiceMana]CPU2: shutdown
 Line 203: [80040.140575] .(6)[1178:PerfServiceMana]CPU1: shutdown
 Line 205: [80040.148867] .(5)[1178:PerfServiceMana]CPU0: shutdown
 Line 226: [80040.353843] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 232: [80040.358386] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 235: [80040.359608] -(2)[0:swapper/2]CPU2: Booted secondary processor
 Line 238: [80040.361265] -(3)[0:swapper/3]CPU3: Booted secondary processor
 Line 242: [80040.366672] .(0)[107:hps_main]CPU3: shutdown
 Line 248: [80040.493285] .(0)[107:hps_main]CPU2: shutdown
 Line 255: [80040.579811] .(0)[107:hps_main]CPU1: shutdown
 Line 285: [80040.740389] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 288: [80040.741851] -(2)[0:swapper/2]CPU2: Booted secondary processor
 Line 306: [80040.826069] .(1)[107:hps_main]CPU7: shutdown
 Line 307: [80040.829006] .(0)[107:hps_main]CPU2: shutdown
 Line 308: [80040.831388] .(0)[107:hps_main]CPU1: shutdown
 Line 310: [80040.837553] .(4)[107:hps_main]CPU0: shutdown
 Line 316: [80040.922198] .(4)[107:hps_main]CPU6: shutdown
 Line 329: [80041.193743] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 401: [80042.211584] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 419: [80042.415565] .(4)[107:hps_main]CPU7: shutdown
 Line 465: [80043.421962] .(4)[107:hps_main]CPU6: shutdown
 Line 480: [80043.636278] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 507: [80043.770713] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 514: [80043.779588] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 517: [80043.780561] -(2)[0:swapper/2]CPU2: Booted secondary processor
 Line 520: [80043.781469] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 525: [80043.785328] .(0)[1178:PerfServiceMana]CPU2: shutdown
 Line 526: [80043.788119] .(4)[1178:PerfServiceMana]CPU1: shutdown
 Line 528: [80043.800376] .(4)[1178:PerfServiceMana]CPU0: shutdown
 Line 629: [80045.089281] .(4)[107:hps_main]CPU7: shutdown
 Line 686: [80045.294509] .(4)[107:hps_main]CPU6: shutdown
 Line 729: [80045.819457] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 732: [80045.820535] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 737: [80045.864148] .(6)[107:hps_main]CPU7: shutdown
 Line 740: [80045.908453] .(4)[107:hps_main]CPU6: shutdown
 Line 783: [80046.359628] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 786: [80046.360802] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 945: [80048.197443] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 984: [80048.372879] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 987: [80048.374537] -(2)[0:swapper/2]CPU2: Booted secondary processor
 Line 1004: [80048.401643] .(0)[107:hps_main]CPU2: shutdown
 Line 1005: [80048.404261] .(0)[107:hps_main]CPU1: shutdown
 Line 1009: [80048.410193] .(7)[107:hps_main]CPU0: shutdown
 Line 1040: [80048.735303] .(5)[107:hps_main]CPU7: shutdown
 Line 1052: [80048.980005] .(4)[107:hps_main]CPU6: shutdown
 Line 1063: [80049.128341] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 1077: [80049.344371] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 1086: [80049.387668] .(5)[107:hps_main]CPU7: shutdown
 Line 1135: [80049.752453] .(4)[107:hps_main]CPU6: shutdown
 Line 1141: [80049.790455] -(6)[0:swapper/6]CPU6: Booted secondary processor
 Line 1144: [80049.791477] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 1214: [80050.277939] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 1255: [80050.567390] .(5)[107:hps_main]CPU7: shutdown
 Line 1257: [80050.572914] .(5)[107:hps_main]CPU0: shutdown
 Line 1342: [80051.411362] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 1375: [80051.619328] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 1382: [80051.623771] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 1385: [80051.624921] -(2)[0:swapper/2]CPU2: Booted secondary processor
 Line 1388: [80051.626646] -(3)[0:swapper/3]CPU3: Booted secondary processor
 Line 1392: [80051.631002] .(5)[107:hps_main]CPU3: shutdown
 Line 1393: [80051.635980] .(0)[107:hps_main]CPU2: shutdown
 Line 1394: [80051.641149] .(0)[107:hps_main]CPU1: shutdown
 Line 1420: [80051.889204] .(7)[107:hps_main]CPU0: shutdown
 Line 1423: [80051.934015] .(5)[107:hps_main]CPU7: shutdown
 Line 1502: [80052.338502] -(7)[0:swapper/7]CPU7: Booted secondary processor
 Line 3547: [80054.371641] -(0)[0:swapper/0]CPU0: Booted secondary processor
 Line 3551: [80054.376534] -(1)[0:swapper/1]CPU1: Booted secondary processor
 Line 3555: [80054.382207] -(2)[0:swapper/2]CPU2: Booted secondary processor

[80053.332505] .(6)[107:hps_main]CPU7: failed to come online
......
[80054.371470] .(4)[1178:PerfServiceMana]CPU0: failed to come online
[80054.371641] -(0)[0:swapper/0]CPU0: Booted secondary processor
[80054.371684] -(0)[0:swapper/0]Detected VIPT I-cache on CPU0
[80054.372380] -(0)[0:swapper/0]CPU0: update cpu_capacity 600
[80054.376391] .(4)[1178:PerfServiceMana]CPU1: failed to come online
[80054.376534] -(1)[0:swapper/1]CPU1: Booted secondary processor
[80054.376572] -(1)[0:swapper/1]Detected VIPT I-cache on CPU1
[80054.376873] -(1)[0:swapper/1]CPU1: update cpu_capacity 600
[80054.382058] .(4)[1178:PerfServiceMana]CPU2: failed to come online
[80054.382207] -(2)[0:swapper/2]CPU2: Booted secondary processor

int __cpu_up(unsigned int cpu, struct task_struct *idle)
{
    int ret;

    secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
    __flush_dcache_area(&secondary_data, sizeof(secondary_data));
#ifdef CONFIG_ARM64_IRQ_STACK
    init_irq_stack(cpu);
#endif
    ret = boot_secondary(cpu, idle);
    if (ret == 0) {
        wait_for_completion_timeout(&cpu_running, msecs_to_jiffies(1000));
        if (!cpu_online(cpu)) {
            pr_crit("CPU%u: failed to come online\n", cpu);
            ret = -EIO;
        }
        ......
    }
    ......
}

[80054.371470] .(4)[1178:PerfServiceMana]CPU0: failed to come online
[80054.371641] -(0)[0:swapper/0]CPU0: Booted secondary processor

asmlinkage void secondary_start_kernel(void)
{
    struct mm_struct *mm = &init_mm;
    unsigned int cpu = smp_processor_id();

    aee_rr_rec_hoplug(cpu, 1, 0);
    atomic_inc(&mm->mm_count);
    current->active_mm = mm;
    cpumask_set_cpu(cpu, mm_cpumask(mm));
    aee_rr_rec_hoplug(cpu, 2, 0);
    set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
    printk("CPU%u: Booted secondary processor\n", cpu);
    aee_rr_rec_hoplug(cpu, 3, 0);
    cpu_set_reserved_ttbr0();
    aee_rr_rec_hoplug(cpu, 4, 0);
    flush_tlb_all();
    aee_rr_rec_hoplug(cpu, 5, 0);
    preempt_disable();
    aee_rr_rec_hoplug(cpu, 6, 0);
    trace_hardirqs_off();
    aee_rr_rec_hoplug(cpu, 7, 0);
    if (cpu_ops[cpu]->cpu_postboot)
        cpu_ops[cpu]->cpu_postboot();
    aee_rr_rec_hoplug(cpu, 8, 0);
    cpuinfo_store_cpu();
    aee_rr_rec_hoplug(cpu, 9, 0);
    notify_cpu_starting(cpu);
    aee_rr_rec_hoplug(cpu, 10, 0);
    smp_store_cpu_info(cpu);
    aee_rr_rec_hoplug(cpu, 11, 0);
    set_cpu_online(cpu, true);
    aee_rr_rec_hoplug(cpu, 12, 0);
    complete(&cpu_running); /* 这里通知__cpu_up()完成了bootup */

    aee_rr_rec_hoplug(cpu, 13, 0);
    local_dbg_enable();
    aee_rr_rec_hoplug(cpu, 14, 0);
    local_irq_enable();
    aee_rr_rec_hoplug(cpu, 15, 0);
    local_async_enable();
    aee_rr_rec_hoplug(cpu, 16, 0);
    cpu_startup_entry(CPUHP_ONLINE);
    aee_rr_rec_hoplug(cpu, 17, 0);
}

20 HWT-在TEE OS里卡死

== 异常报告v1.4(仅供参考) ==
详细描述: 看门狗复位, 其中CPU0,1,2没有喂狗, 请检查对应调用栈是否存在死锁或长时间关中断
参考信息: MediaTek On-Line> Quick Start> 深入分析看门狗框架
异常时间: 140.634246秒, Tue Apr 12 19:59:16 CST 2016


== CPU信息 ==
无喂狗CPU信息:
 CPU0: 进程名: (null), 进程标识符(pid): 0
 寄存器异常: SP(0x0000000000000000)异常
 本地调用栈:
 ...... 0x0000000007F08038()
 == 栈结束 ==

 CPU1: 进程名: migration/1, 进程标识符(pid): 11
 本地调用栈:
 vmlinux stop_machine_cpu_stop(data=0xFFFFFFC0ACABFB58) + 84 <kernel/stop_machine.c:427>
 vmlinux cpu_stopper_thread(cpu=1) + 180 <kernel/stop_machine.c:285>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0B5238B00) + 492 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0B524FC30) + 176 <kernel/kthread.c:200>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:764>
 == 栈结束 ==

 CPU2: 进程名: migration/2, 进程标识符(pid): 15
 本地调用栈:
 vmlinux stop_machine_cpu_stop(data=0xFFFFFFC0ACABFB58) + 84 <kernel/stop_machine.c:427>
 vmlinux cpu_stopper_thread(cpu=2) + 180 <kernel/stop_machine.c:285>
 vmlinux smpboot_thread_fn(data=0xFFFFFFC0B5238D40) + 492 <kernel/smpboot.c:160>
 vmlinux kthread(_create=0xFFFFFFC0B524FC30) + 176 <kernel/kthread.c:200>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:764>
 == 栈结束 ==

WDT status: 2 fiq step: 41 exception type: 1
[LAST PC] CORE_0 PC = 0x7f08038( + 0x0), FP = 0x7f12f0c, SP = 0x0
[LAST PC] CORE_1 PC = 0xffffffc0002f3d70(number.isra.2 + 0x1a0), FP = 0xffffffc0b3862f30, SP = 0xffffffc0b3862f30
[LAST PC] CORE_2 PC = 0xffffffc0003d962c(aee_wdt_atf_info + 0x474), FP = 0xffffffc0b3877af0, SP = 0xffffffc0b3877af0

[ATF][ 139.633485]tbase_fastcall_handler TBASE_SMC_FASTCALL_FORWARD_FIQ [ATF][ 139.633537]core 0 EL3 received forwarded FIQ 160 from [ATF][ 139.633588]core 0 is dumped !
[ATF][ 139.633614]=> aee_wdt_dump, cpu 0 
[ATF][ 139.633644]core 1 is dumped !
[ATF][ 139.633671]=> aee_wdt_dump, cpu 1 
[ATF][ 139.633701]core 2 is dumped !
[ATF][ 139.633728]=> aee_wdt_dump, cpu 2 

21 HWT-spinlock嵌套死锁

== 异常报告v1.4(仅供参考) ==
报告解读: MediaTek On-Line> Quick Start> E-Consulter之NE/KE分析报告解读> KE分析报告
详细描述: 看门狗复位, 其中CPU4,5,6没有喂狗, 请检查对应调用栈是否存在死锁或长时间关中断
参考信息: MediaTek On-Line> Quick Start> 深入分析看门狗框架
异常时间: 179.520409秒, Fri Apr 29 16:34:38 CST 2016

== CPU信息 ==
无喂狗CPU信息:
 CPU4: 进程名: PerfServiceMana, 进程标识符(pid): 1088, 中断: 关
 本地调用栈:
 vmlinux arch_counter_get_cntpct() + 4 <arch/arm64/include/asm/arch_timer.h:118>
 vmlinux arch_timer_read_counter() + 16 <drivers/misc/mediatek/mach/mt6755/ca53_timer.c:268>
 vmlinux read_current_timer(timer_value=0xFFFFFFC06BF4B5B8) + 16 <arch/arm64/kernel/time.c:66>
 vmlinux __delay(cycles=1) + 20 <arch/arm64/lib/delay.c:29>
 vmlinux __spin_lock_debug() + 128 <lib/spinlock_debug.c:144>
 vmlinux do_raw_spin_lock(lock=0xFFFFFFC0BB140DC0) + 396 <lib/spinlock_debug.c:207>
 vmlinux __raw_spin_lock_irq() + 52 <include/linux/spinlock_api_smp.h:129>
 vmlinux _raw_spin_lock_irq(lock=0xFFFFFFC0BB140DC0) + 72 <kernel/spinlock.c:153>
 vmlinux __schedule() + 196 <kernel/sched/core.c:3263>
 vmlinux schedule() + 36 <kernel/sched/core.c:3346>
 vmlinux schedule_timeout(timeout=9223372036854775807) + 372 <kernel/timer.c:1458>
 vmlinux __down_common(timeout=9223372036854775807) + 64 <kernel/semaphore.c:221>
 vmlinux __down(sem=console_sem) + 80 <kernel/semaphore.c:238>
 vmlinux down(sem=console_sem) + 68 <kernel/semaphore.c:61>
 vmlinux console_lock() + 28 <kernel/printk.c:2122>
 vmlinux console_cpu_notify(action=2, hcpu=6) + 52 <kernel/printk.c:2104>
 vmlinux notifier_call_chain() + 104 <kernel/notifier.c:127>
 vmlinux __raw_notifier_call_chain(nr_to_call=-1, nr_calls=0) + 8 <kernel/notifier.c:442>
 vmlinux __cpu_notify() + 36 <kernel/cpu.c:240>
 vmlinux cpu_notify(val=2, v=6) + 40 <kernel/cpu.c:248>
 vmlinux _cpu_up(cpu=6, tasks_frozen=0) + 340 <kernel/cpu.c:518>
 vmlinux _cpu_up_profile() + 20 <kernel/cpu.c:538>
 vmlinux cpu_up(cpu=6) + 152 <kernel/cpu.c:613>
 vmlinux hps_algo_cpu_cluster_action(online_cores=2, target_cores=4) + 240 <drivers/misc/mediatek/mach/mt6755/mt_hotplug_strategy_algo.c:127>
 vmlinux hps_algo_amp() + 1840 <drivers/misc/mediatek/mach/mt6755/mt_hotplug_strategy_algo.c:1254>
 vmlinux ppm_limit_callback() + 244 <drivers/misc/mediatek/mach/mt6755/mt_hotplug_strategy_core.c:305>
 vmlinux mt_ppm_main() + 1368 <drivers/misc/mediatek/ppm/src/mt_ppm_main.c:560>
 vmlinux ppm_task_wakeup() + 24 <drivers/misc/mediatek/ppm/src/mt_ppm_main.c:191>
 vmlinux ppm_userlimit_max_cpu_core_proc_write(count=4) + 396 <drivers/misc/mediatek/ppm/src/mt_ppm_policy_user_limit.c:269>
 vmlinux proc_reg_write() + 76 <fs/proc/inode.c:225>
 vmlinux vfs_write(file=0xFFFFFFC09BB8B000, buf=0x0000007F6C7D5E08, count=4, pos=0xFFFFFFC06BF4BEC8) + 164 <fs/read_write.c:446>
 vmlinux SYSC_write() + 48 <fs/read_write.c:494>
 vmlinux SyS_write(buf=547281001992) + 64 <fs/read_write.c:486>
 == 栈结束 ==

 CPU5: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_counter_get_cntpct() + 4 <arch/arm64/include/asm/arch_timer.h:118>
 vmlinux arch_timer_read_counter() + 16 <drivers/misc/mediatek/mach/mt6755/ca53_timer.c:268>
 vmlinux read_current_timer() + 16 <arch/arm64/kernel/time.c:66>
 ...... ......
 vmlinux vsnprintf(args=0xFFFFFFC0A709F7D0) + 132 <lib/vsprintf.c:1440>
 vmlinux snprintf(buf=0xFFFFFFC0A98A6080, size=80) + 80 <lib/vsprintf.c:1631>
 vmlinux __update_entity_runnable_avg() + 296 <kernel/sched/fair.c:1790>
 vmlinux update_entity_load_avg() + 320 <kernel/sched/fair.c:2186>
 vmlinux put_prev_entity(cfs_rq=0xFFFFFFC0BB14CE68, prev=2) + 496 <kernel/sched/fair.c:2815>
 ...... ......
 vmlinux __buffer_unlock_commit() + 52 <kernel/trace/trace.c:1622>
 ...... ......
 vmlinux ftrace_raw_event_sched_switch() + 344 <include/trace/events/sched.h:257>
 ...... ......
 vmlinux preempt_schedule() + 64 <kernel/sched/core.c:3396>
 ...... ......
 vmlinux task_rq_unlock() + 16 <kernel/sched/core.c:361>
 vmlinux set_cpus_allowed_ptr() + 160 <kernel/sched/core.c:5212>
 vmlinux cpuset_change_cpumask(tsk=0xFFFFFFC04E8A0000) + 20 <kernel/cpuset.c:833>
 vmlinux cgroup_scan_tasks(scan=0xFFFFFFC0A709FCA8) + 340 <kernel/cgroup.c:3303>
 vmlinux update_tasks_cpumask() + 40 <kernel/cpuset.c:857>
 vmlinux cpuset_propagate_hotplug_workfn(work=0xFFFFFFC0A456D398) + 316 <kernel/cpuset.c:2099>
 vmlinux process_one_work(worker=0xFFFFFFC0A779F8C0, work=0xFFFFFFC0A456D398) + 404 <kernel/workqueue.c:2182>
 vmlinux worker_thread(__worker=0xFFFFFFC0A779F8C0) + 304 <kernel/workqueue.c:2309>
 vmlinux kthread(_create=0xFFFFFFC0AE19BC00) + 176 <kernel/kthread.c:200>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:764>
 == 栈结束 ==

 CPU6: 进程名: (null), 进程标识符(pid): 0
 本地调用栈:
 vmlinux arch_timer_read_counter() <drivers/misc/mediatek/mach/mt6755/ca53_timer.c:265>
 ...... ......
 ...... ......
 vmlinux __raw_spin_unlock_irqrestore() + 28 <include/linux/spinlock_api_smp.h:159>
 vmlinux _raw_spin_unlock_irqrestore(flags=-272031334328) + 32 <kernel/spinlock.c:177>
 vmlinux unlock_hrtimer_base() + 12 <kernel/hrtimer.c:852>
 vmlinux __hrtimer_start_range_ns() + 388 <kernel/hrtimer.c:1056>
 ...... ......
 vmlinux wake_up_worker() + 28 <kernel/workqueue.c:788>
 vmlinux insert_work(pwq=0xFFFFFFC0B286F000, work=gCmdqContext + 56, extra_flags=0) + 104 <kernel/workqueue.c:1271>
 vmlinux __queue_work(cpu=8, wq=0xFFFFFFC0B284F840, work=gCmdqContext + 56) + 300 <kernel/workqueue.c:1386>
 vmlinux queue_work_on(cpu=8, wq=0xFFFFFFC0B284F840, work=gCmdqContext + 56) + 116 <kernel/workqueue.c:1411>
 vmlinux queue_work() + 12 <include/linux/workqueue.h:469>
 vmlinux cmdq_core_add_consume_task() + 64 <drivers/misc/mediatek/cmdq/v2/cmdq_core.c:3373>
 vmlinux cmdq_irq_handler() + 176 <drivers/misc/mediatek/cmdq/v2/cmdq_driver.c:867>
 ...... 0xFFFFFFC091D8BD50()
 == 栈结束 ==



== 日志信息 ==
kernel log:
[ 148.504613]-(5)[197:kworker/u16:4]BUG: spinlock recursion on CPU#5, kworker/u16:4/197
[ 148.504618]-(5)[197:kworker/u16:4]BUG: spinlock recursion on CPU#5, kworker/u16:4/197
[ 148.505099]-(5)[197:kworker/u16:4]BUG: spinlock already unlocked on CPU#5, kworker/u16:4/197
[ 148.505351]-(5)[197:kworker/u16:4]BUG: spinlock wrong owner on CPU#5, kworker/u16:4/197

Line 2350: [ 148.504645]-(5)[197:kworker/u16:4] lock: console_sem+0x0/0x50, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: -1 value:0x00000000
 Line 2387: [ 148.505111]-(5)[197:kworker/u16:4] lock: logbuf_lock+0x0/0x38, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 value:0x00000000
 Line 2420: [ 148.505361]-(5)[197:kworker/u16:4] lock: logbuf_lock+0x0/0x38, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 value:0x00000000

 Line 164: [ 148.504645]-(5)[197:kworker/u16:4] lock: console_sem+0x0/0x50, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: -1 value:0x00000000
 Line 200: [ 148.505111]-(5)[197:kworker/u16:4] lock: logbuf_lock+0x0/0x38, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 value:0x00000000
 Line 233: [ 148.505361]-(5)[197:kworker/u16:4] lock: logbuf_lock+0x0/0x38, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 value:0x00000000
 Line 266: [ 148.505603]-(5)[197:kworker/u16:4] lock: logbuf_lock+0x0/0x38, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 value:0x00000000
 Line 298: [ 148.506095]-(5)[197:kworker/u16:4] lock: 0xffffffc0bb140dc0, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: 5 value:0x00000001
 Line 325: [ 148.506394]-(5)[197:kworker/u16:4] lock: 0xffffffc0bb140dc0, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: 5 value:0x00000001
 Line 353: [ 149.504635]-(4)[1088:PerfServiceMana] lock: 0xffffffc0bb140dc0, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: 5 value:0x00000001
 Line 385: [ 149.505416]-(6)[757:mobile_log_d.rd] lock: 0xffffffc0bb140dc0, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: 5 value:0x00000001
 Line 432: [ 149.506705]-(5)[197:kworker/u16:4] lock: 0xffffffc0bb140dc0, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: 5 value:0x00000001

[ 148.504645]-(5)[197:kworker/u16:4] lock: console_sem+0x0/0x50, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: -1 value:0x00000000
[ 148.504659]-(5)[197:kworker/u16:4]CPU: 5 PID: 197 Comm: kworker/u16:4 Tainted: G W 3.10.72+ #4
[ 148.473856]-(5)[197:kworker/u16:4]powerkey_kick:primary_display_trigger:2,6
[ 148.504680]-(5)[197:kworker/u16:4]Workqueue: cpuset_hotplug cpuset_propagate_hotplug_workfn
[ 148.504687]-(5)[197:kworker/u16:4]
[ 148.504694]-(5)[197:kworker/u16:4]Call trace:
[ 148.504709]-(5)[197:kworker/u16:4][] dump_backtrace+0x0/0x16c
[ 148.504728]-(5)[197:kworker/u16:4][] show_stack+0x10/0x1c
[ 148.504743]-(5)[197:kworker/u16:4][] dump_stack+0x1c/0x28
[ 148.504758]-(5)[197:kworker/u16:4][] spin_dump+0x6c/0x8c
[ 148.504770]-(5)[197:kworker/u16:4][] spin_bug+0x18/0xdc
[ 148.504782]-(5)[197:kworker/u16:4][] do_raw_spin_lock+0xd0/0x274
[ 148.504795]-(5)[197:kworker/u16:4][] _raw_spin_lock_irqsave+0x4c/0x60
[ 148.504808]-(5)[197:kworker/u16:4][] down_trylock+0x10/0x40
[ 148.504821]-(5)[197:kworker/u16:4][] console_trylock+0x24/0xb8    <= 这里要去拿console_sem
[ 148.504834]-(5)[197:kworker/u16:4][] vprintk_emit+0x1f4/0x5b4
[ 148.504850]-(5)[197:kworker/u16:4][] printk+0xb4/0xc0
[ 148.504862]-(5)[197:kworker/u16:4][] spin_dump+0x40/0x8c
[ 148.504874]-(5)[197:kworker/u16:4][] spin_bug+0x18/0xdc
[ 148.504885]-(5)[197:kworker/u16:4][] do_raw_spin_lock+0xd0/0x274
[ 148.504896]-(5)[197:kworker/u16:4][] _raw_spin_lock+0x40/0x50
[ 148.504910]-(5)[197:kworker/u16:4][] try_to_wake_up+0x204/0x310
[ 148.504922]-(5)[197:kworker/u16:4][] wake_up_process+0x28/0x64
[ 148.504940]-(5)[197:kworker/u16:4][] __up+0x24/0x34
[ 148.504951]-(5)[197:kworker/u16:4][] up+0x50/0x6c
[ 148.504962]-(5)[197:kworker/u16:4][] console_unlock+0x190/0x3bc     <= 到这里正要释放console_sem
[ 148.504973]-(5)[197:kworker/u16:4][] vprintk_emit+0x238/0x5b4
[ 148.504984]-(5)[197:kworker/u16:4][] printk+0xb4/0xc0
[ 148.504995]-(5)[197:kworker/u16:4][] printk_deferred+0xf4/0x168
[ 148.505007]-(5)[197:kworker/u16:4][] set_cpus_allowed_ptr+0x118/0x120
[ 148.505019]-(5)[197:kworker/u16:4][] cpuset_change_cpumask+0x14/0x20
[ 148.505034]-(5)[197:kworker/u16:4][] cgroup_scan_tasks+0x154/0x220
[ 148.505046]-(5)[197:kworker/u16:4][] cpuset_propagate_hotplug_workfn+0x13c/0x208
[ 148.505061]-(5)[197:kworker/u16:4][] process_one_work+0x194/0x524
[ 148.505073]-(5)[197:kworker/u16:4][] worker_thread+0x130/0x3b8
[ 148.505085]-(5)[197:kworker/u16:4][] kthread+0xb0/0xbc

asmlinkage int vprintk_emit(int facility, int level, const char *dict, size_t dictlen, const char *fmt, va_list args)
{
    ......
    local_irq_save(flags);
    this_cpu = smp_processor_id();
    /* Ouch, printk recursed into itself! */
    if (unlikely(logbuf_cpu == this_cpu)) {
        /*
        * If a crash is occurring during printk() on this CPU,
        * then try to get the crash message out but make sure
        * we can't deadlock. Otherwise just return to avoid the
        * recursion and return - but flag the recursion so that
        * it can be printed at the next appropriate moment:
        */
        if (!oops_in_progress && !lockdep_recursing(current)) {
            recursion_bug = 1;
            goto out_restore_irqs;
        }
        zap_locks();
    }
#ifdef CONFIG_DEBUG_SPINLOCK
    if (logbuf_lock.owner!= NULL && logbuf_lock.owner == current) {
        /*
        * If a crash is occurring during printk() on this CPU,
        * then try to get the crash message out but make sure
        * we can't deadlock. Otherwise just return to avoid the
        * recursion and return - but flag the recursion so that
        * it can be printed at the next appropriate moment:
        */
        if (!oops_in_progress && !lockdep_recursing(current)) {
            recursion_bug = 1;
            goto out_restore_irqs;
        }
        zap_locks();
    }
#endif
    lockdep_off();
    raw_spin_lock(&logbuf_lock);
    logbuf_cpu = this_cpu;
    ......
}

static void zap_locks(void)
{
    static unsigned long oops_timestamp;

    if (time_after_eq(jiffies, oops_timestamp) && !time_after(jiffies, oops_timestamp + 30 * HZ))
        return;
    oops_timestamp = jiffies;
    debug_locks_off();
    /* If a crash is occurring, make sure we can't deadlock */
    raw_spin_lock_init(&logbuf_lock);
    /* And make sure that we print immediately */
    sema_init(&console_sem, 1);
}

lock: console_sem+0x0/0x50, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: -1 value:0x00000000

[ 148.506095]-(5)[197:kworker/u16:4] lock: 0xffffffc0bb140dc0, .magic: dead4ead, .owner: kworker/u16:4/197, .owner_cpu: 5 value:0x00000001
[ 148.506105]-(5)[197:kworker/u16:4]CPU: 5 PID: 197 Comm: kworker/u16:4 Tainted: G W 3.10.72+ #4
[ 148.506114]Workqueue: cpuset_hotplug cpuset_propagate_hotplug_workfn
[ 148.506124]-(5)[197:kworker/u16:4]Call trace:
[ 148.506136]-(5)[197:kworker/u16:4][] dump_backtrace+0x0/0x16c
[ 148.506147]-(5)[197:kworker/u16:4][] show_stack+0x10/0x1c
[ 148.506159]-(5)[197:kworker/u16:4][] dump_stack+0x1c/0x28
[ 148.506170]-(5)[197:kworker/u16:4][] spin_dump+0x6c/0x8c
[ 148.506182]-(5)[197:kworker/u16:4][] spin_bug+0x18/0xdc
[ 148.506194]-(5)[197:kworker/u16:4][] do_raw_spin_lock+0xd0/0x274
[ 148.506205]-(5)[197:kworker/u16:4][] _raw_spin_lock+0x40/0x50
[ 148.506216]-(5)[197:kworker/u16:4][] try_to_wake_up+0x204/0x310
[ 148.506228]-(5)[197:kworker/u16:4][] wake_up_process+0x28/0x64
[ 148.506238]-(5)[197:kworker/u16:4][] __up+0x24/0x34
[ 148.506249]-(5)[197:kworker/u16:4][] up+0x50/0x6c
[ 148.506260]-(5)[197:kworker/u16:4][] console_unlock+0x190/0x3bc
[ 148.506271]-(5)[197:kworker/u16:4][] vprintk_emit+0x238/0x5b4
[ 148.506281]-(5)[197:kworker/u16:4][] printk+0xb4/0xc0
[ 148.506292]-(5)[197:kworker/u16:4][] printk_deferred+0xf4/0x168
[ 148.506304]-(5)[197:kworker/u16:4][] set_cpus_allowed_ptr+0x118/0x120
[ 148.506315]-(5)[197:kworker/u16:4][] cpuset_change_cpumask+0x14/0x20
[ 148.506326]-(5)[197:kworker/u16:4][] cgroup_scan_tasks+0x154/0x220
[ 148.506337]-(5)[197:kworker/u16:4][] cpuset_propagate_hotplug_workfn+0x13c/0x208
[ 148.506349]-(5)[197:kworker/u16:4][] process_one_work+0x194/0x524
[ 148.506361]-(5)[197:kworker/u16:4][] worker_thread+0x130/0x3b8
[ 148.506372]-(5)[197:kworker/u16:4][] kthread+0xb0/0xbc

static int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
{
    ......
    if (p->on_rq && ttwu_remote(p, wake_flags))
        goto stat;
}

static int ttwu_remote(struct task_struct *p, int wake_flags)
{
    struct rq *rq;
    int ret = 0;

    rq = __task_rq_lock(p); /* 卡在这里，锁是rq->lock */
    if (p->on_rq) {
        ttwu_do_wakeup(rq, p, wake_flags);
        ret = 1;
    }
    __task_rq_unlock(rq);

    return ret;
}

int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
{
    unsigned long flags;
    struct rq *rq;
    unsigned int dest_cpu;
    int ret = 0;

    rq = task_rq_lock(p, &flags); /* 这里把rq->lock锁拿走了！！！ */

    if (cpumask_equal(&p->cpus_allowed, new_mask))
        goto out;

    if (!cpumask_intersects(new_mask, cpu_active_mask)) {
        ret = -EINVAL;
        printk_deferred("SCHED: intersects new_mask: %lu, cpu_active_mask: %lu\n", new_mask->bits[0], cpu_active_mask->bits[0]); /* 这里最后调用到ttwu_remote()再去拿rq->lock，导致了spinlock嵌套死锁！！！ */
        goto out;
    }
    ......
}

22 HWT-两把不同的spinlock造成死锁

== 异常报告v1.5(仅供参考) ==
报告解读: MediaTek On-Line> Quick Start> E-Consulter之NE/KE分析报告解读> KE分析报告
详细描述: 看门狗复位, 其中CPU0,1,4,5,6,7没有喂狗, 请检查对应调用栈是否存在死锁或长时间关中断
参考信息: MediaTek On-Line> Quick Start> 深入分析看门狗框架
平台 : MT6757
版本 : alps-mp-n0.mp7/userdebug build
异常时间: 49923.584455秒, Sun Dec 4 01:41:54 CST 2016


== CPU信息 ==
无喂狗CPU信息:
 CPU0: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock() + 36 <include/linux/spinlock.h:150>
 vmlinux __raw_spin_lock() + 44 <include/linux/spinlock_api_smp.h:145>
 vmlinux _raw_spin_lock(lock=0xFFFFFFC07DE3EF00) + 60 <kernel/locking/spinlock.c:151>
 ...... 0x0000000000000080()
 == 栈结束 ==

 CPU1: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock_flags() + 36 <include/linux/spinlock.h:157>
 vmlinux __raw_spin_lock_irqsave() + 52 <include/linux/spinlock_api_smp.h:121>
 vmlinux _raw_spin_lock_irqsave(lock=0xFFFFFFC07DE98700) + 68 <kernel/locking/spinlock.c:159>
 ...... 0x0000000000000003()
 ...... ......
 vmlinux rb_time_stamp() + 4 <kernel/trace/ring_buffer.c:699>
 vmlinux rb_reserve_next_event(cpu_buffer=0xFFFFFFC07420FC00) + 148 <kernel/trace/ring_buffer.c:2809>
 vmlinux ring_buffer_lock_reserve() + 144 <kernel/trace/ring_buffer.c:2884>
 vmlinux 0xFFFFFFC001569390()
 ...... ......
 vmlinux event_trigger_unlock_commit() + 136 <include/linux/trace_events.h:517>
 vmlinux trace_event_buffer_commit(fbuffer=0xFFFFFFC005063C20) + 180 <kernel/trace/trace_events.c:298>
 vmlinux trace_event_raw_event_sched_switch(preempt=false, prev=0xFFFFFFC005063C20, next=0xFFFFFFC06BDF9000) + 224 <include/trace/events/sched.h:275>
 vmlinux trace_sched_switch() + 640 <include/trace/events/sched.h:275>
 vmlinux __schedule(preempt=false) + 996 <kernel/sched/core.c:3280>
 vmlinux schedule() + 100 <kernel/sched/core.c:3312>
 vmlinux schedule_timeout(timeout=25) + 300 <kernel/time/timer.c:1541>
 vmlinux kswapd_try_to_sleep() + 824 <mm/vmscan.c:3546>
 vmlinux kswapd(p=contig_page_data) + 912 <mm/vmscan.c:3665>
 vmlinux kthread(_create=0xFFFFFFC0777FC180) + 236 <kernel/kthread.c:209>
 vmlinux ret_from_fork() + 16 <arch/arm64/kernel/entry.S:690>
 == 栈结束 ==

 CPU4: 进程名: aee_dumpstate, 进程标识符(pid): 17405, 中断: 关
 本地调用栈:
 vmlinux cpu_relax() <arch/arm64/include/asm/processor.h:149>
 vmlinux try_to_wake_up(p=0xFFFFFFC077672000, wake_flags=0) + 140 <kernel/sched/core.c:1996>
 vmlinux default_wake_function(curr=0xFFFFFFC005063E08) + 12 <kernel/sched/core.c:3451>
 vmlinux autoremove_wake_function(mode=1, sync=0, key=0) + 20 <kernel/sched/wait.c:293>
 vmlinux __wake_up_common(q=contig_page_data + 6632, mode=1, nr_exclusive=1, wake_flags=0, key=0) + 88 <kernel/sched/wait.c:73>
 vmlinux __wake_up(q=contig_page_data + 6632, mode=1, key=0) + 56 <kernel/sched/wait.c:95>
 vmlinux wakeup_kswapd(zone=contig_page_data, order=4) + 160 <mm/vmscan.c:3721>
 vmlinux wake_all_kswapds() + 1232 <mm/page_alloc.c:2993>
 vmlinux __alloc_pages_slowpath() + 1340 <mm/page_alloc.c:3096>
 vmlinux __alloc_pages_nodemask(gfp_mask=34078752, order=4, zonelist=contig_page_data + 6528, nodemask=0) + 2220 <mm/page_alloc.c:3360>
 vmlinux __alloc_pages() + 16 <include/linux/gfp.h:437>
 vmlinux __alloc_pages_node() + 16 <include/linux/gfp.h:450>
 vmlinux alloc_pages_node() + 16 <include/linux/gfp.h:464>
 vmlinux __get_free_pages() + 20 <mm/page_alloc.c:3409>
 vmlinux alloc_loc_track(t=0xFFFFFFC03BD73D98, max=624) + 76 <mm/slub.c:4327>
 vmlinux mtk_memcfg_add_location() + 1008 <mm/slub.c:5763>
 vmlinux mtk_memcfg_process_slab(t=0xFFFFFFC03BD73D98, s=0xFFFFFFC07DE3C180, page=0xFFFFFFBDC1D65240, map=0xFFFFFFC003110900) + 1268 <mm/slub.c:5798>
 vmlinux mtk_memcfg_list_locations() + 692 <mm/slub.c:5831>
 vmlinux mtk_memcfg_slabtrace_show(m=0xFFFFFFC057537000) + 764 <mm/slub.c:5870>
 vmlinux seq_read(file=0xFFFFFFC059607E00, buf=0x00000000F4E7E698, size=4096, ppos=0xFFFFFFC03BD73EC8) + 424 <fs/seq_file.c:243>
 vmlinux proc_reg_read(file=0xFFFFFFC059607E00, buf=0x00000000F4E7E698, count=4096, ppos=0xFFFFFFC03BD73EC8) + 96 <fs/proc/inode.c:202>
 vmlinux __vfs_read() + 28 <fs/read_write.c:432>
 vmlinux vfs_read(file=0xFFFFFFC059607E00, buf=0x00000000F4E7E698, count=4096, pos=0xFFFFFFC03BD73ED0) + 132 <fs/read_write.c:454>
 vmlinux SYSC_read() + 40 <fs/read_write.c:569>
 vmlinux SyS_read() + 64 <fs/read_write.c:562>
 vmlinux el0_svc() + 48 <arch/arm64/kernel/entry.S:714>
 == 栈结束 ==

 CPU5: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock() + 36 <include/linux/spinlock.h:150>
 vmlinux __raw_spin_lock() + 44 <include/linux/spinlock_api_smp.h:145>
 vmlinux _raw_spin_lock(lock=0xFFFFFFC07DE98700) + 60 <kernel/locking/spinlock.c:151>
 vmlinux 0xFFFFFFC001645000()
 ...... ......
 vmlinux freezable_schedule() + 24 <include/linux/freezer.h:171>
 vmlinux futex_wait_queue_me(hb=0xFFFFFFC005001F80, q=0xFFFFFFC001DFFD40, timeout=0) + 224 <kernel/futex.c:2198>
 vmlinux futex_wait(uaddr=0x0000007404D015B0, flags=0, bitset=4294967295) + 228 <kernel/futex.c:2313>
 vmlinux do_futex(uaddr=0x0000007404D015B0, op=137, val=0) + 360 <kernel/futex.c:3091>
 vmlinux SYSC_futex() + 232 <kernel/futex.c:3151>
 vmlinux SyS_futex(uaddr=498296952240, op=137, val=0, utime=0) + 272 <kernel/futex.c:3119>
 vmlinux el0_svc() + 44 <arch/arm64/kernel/entry.S:713>
 == 栈结束 ==

 CPU6: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock() + 36 <include/linux/spinlock.h:150>
 vmlinux __raw_spin_lock() + 44 <include/linux/spinlock_api_smp.h:145>
 vmlinux _raw_spin_lock(lock=0xFFFFFFC07DE3EF00) + 60 <kernel/locking/spinlock.c:151>
 vmlinux preempt_count_sub(val=1) + 40 <kernel/sched/core.c:3045>
 vmlinux __raw_spin_unlock_irqrestore() + 20 <include/linux/spinlock_api_smp.h:163>
 vmlinux _raw_spin_unlock_irqrestore(flags=-274849011056) + 28 <kernel/locking/spinlock.c:191>
 vmlinux spin_unlock_irqrestore() + 8 <include/linux/spinlock.h:364>
 vmlinux sched_update_nr_heavy_prod(invoker=6, p=event_hash + 688, cpu=2112936896) + 400 <drivers/misc/mediatek/sched/sched_avg.c:681>
 vmlinux inc_nr_heavy_running() + 52 <drivers/misc/mediatek/sched/rq_stats.c:401>
 ...... ......
 ...... ......
 vmlinux slab_free() + 420 <mm/slub.c:2907>
 vmlinux kfree(x=0xFFFFFFC001BBBBB0) + 564 <mm/slub.c:3736>
 vmlinux binder_free_buf(proc=0xFFFFFFC001BBBBB0, buffer=0xFFFFFFC001BBBBB0) + 540 <drivers/android/binder.c:1799>
 vmlinux rb_time_stamp() + 4 <kernel/trace/ring_buffer.c:699>
 vmlinux rb_reserve_next_event(cpu_buffer=0xFFFFFFC001BBBBF0) + 148 <kernel/trace/ring_buffer.c:2809>
 ...... 0x00000000F10A7D1C()
 ...... ......
 vmlinux __do_page_fault() + 244 <arch/arm64/mm/fault.c:271>
 vmlinux do_page_fault(addr=-274811919488, esr=8, regs=17) + 496 <arch/arm64/mm/fault.c:342>
 vmlinux do_mem_abort() + 56 <arch/arm64/mm/fault.c:554>
 vmlinux el0_sync() + 732 <arch/arm64/kernel/entry.S:507>
 == 栈结束 ==

 CPU7: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock_flags() + 36 <include/linux/spinlock.h:157>
 vmlinux __raw_spin_lock_irqsave() + 52 <include/linux/spinlock_api_smp.h:121>
 vmlinux _raw_spin_lock_irqsave(lock=0xFFFFFFC07DE98700) + 68 <kernel/locking/spinlock.c:159>
 ...... 0xFFFFFFC001D27B3C()
 ...... ......
 vmlinux ring_buffer_unlock_commit(buffer=0xFFFFFFC07420F900) + 40 <kernel/trace/ring_buffer.c:2682>
 vmlinux rb_end_commit() + 204 <kernel/trace/ring_buffer.c:2487>
 vmlinux rb_commit(cpu_buffer=0xFFFFFFC07420F900) + 280 <kernel/trace/ring_buffer.c:2566>
 vmlinux ring_buffer_unlock_commit(buffer=0xFFFFFFC07DE3A200) + 40 <kernel/trace/ring_buffer.c:2682>
 vmlinux __buffer_unlock_commit() + 64 <kernel/trace/trace.c:1790>
 vmlinux trace_buffer_unlock_commit(tr=global_trace, buffer=0xFFFFFFC07DE3A200, event=0xFFFFFFC07311BDA4, pc=3) + 80 <kernel/trace/trace.c:1798>
 vmlinux event_trigger_unlock_commit() + 136 <include/linux/trace_events.h:517>
 vmlinux trace_event_buffer_commit() + 180 <kernel/trace/trace_events.c:298>
 vmlinux 0xFFFFFFC0014FCC18()
 ...... ......
 vmlinux __switch_to(next=0xFFFFFFC0717E0000) + 128 <arch/arm64/kernel/process.c:414>
 vmlinux context_switch() + 264 <kernel/sched/core.c:2734>
 vmlinux __schedule(preempt=true) + 656 <kernel/sched/core.c:3281>
 vmlinux preempt_schedule_common() + 40 <kernel/sched/core.c:3353>
 vmlinux preempt_schedule() + 24 <kernel/sched/core.c:3378>
 vmlinux preempt_schedule() + 32 <kernel/sched/core.c:3379>
 vmlinux __raw_spin_unlock_irqrestore() + 56 <include/linux/spinlock_api_smp.h:163>
 vmlinux _raw_spin_unlock_irqrestore(lock=0xFFFFFFC0552666D4, flags=-274847335032) + 64 <kernel/locking/spinlock.c:191>
 vmlinux try_to_wake_up(p=0xFFFFFFC055266000, state=3, wake_flags=0) + 540 <kernel/sched/core.c:2025>
 vmlinux wake_up_process() + 16 <kernel/sched/core.c:2090>
 vmlinux wake_up_q(head=0xFFFFFFC001D27D88) + 84 <kernel/sched/core.c:575>
 vmlinux futex_wake(uaddr=0x00000000E82E43C8, flags=1, nr_wake=1, bitset=4294967295) + 324 <kernel/futex.c:1364>
 vmlinux do_futex(uaddr=0x00000000E82E43C8, op=1, val3=0) + 304 <kernel/futex.c:3095>
 vmlinux C_SYSC_futex() + 84 <kernel/futex_compat.c:200>
 vmlinux compat_SyS_futex(uaddr=3895346120) + 120 <kernel/futex_compat.c:174>
 vmlinux el0_svc() + 48 <arch/arm64/kernel/entry.S:714>
 == 栈结束 ==

CPU0: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock() + 36 <include/linux/spinlock.h:150>
 vmlinux __raw_spin_lock() + 44 <include/linux/spinlock_api_smp.h:145>
 vmlinux _raw_spin_lock(lock=0xFFFFFFC07DE3EF00) 

 CPU1: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock_flags() + 36 <include/linux/spinlock.h:157>
 vmlinux __raw_spin_lock_irqsave() + 52 <include/linux/spinlock_api_smp.h:121>
 vmlinux _raw_spin_lock_irqsave(lock=0xFFFFFFC07DE98700) 

CPU5: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock() + 36 <include/linux/spinlock.h:150>
 vmlinux __raw_spin_lock() + 44 <include/linux/spinlock_api_smp.h:145>
 vmlinux _raw_spin_lock(lock=0xFFFFFFC07DE98700) 

CPU6: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock() + 36 <include/linux/spinlock.h:150>
 vmlinux __raw_spin_lock() + 44 <include/linux/spinlock_api_smp.h:145>
 vmlinux _raw_spin_lock(lock=0xFFFFFFC07DE3EF00) 

CPU7: 进程名: (null), 进程标识符(pid): 0, 中断: 关
 本地调用栈:
 vmlinux arch_spin_lock() + 36 <arch/arm64/include/asm/spinlock.h:40>
 vmlinux do_raw_spin_lock_flags() + 36 <include/linux/spinlock.h:157>
 vmlinux __raw_spin_lock_irqsave() + 52 <include/linux/spinlock_api_smp.h:121>
 vmlinux _raw_spin_lock_irqsave(lock=0xFFFFFFC07DE98700)