Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Broken crash stack traces with NDK r22 and SDK 29 #1589

Closed
pharscoet opened this issue Sep 25, 2021 · 20 comments
Closed

[BUG] Broken crash stack traces with NDK r22 and SDK 29 #1589

pharscoet opened this issue Sep 25, 2021 · 20 comments
Assignees
@ZijunZhaoCCK
Labels
bug

Comments

@pharscoet
Copy link

Description

I am seeing a strange issue where after upgrading code to use NDK r22, the crash stack traces in the log are bogus or truncated.
And it seems to happen when using a min SDK version of 29 and not with earlier versions (I haven't tried 30). NDK r21 with min SDK 29 does not have the issue.

I am able to reproduce the issue with the native-activity app from the NDK samples.
That sample app has ndkVersion set to 22.1.7171670.
I added a packagingOptions.doNotStrip '**' to the build.gradle file so the symbols are not stripped and an assert(false); in main.cpp:engine_draw_frame() to cause a crash.

When run, the assert produces the stack trace:

09-25 00:02:43.208 13587 13587 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-25 00:02:43.208 13587 13587 F DEBUG   : Build fingerprint: 'samsung/x1qsqx/x1q:10/QP1A.190711.020/G981USQE1ATG1:userdebug/test-keys'
09-25 00:02:43.208 13587 13587 F DEBUG   : Revision: '13'
09-25 00:02:43.208 13587 13587 F DEBUG   : ABI: 'arm64'
09-25 00:02:43.208 13587 13587 F DEBUG   : Timestamp: 2021-09-25 00:02:43+0000
09-25 00:02:43.208 13587 13587 F DEBUG   : pid: 13525, tid: 13580, name: Thread-8  >>> com.example.native_activity <<<
09-25 00:02:43.208 13587 13587 F DEBUG   : uid: 10586
09-25 00:02:43.208 13587 13587 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
09-25 00:02:43.208 13587 13587 F DEBUG   : Abort message: '/home/p.harscoet/src/external/ndk-samples/native-activity/app/src/main/cpp/main.cpp:171: void engine_draw_frame(struct engine *): assertion "false" failed'
09-25 00:02:43.208 13587 13587 F DEBUG   :     x0  0000000000000000  x1  000000000000350c  x2  0000000000000006  x3  00000074ea51ba10
09-25 00:02:43.208 13587 13587 F DEBUG   :     x4  0000800000000000  x5  0000800000000000  x6  0000800000000000  x7  0000000000000018
09-25 00:02:43.208 13587 13587 F DEBUG   :     x8  00000000000000f0  x9  fc919477fc66a6df  x10 0000000000000001  x11 0000000000000000
09-25 00:02:43.208 13587 13587 F DEBUG   :     x12 fffffff0fffffbdf  x13 00000000614e6723  x14 000a36f96e853e42  x15 0000343a9094ffc3
09-25 00:02:43.208 13587 13587 F DEBUG   :     x16 00000075840d28c0  x17 00000075840aefe0  x18 0000007490ec2000  x19 00000000000034d5
09-25 00:02:43.208 13587 13587 F DEBUG   :     x20 000000000000350c  x21 00000000ffffffff  x22 00000074ea51c060  x23 00000074ea51bdd8
09-25 00:02:43.208 13587 13587 F DEBUG   :     x24 00000074ea51bd50  x25 00000074ea51bd50  x26 00000074ea51c020  x27 0000007585a2a020
09-25 00:02:43.208 13587 13587 F DEBUG   :     x28 0000007fdb86c8c0  x29 00000074ea51bab0
09-25 00:02:43.208 13587 13587 F DEBUG   :     sp  00000074ea51b9f0  lr  000000758406027c  pc  00000075840602a8
09-25 00:02:43.208 13587 13587 F DEBUG   : 
09-25 00:02:43.208 13587 13587 F DEBUG   : backtrace:
09-25 00:02:43.209 13587 13587 F DEBUG   :       #00 pc 00000000000832a8  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #01 pc 00000000000839e8  /apex/com.android.runtime/lib64/bionic/libc.so (__assert2+36) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #02 pc 0000000000016ee4  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (engine_draw_frame(engine*)+68) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #03 pc 0000000000016cc4  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (engine_handle_cmd(android_app*, int)+188) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #04 pc 0000000000018ddc  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (process_cmd+72) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #05 pc 0000000000016ae8  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (android_main+376) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #06 pc 0000000000018d78  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (android_app_entry+288) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #07 pc 00000000000e6f10  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #08 pc 00000000000850c8  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: c2f522895fa97ff8c517cefefec04171)

which is correct.

Now, if I change the minSdkVersion to 29 in build.gradle and I run it again, the new stack trace becomes:

09-25 00:03:43.940 14006 14006 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-25 00:03:43.940 14006 14006 F DEBUG   : Build fingerprint: 'samsung/x1qsqx/x1q:10/QP1A.190711.020/G981USQE1ATG1:userdebug/test-keys'
09-25 00:03:43.940 14006 14006 F DEBUG   : Revision: '13'
09-25 00:03:43.940 14006 14006 F DEBUG   : ABI: 'arm64'
09-25 00:03:43.941 14006 14006 F DEBUG   : Timestamp: 2021-09-25 00:03:43+0000
09-25 00:03:43.941 14006 14006 F DEBUG   : pid: 13955, tid: 13997, name: Thread-8  >>> com.example.native_activity <<<
09-25 00:03:43.941 14006 14006 F DEBUG   : uid: 10586
09-25 00:03:43.941 14006 14006 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
09-25 00:03:43.941 14006 14006 F DEBUG   : Abort message: '/home/p.harscoet/src/external/ndk-samples/native-activity/app/src/main/cpp/main.cpp:171: void engine_draw_frame(struct engine *): assertion "false" failed'
09-25 00:03:43.941 14006 14006 F DEBUG   :     x0  0000000000000000  x1  00000000000036ad  x2  0000000000000006  x3  00000074ea51ba10
09-25 00:03:43.941 14006 14006 F DEBUG   :     x4  0000800000000000  x5  0000800000000000  x6  0000800000000000  x7  0000000000000018
09-25 00:03:43.941 14006 14006 F DEBUG   :     x8  00000000000000f0  x9  fc919477fc66a6df  x10 0000000000000001  x11 0000000000000000
09-25 00:03:43.941 14006 14006 F DEBUG   :     x12 fffffff0fffffbdf  x13 00000000614e675f  x14 00359b5f575ac642  x15 000055dea014c8a5
09-25 00:03:43.941 14006 14006 F DEBUG   :     x16 00000075840d28c0  x17 00000075840aefe0  x18 0000007490a00000  x19 0000000000003683
09-25 00:03:43.941 14006 14006 F DEBUG   :     x20 00000000000036ad  x21 00000000ffffffff  x22 00000074ea51c060  x23 00000074ea51bdd8
09-25 00:03:43.941 14006 14006 F DEBUG   :     x24 00000074ea51bd50  x25 00000074ea51bd50  x26 00000074ea51c020  x27 0000007585a2a020
09-25 00:03:43.941 14006 14006 F DEBUG   :     x28 0000007fdb86c8c0  x29 00000074ea51bab0
09-25 00:03:43.941 14006 14006 F DEBUG   :     sp  00000074ea51b9f0  lr  000000758406027c  pc  00000075840602a8
09-25 00:03:43.941 14006 14006 F DEBUG   : 
09-25 00:03:43.941 14006 14006 F DEBUG   : backtrace:
09-25 00:03:43.941 14006 14006 F DEBUG   :       #00 pc 00000000000832a8  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:03:43.941 14006 14006 F DEBUG   :       #01 pc 00000000000839e8  /apex/com.android.runtime/lib64/bionic/libc.so (__assert2+36) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:03:43.942 14006 14006 F DEBUG   :       #02 pc 0000000000016f1c  /data/app/com.example.native_activity-qD7yK_my8queF7hezbbAfA==/base.apk!libnative-activity.so (offset 0x2c000) (BuildId: 5c54360fd3b54ce346c314db3122af123693b9c8)

it is now truncated and not useful. With other code I have seen the trace containing symbols completely unrelated to the crash.

If leaving minSdkVersion at 29 and changing ndkVersion to 21.4.7075529, the correct stack trace is produced.

Environment Details

Not all of these will be relevant to every bug, but please provide as much
information as you can.

  • NDK Version: 22.1.7171670
  • Build system: CMake
  • Host OS: Linux (Ubuntu 18.04)
  • ABI: arm64
  • NDK API level: 29
  • Device API level: 29
@pharscoet pharscoet added the bug label Sep 25, 2021
@DanAlbert
Copy link
Member

r22 is no longer supported. Do you see the same issue with NDK r23?

@pharscoet
Copy link
Author

r22 is no longer supported. Do you see the same issue with NDK r23?

Yes, having the same issue with r23.

@DanAlbert
Copy link
Member

Thanks for confirming. Not a regression from r22 but is from the previous LTS so our rules here are a little hazy. I'd be in favor of a cherry-pick to r23, but it depends on the scope of the fix. For now triaging to r24.

@pharscoet
Copy link
Author

Thanks for confirming. Not a regression from r22 but is from the previous LTS so our rules here are a little hazy. I'd be in favor of a cherry-pick to r23, but it depends on the scope of the fix. For now triaging to r24.

Ok thanks. I noticed the issue on a Galaxy S20 with Android 10 and I just tried an S21 with Android 11 and I'm not seeing the problem there so it could depend on the target device's version too.

@DanAlbert
Copy link
Member

That reminds me of #1196 (comment), but I don't think it's actually related (that issue was fixed in r22 after all).

@pharscoet
Copy link
Author

That reminds me of #1196 (comment), but I don't think it's actually related (that issue was fixed in r22 after all).

It does sound like a similar type of issue.

@pharscoet
Copy link
Author

That reminds me of #1196 (comment), but I don't think it's actually related (that issue was fixed in r22 after all).

Adding -Wl,--no-rosegment with r23 as mentioned in that comment fixes this issue. It's also happening on a somewhat older device (S10 with Android 10) and not on a more recent one and your explanation in that other comment about the unwinder in older devices now being about to understand the output from LLD could explain it too.

@DanAlbert
Copy link
Member

Huh. I saw earlier that we do have a test for this, but maybe the test was only for ndk-build? I'll have to look again tomorrow. Thanks for confirming. Really didn't expect that to be the answer :(

You're just using whatever version of CMake and AGO are the defaults for that sample, right? Only changed the NDK version?

@pharscoet
Copy link
Author

You're just using whatever version of CMake and AGO are the defaults for that sample, right? Only changed the NDK version?

The sample uses NDK r22 by default and I only need to change the min SDK version to 29 to see the issue. Changing the NDK version to r21 does not show the issue even with the min SDK version still at 29. I didn't change the CMake version or any other version.

@DanAlbert
Copy link
Member

Hmm. It sounds like we got the API level for the min version off by one then, but I checked and https://android-review.googlesource.com/c/platform/system/core/+/777288/ was in Q (29). I wonder if Samsung for some reason reverted that patch (I can't imagine why they would, so that seems really unlikely to me), or if that was only a partial fix and we just need to raise that to 30.

@pharscoet
Copy link
Author

Hmm. It sounds like we got the API level for the min version off by one then, but I checked and https://android-review.googlesource.com/c/platform/system/core/+/777288/ was in Q (29). I wonder if Samsung for some reason reverted that patch (I can't imagine why they would, so that seems really unlikely to me), or if that was only a partial fix and we just need to raise that to 30.

I was able to reproduce the same issue with the sample native-activity on a Pixel 3a with Android 10 so it does not appear to be something Samsung changed.

@DanAlbert
Copy link
Member

Excellent, thanks for checking that (was on my list but you beat me to it). Sounds like all we need to do then is just change the < 29 to < 30 then and the bug is fixed. We can include that in r24 for sure, and probably r23c as well.

@pharscoet
Copy link
Author

pharscoet commented Sep 29, 2021 •
edited
Loading

Excellent, thanks for checking that (was on my list but you beat me to it). Sounds like all we need to do then is just change the < 29 to < 30 then and the bug is fixed. We can include that in r24 for sure, and probably r23c as well.

Hopefully it's as simple as that but I don't know enough about the issue to confirm. Thanks for looking into it :-)

@DanAlbert
Copy link
Member

@pharscoet should be fixed in https://ci.android.com/builds/branches/aosp-master-ndk/grid?head=7821997&tail=7821997 if you want to give that a shot.

@pharscoet
Copy link
Author

@pharscoet should be fixed in https://ci.android.com/builds/branches/aosp-master-ndk/grid?head=7821997&tail=7821997 if you want to give that a shot.

this version does fix the issue with the NDK's native-activity sample and with some other code I was seeing this issue with :-)
Will the fix go into r23c or is r24 the earliest it will be merged in?

@DanAlbert
Copy link
Member

Thanks for confirming 👍

It'll be in r24 beta 1 in a couple weeks, but for a stable release r23c will be sooner than r24 stable (both dates tbd, but r23c will be first).

@pharscoet
Copy link
Author

Thanks for confirming 👍

It'll be in r24 beta 1 in a couple weeks, but for a stable release r23c will be sooner than r24 stable (both dates tbd, but r23c will be first).

Ok, so the fix will also be merged into r23c, and not just in r24, correct?

@DanAlbert
Copy link
Member

Yep.

@pharscoet
Copy link
Author

Yep.

Sounds good, thanks!

@DanAlbert
Copy link
Member

Should be fixed in r23 build 8486889.

MaoHan001 pushed a commit to riscv-android-src/platform-ndk that referenced this issue Jun 22, 2022
@ZijunZhaoCCK
Fix incorrect min API level for rosegment.
839963f 
Broken crash stack traces with SDK 29, and need to raise the min SDK version to 30 to solve the problem.

Test: added test
fix bug 1589 android/ndk#1589
Change-Id: Ie39ed3e48048a90caa69a462ff44e48a02c361d8
(cherry picked from commit e3436c7)
Merged-In: Ie39ed3e48048a90caa69a462ff44e48a02c361d8
MaoHan001 pushed a commit to riscv-android-src/platform-ndk that referenced this issue Jun 22, 2022
@DanAlbert
Add changelog note for rosegment fix.
7b6a8ca 
Bug: android/ndk#1589
Test: None
Change-Id: I2890bbcd625ba74d0869649de4b17e10efee7112
(cherry picked from commit ab24010d325f935fe2bb9c74af2f5b9bed0c9315)
Merged-In: I2890bbcd625ba74d0869649de4b17e10efee7112
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ZijunZhaoCCK ZijunZhaoCCK

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DanAlbert @pharscoet @ZijunZhaoCCK