I’m hitting a persistent issue with the Automated Behavior Analysis (ABA) certification tests that is blocking my app update submission.
The test suite appears to be executing tests 2–6 (i.e. Deep Linking and Playback) before running the On-Screen Sign-In script (Test 1). Because our app requires an active login to initiate playback and landing on deep links defaults unauthenticated users to our WelcomeAuthScreen, tests 2–6 continuously fail with:
“While watching media player events, nothing was found. The active application Nitro.Film (dev) shows “WelcomeAuthScreen” as the active element. Suggest looking if not logged in, channel UI has changed or maximum streams limit was reached”
It is expect for a subscription service that you need to be signed in with an account for playback to occur whether that happens from a deeplink or inside the app. Because it’s not running the sign in script first as expected, it’s hitting the sign in screen and thinks it failed. If you sign in at that point, playback will occur. If you’re already signed in, the deeplink works. And when the Sign-In and Sign-Out scripts do eventually run at the very end of the test execution, they PASS.
Per the Roku Authenticated Cert Testing documentation, testing devices are supposed to sign in first, run the deep linking/performance tests while authenticated, and sign out at the end. This is not happening.
Has anyone encountered this sequence issue where ABA runs deep linking before executing the sign-in script, or found a fix/workaround to ensure authentication completes first? I’ve had this issue on every prior submission and the workaround was to just, submit it again and eventually it tests in the right order and passes. I’ve had 4 prior releases this way but this time the workaround is not working. I opened a ticket a few days ago but no response yet and I’m still blocked.
Thank you. Saw your note and provided additional info. Hopefully we can find the root cause so I can continue with my submission now and going forward.
I’m running into this as well when running the App Behavior Analysis tool.
Sign In Flow and Sign Out Flow both pass in ABA. However, all other tests fail with screenshots showing the login screen.
Locally, replaying our sign-in script succeeds, and the exact deep link from the failing test reaches playback, including after exiting and relaunching the app.
Does ABA execute the uploaded sign-in script before each independent test? Where can we inspect that setup execution? The displayed playback script contains only launch and wait-for-player-state steps.
similar issue as mine. Partner success never responded to my ticket. No further help here either. It is not resolved. I ended up just skipping ABA and submitting anyway.
I got a follow up from the team that of the users who reported issues with behavior analysis failing, there was no reproducible or distinguishing connection for the failures across the different apps.
@dhoffman can you DM me your channel ID and when you experienced this, and I can include that in the ticket and share with engineering to see if there is any connection? Thanks!
I was able to re-upload my app/package and skip running the App Behavior Analysis Tool. I was able to schedule a publish (and get published) without running it.
