Utilizing OAuth2 with a browser popup for authentication can be challenging when implementing automated testing in an Android Emulator.

We’ve invested some effort implementing a fully automated end-to-end test while developing the FusionAuth Android SDK and made the tests available in the Android Kotlin - Fusionauth SDK Quickstart as well.

In this blog, we explore the importance of implementing end-to-end testing, how to create a test using a real FusionAuth backend, and discuss key considerations for an automated testing environment. For the most up-to-date information on our live implementation, please visit: Android Kotlin - Fusionauth SDK Quickstart - Testing.

Why We Need End-To-End Testing

Implementing FusionAuth as the authentication and authorization platform enables seamless integration of various features throughout your application. This includes essential functions such as identity management, authentication and authorization, as well as advanced capabilities such as detailed user profiles, role management, or custom registrations. As a result, numerous user-facing functionalities within the Android App will rely on FusionAuth.

To ensure the reliability of our Android SDK, we aim to verify that all SDK functionalities integrated into our Android App work correctly with each new FusionAuth release and across multiple versions of Android. This testing is automated, ensuring thorough checks for compatibility and functionality with every new update.

Once the end-to-end test suite has been thoroughly executed against various updates and releases throughout the development cycle, we can confidently use it as pass criteria for any pull request, and as a test before the release workflows in the GitHub project.

Developers often reach for mocks to simulate authentication flows, but mocking critical services like FusionAuth can be dangerously misleading. As discussed in The Top 6 Indications You’re Doing Mocking Wrong, mocks hide real-world failures, creating a false sense of stability. The article highlights how mocking leads to brittle tests, increased maintenance, and missed integration issues especially in complex, interconnected systems.

When mocking authentication in Android apps, you risk several pitfalls:

1. False sense of security - Mocks only behave the way you program them to, giving you the illusion that your system is working perfectly while masking breaking changes in the real authentication flow.
2. Increased maintenance burden - As APIs evolve, you’ll constantly need to update your test suite to keep up with changes in the authentication service.
3. Inability to test interconnections - In a distributed system like Android apps with OAuth and OIDC authentication, service interactions are dynamic and often undocumented, making mocks unreliable.
4. Delayed discovery of integration issues - Problems with real authentication flows are only discovered later in the development cycle, often in QA or worse, in production.

In contrast, using a real FusionAuth instance ensures that our Android app is tested against actual login flows, user roles, and token handling — just like it would be in production. This “local-first” development approach (as opposed to “mock-driven” development) catches regressions early, validates the entire authentication flow, and improves confidence with every release.

How We Planned To Achieve Android E2E Testing

Initially, the test needed to operate within the IDE where development takes place. To automate the frequent manual clicks an SDK developer performs when testing the Android app, we first convert these interactions into a test script. This script can be executed repeatedly in the IDE, ensuring consistent and efficient testing.

During this stage, it’s crucial to consider that testing OAuth2 with a browser popup for authentication in conjunction with an Android app can be challenging when conducting automated testing in an Android Emulator. This requires the use of testing libraries capable of handling both scenarios effectively.

In the second step, we integrate this test into GitHub workflows to fully automate it for every new pull request and release. This involves selecting appropriate GitHub actions to implement the workflow and configuring Dependabot to assist with automated dependency management.

This requires careful consideration to ensure that the tests function reliably across different Android versions. Setting up multiple emulators and testing environments repeatedly is essential to cover various scenarios.

The End-To-End Test Implementation

In this section, our focus shifts to the steps taken to build the automated test. Initially, we needed to understand the specific test cases and gather the required test data, along with ensuring all necessary prerequisites for successful automated execution. Furthermore, we implemented measures to ensure adequate data collection in case of test failures, facilitating effective debugging.

Test Cases

We aim to test all relevant use cases for this SDK, which include: Login Refresh Token User Info Logout, in combination with certain Test Data.

At first glance, these tests may seem trivial, but we are seeking specific results for each use case and using them repeatedly in automated testing.

Login

• Does it redirect to FusionAuth?
• Does a successful login result in a redirect to the Home screen?

Refresh Token

• Is it refreshing the token?

User Info

• Is the User Info received?
• Are all the different data sets handled properly depending on the User Info returned?

Logout

• Does it redirect to FusionAuth?
• Does it invalidate the user session?
• Does it return to the Login screen?

Test Data

All the relevant data for testing is defined in FusionAuth including multiple flavors of Applications and Users.

Kickstart Details

Why 10.0.2.2

Typically, the Tenant Issuer is set to your developer machine or localhost. However, with the Android Emulator being a virtual machine, this setup works slightly differently. Set up Android Emulator networking explains in more details how it works. In our case, using the Android Emulator allows us to standardize the development environment and enables the creation of test automation in GitHub workflows.

In the case of the Android Emulator, 10.0.2.2 serves as a special alias to your host loopback interface (127.0.0.1 on your development machine), where FusionAuth is running. Similarly, in the case of GitHub workflows, it refers to the virtual machine on which the workflow job is executed.

Test Automation

The Quickstart includes a Full End 2 End Test that utilizes all the different functionalities provided by the example app.

Test Automation Prerequisites

With the test cases and test data defined, we have to specify additional prerequisites for our testing environment and strategy.

Emulator Image

Starting the emulator requires the correct image for the test; we’re using the google_apis images for the last 5 API levels.

Specifically, we are using API levels 29, 30, 31, 33, 34 and 35. API level 32 is omitted because it is a special Android 12L release designed for tablets and foldables with different layouts, which is not a focus for our testing.

Android Studio provides tools to configure and handle emulators directly within the IDE. Alternatively, you can also manage it in the emulator commandline. These tools and command line options differ depending on the chosen image.

When automating tests using a test matrix, it’s crucial to note that starting from API level 31, the architecture changes to x86_64 from the previous x86 .

This results in the following six emulator configurations:

You can find an example of such automation in the e2e-test workflows of the FusionAuth Android SDK.

Browser

Every time an emulator is started for the first time, the browser setup is not suitable for testing because it may display different modals such as Welcome to Chrome or Sign in to Chrome, which can vary with each Chrome version.

To prevent Chrome from displaying any of these modals, regardless of the version, we execute the following commands once the emulator is started:

adb shell pm clear com.android.chrome
adb shell am set-debug-app --persistent com.android.chrome
adb shell 'echo "chrome --disable-fre --no-default-browser-check --no-first-run" > /data/local/tmp/chrome-command-line'

This setup is required only once for your emulator used in local development but becomes crucial during fully automated testing scenarios in a workflow where the emulator is set up from scratch for each test.

With Android Studio, you can start the emulator and manually open Chrome to bypass the modals, or utilize the adb command (Android Debug Bridge) located in your Android SDK installation:

$HOME/Android/Sdk/platform-tools/adb

Recording

Although there are extensive log details available in your IDE and automated workflow to aid in debugging a failed test, it is beneficial to observe the test directly in the emulator or record the test as a video in your workflow for additional debugging input and context.

If you automate your test in a workflow, you can include this command before starting the test:

adb emu screenrecord start --time-limit 300 ./recording_video.webm

Depending on the build time of your app, you may see only a mobile screen for some time until your app starts and is displayed.

Test Automation Gradle Task

With the test cases and test data defined and additional prerequisites for our testing environment and strategy in place, we are able to create the FullEnd2EndTest Gradle task, which we will explain in more detail below.

FullEnd2EndTest.kt

package io.fusionauth.sdk

import android.accessibilityservice.AccessibilityServiceInfo
import android.view.accessibility.AccessibilityWindowInfo
import androidx.test.espresso.Espresso.onView
import androidx.test.espresso.action.ViewActions.click
import androidx.test.espresso.assertion.ViewAssertions.matches
import androidx.test.espresso.intent.Intents
import androidx.test.espresso.matcher.ViewMatchers.isDisplayed
import androidx.test.espresso.matcher.ViewMatchers.withId
import androidx.test.ext.junit.rules.ActivityScenarioRule
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import androidx.test.uiautomator.By
import androidx.test.uiautomator.UiDevice
import androidx.test.uiautomator.Until
import io.fusionauth.mobilesdk.AuthorizationManager
import org.junit.After
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import java.util.logging.Logger

/**
 * This class represents a full end-to-end test for the application.
 * It performs a series of UI actions and checks for certain conditions to verify the functionality of the application.
 *
 * @property loginActivityRule The rule used to launch the LoginActivity before each test.
 * @property repeatRule The rule used to repeat the test multiple times.
 */
@RunWith(AndroidJUnit4::class)
internal class FullEnd2EndTest {
    @get:Rule
    val loginActivityRule = ActivityScenarioRule(LoginActivity::class.java)

    @get:Rule
    val repeatRule = RepeatRule()

    @Before
    fun setUp() {
        logger.info("Setting up test")

        Intents.init()

        val automation = InstrumentationRegistry.getInstrumentation().uiAutomation
        val info = automation.serviceInfo
        info.flags = info.flags or AccessibilityServiceInfo.FLAG_RETRIEVE_INTERACTIVE_WINDOWS
        automation.serviceInfo = info
    }

    /**
     * Executes an end-to-end test for the login functionality. It performs the following steps:
     * 1. Clicks the login button.
     * 2. Waits for the login form to appear.
     * 4. Sets the username and password on the login form.
     * 5. Submits the form by pressing the enter key.
     * 6. Waits for the token activity to be displayed.
     * 7. Checks the refresh token functionality.
     * 8. Checks if the token was refreshed.
     * 9. Clicks the sign-out button.
     * 10. Waits for the login activity to be displayed.
     *
     * This test is repeated twice to ensure logout was successful and the login form is displayed again.
     */
    @Test
    @Repeat(2)
    fun e2eTest() {
        logger.info("Click login button")
        onView(withId(R.id.start_auth)).perform(click())
        logger.info("Login button clicked")

        logger.info("Waiting for login form to appear")
        val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())

        handleFALoginForm(device, USERNAME, PASSWORD)

        // Check that the token activity is displayed
        device.wait(Until.findObject(By.res("io.fusionauth.app:id/sign_out")), TIMEOUT_MILLIS)
        onView(withId(R.id.sign_out)).check(matches(isDisplayed()))

        logger.info("Token activity displayed")

        // Check refresh token functionality
        val expirationTime = AuthorizationManager.getAccessTokenExpirationTime()!!
        logger.info("Check refresh token")
        onView(withId(R.id.refresh_token))
            .check(matches(isDisplayed()))
            .perform(click())

        val newExpirationTime = AuthorizationManager.getAccessTokenExpirationTime()!!

        logger.info("Token was refreshed (${expirationTime} to ${newExpirationTime})")
        check(newExpirationTime > expirationTime) { "Token was not refreshed" }

        Thread.sleep(1000)

        // Click the sign-out button
        logger.info("Click sign out button")
        onView(withId(R.id.sign_out)).perform(click())

        // Check that the login activity is displayed
        logger.info("Check that the login activity is displayed")
        device.wait(Until.findObject(By.res("io.fusionauth.app:id/start_auth")), TIMEOUT_MILLIS)
        onView(withId(R.id.start_auth)).check(matches(isDisplayed()))

        logger.info("Click login button for second user login")
        onView(withId(R.id.start_auth)).perform(click())
        logger.info("Login button clicked")

        logger.info("Waiting for login form to appear")

        handleFALoginForm(device, USERNAME2, PASSWORD2)

        // Check that the token activity is displayed
        device.wait(Until.findObject(By.res("io.fusionauth.app:id/sign_out")), TIMEOUT_MILLIS)
        onView(withId(R.id.sign_out)).check(matches(isDisplayed()))

        logger.info("Token activity displayed for second user")

        // Click the sign-out button
        logger.info("Click sign out button for second user")
        onView(withId(R.id.sign_out)).perform(click())

        // Check that the login activity is displayed
        logger.info("Check that the login activity is displayed")
        device.wait(Until.findObject(By.res("io.fusionauth.app:id/start_auth")), TIMEOUT_MILLIS)
        onView(withId(R.id.start_auth)).check(matches(isDisplayed()))

    }

    /**
     * Sets the username and password on the login form.
     *
     * @param device The UiDevice used to interact with the UI.
     * @param username The username to set on the login form.
     * @param password The password to set on the login form.
     */
    private fun handleFALoginForm(
        device: UiDevice,
        username: String,
        password: String
    ) {
        device.wait(
            Until.findObject(By.clazz("android.webkit.WebView")),
            TIMEOUT_MILLIS
        )

        val textFields = device.findObjects(By.clazz("android.widget.EditText"))

        closeKeyboardIfOpen()
        logger.info("Set username")
        val userNameInputObject = textFields[0]
        userNameInputObject.setText(username)

        closeKeyboardIfOpen()
        logger.info("Set password")
        val passwordInputObject = textFields[1]
        passwordInputObject.setText(password)

        // Submit the form by pressing the enter key
        logger.info("Submit form by pressing enter key")
        passwordInputObject.click()
        device.pressEnter()
    }

    /**
     * Closes the keyboard if it is open on the screen.
     *
     * When the (automated test) device has a small vertical resolution, the keyboard may be open and cover the login
     * form, thus preventing the UISelector from targeting the form fields.
     *
     * @throws IllegalStateException if the keyboard cannot be closed.
     */
    private fun closeKeyboardIfOpen() {
        val automation = InstrumentationRegistry.getInstrumentation().uiAutomation
        for (window in automation.windows) {
            if (window.type == AccessibilityWindowInfo.TYPE_INPUT_METHOD) {
                UiDevice.getInstance(InstrumentationRegistry.getInstrumentation()).pressBack()
                return
            }
        }
    }

    @After
    fun tearDown() {
        logger.info("Tearing down test")

        Intents.release()
    }

    companion object {
        private val logger = Logger.getLogger(FullEnd2EndTest::class.java.name)

        private const val USERNAME = "richard@example.com"
        private const val PASSWORD = "password"
        private const val USERNAME2 = "gilfoyle@example.com"
        private const val PASSWORD2 = "password"
        private const val TIMEOUT_MILLIS = 10_000L
    }
}

Test Automation Setup

The setUp test initialization includes the following steps:

• Initializes Intents.
• Sets up uiAutomation to interact with the system UI.

Automated End-to-End Test

This test validates the application by executing the following steps, as a user would:

1. Taps the login button (start_auth).
2. Waits for the login form to appear.
3. Enters the username (richard@example.com) and password (password) in the login form.
4. Submits the form using the enter key.
5. Waits for the token view to be shown (sign_out).
6. Checks the refresh token functionality by comparing the token expiration time before and after the refresh.
7. Taps the logout button (sign_out).
8. Waits for the login activity to be displayed again.
9. Repeats steps 3-8 for a second user, using username gilfoyle@example.com and their password.

This test is then repeated a second time with RepeatRule to ensure logout was successful and the login form is displayed again.

Helper methods are utilized throughout the test to interact with the UI:

• closeKeyboardIfOpen: Closes the system’s keyboard if it’s open to prevent it from obscuring the UI elements being interacted with.

Test Teardown

The test concludes by releasing the initialized intents in the tearDown method.

Constants

The constants used in the test include:

• USERNAME, PASSWORD: Credentials of the first user.
• USERNAME2, PASSWORD2: Credentials of the second user.
• TIMEOUT_MILLIS: The duration for which the test waits for the UI elements to appear in the system, expressed in milliseconds.

Please note that the username, password, and timeouts would typically be environment-specific and are not part of the test code itself.

GitHub Automation

Once the automation in Android Studio works as expected, we can proceed to automate the workflows in GitHub.

Workflow

In the case of the Android SDK we created 3 different workflows:

The Weekly Testing ensures that every new version of FusionAuth is automatically tested. Pull Request Testing ensures backward compatibility with multiple versions of Android and the latest successfully tested FusionAuth version. Pre-Release and Release Testing ensure a certain level of backward compatibility is maintained with FusionAuth.

With the Latest Android API level we follow the minimum requirement of Target API level requirements for Google Play apps.

In all the workflows, we employ specific actions to set up and execute the end-to-end tests in GitHub.

Create and Start FusionAuth Containers Used by E2E Test

Since we are testing against a real FusionAuth backend, we utilize the fusionauth-action to create a running instance.

- name: Start FusionAuth
  uses: fusionauth/fusionauth-github-action@v1.0.4
  with:
    FUSIONAUTH_VERSION: ${{ env.fusionauth-docker-image-version }}
    FUSIONAUTH_APP_KICKSTART_DIRECTORY_PATH: fusionauth/${{ env.fusionauth-docker-image-version }}/kickstart

Android Emulator Runner

Running the Android Emulator, configuring Chrome, recording, and executing the test is done using Android Emulator Runner.

- name: run tests
  uses: reactivecircus/android-emulator-runner@v2.31.0
  with:
    api-level: ${{ matrix.api-level }}
    target: ${{ matrix.target }}
    arch: ${{ matrix.arch }}
    script: |
      adb shell pm clear com.android.chrome
      adb shell am set-debug-app --persistent com.android.chrome
      adb shell 'echo "chrome --disable-fre --no-default-browser-check --no-first-run" > /data/local/tmp/chrome-command-line'
      adb emu screenrecord start --time-limit 300 ./recording_video.webm
      ./gradlew connectedAndroidTest

Upload E2E Test Recording

And in case of a failure, we upload the screen recording with Upload a Build Artifact for additional context during debugging.

- name: Upload recording
  uses: actions/upload-artifact@v4.3.3
  if: ${{ failure() }}
  with:
    name: 'E2E Test recording - ${{ matrix.api-level }} ${{ matrix.target }} ${{ matrix.arch }} ${{ env.fusionauth-docker-image-version }}'
    path: recording_video.webm

Dependabot

Once the end-to-end tests are fully automated in GitHub workflows, which are triggered on each pull request to main, the next step is to create a Dependabot configuration with grouped dependencies for GitHub Actions and Gradle.

This setup helps verify changes to the GitHub workflows themselves by automatically testing them. Dependabot creates pull requests with all dependencies updated at once, which is especially crucial for Gradle dependencies where updates are frequently released for a group of packages simultaneously.

Maintain dependencies for GitHub Actions

For GitHub Actions, we create a Dependabot configuration to check both development and production dependencies on a weekly basis. Although currently all actions are treated as production dependencies by Dependabot, this behavior could change in the future. Therefore, we include both to ensure coverage.

Versioning your GitHub Actions to MAJOR.MINOR.PATCH provides a more stable testing environment. Dependabot addresses the otherwise problematic disadvantage of manually keeping tests up to date.

- package-ecosystem: "github-actions"
  directory: "/" # Uses default location of GitHub Workflows
  schedule:
    interval: "weekly"
  groups:
    prod-github-actions:
      dependency-type: "production"
    dev-github-actions:
      dependency-type: "development"

Maintain Dependencies for Gradle

For Gradle, we’ve opted to implement a straightforward rule: create a weekly check that groups all minor and patch releases into one pull request, while handling major version changes in separate PRs. It’s important to note that once Dependabot includes a dependency in a pull request, it won’t use it in another PR simultaneously.

This approach simplifies our configuration to handle most cases without diving into the details of Gradle dependency management. Major versions often introduce breaking changes that typically require manual intervention regardless.

- package-ecosystem: "gradle"
  directory: "/"
  schedule:
    interval: "weekly"
  groups:
    gradle-minor-dependencies:
      update-types:
        - "minor"
        - "patch"

Conclusion

Implementing FusionAuth enables seamless integration of various features, ranging from basic identity management to advanced functionalities such as detailed user profiles and role management. Given that user-facing functionalities heavily depend on FusionAuth, it is crucial to verify the correct operation of all features with each new release. Automated end-to-end testing ensures comprehensive validation of compatibility and functionality.

Unlike mocking, which can create a false sense of stability by simulating ideal conditions, real-world tests surface unexpected regressions early as highlighted in The Top 6 Indications You’re Doing Mocking Wrong. Mocks don’t catch changes in authentication flows, browser behavior, or token lifecycles — all of which are critical in OAuth-based systems like ours.

As explained in To Mock Or Not To Mock Your Auth, authentication is not just a gate but the foundation of your application’s security and user experience. When you mock authentication in security-sensitive environments like mobile apps, you’re likely bypassing critical security checks including JWT signature validation, audience and expiration checks, and MFA enforcement. For Android apps specifically, this means you might miss real-world issues with browser redirects, token storage, and session management that only appear when testing with a real authentication provider.

Although automating end-to-end testing for an Android app with OAuth can be complex, we successfully make use of it for our SDK.

Our end-to-end test suite has been tested against numerous updates and releases during the SDK development. It serves as a pass criterion for pull requests and pre-release checks in our GitHub project. With this setup, we have successfully and proactively identified functional issues, including one related to compatibility with the new scope feature of FusionAuth and another with external libraries.

With these robust automated tests in place, we maintain consistent standards of quality and reliability for our Android SDK, ensuring a seamless and secure user experience.