Kinde accidentally introduced an authentication bug during a routine feature deployment, which caused some users to be redirected to an incorrect profile on login. The incident lasted 1 hour and only affected users who logged in via OAuth2/OIDC connections during the incident window.

The first user to authenticate via OAuth2/OIDC connections during the incident window became the profile for subsequent users on that same OAuth2/OIDC connection.

For example, say “user 1” authenticated with a social connection at the beginning of the incident. When end users 2, 3, and 4 also authenticated via this same connection during the incident window, they were shown the profile for “user 1”.

Because user profiles were exposed to other users, we are treating the incident as a privacy breach. While the number of impacted users were small, we acknowledge that this doesn’t lessen the seriousness of a user’s data having been potentially exposed to other people.

• Incident window: 3 September between 10:33am to 11:41am Sydney time (12:33am to 01:41am UTC).
• This was not a malicious attack and there was no Kinde data breach from an external attacker.
• The buggy deployment was rolled back 68 minutes after the bug appeared, fixing the issue.
• Users who logged in outside of the incident window (i.e. were already logged in before, or logged in after) were not affected.
• Only OAuth and OIDC type connections were impacted, such as all social authentication connections and the Entra OAuth enterprise connection.
• Less than 150 user profiles globally were exposed to other users during the incident.
• Email, username, SMS, WS-Fed, and SAML connections were not impacted.
• There was no cross-business profile exposure.
• Sessions and tokens generated during the incident window were identified and invalidated by the evening of 3 September, Sydney time.

3 Sep 12:29am UTC (10:29am AEST) - Deployment of change started

3 Sep 12:33am UTC - Bug was active in all regions

3 Sep 12:40am UTC - Full deployment completed

3 Sep 01:07am UTC - First notifications of incorrect logins

3 Sep 01:41am UTC - Change successfully reverted, returning authentication to normal

Total duration where a user could log into the wrong profile: 68 minutes

We released a new social OAuth2 authentication connection on 3 September at 10:33am Sydney time. This introduced a bug where OAuth and OIDC based logins passed through the incorrect user key to the database and added an identity to a single user per OAuth or OIDC connection per business.

We discovered that when the first user (“user 1”) logged into a business, say via Google social, the business ended up with an identity placeholder on that connection type for subsequent logins in that business. This resulted in the profile for “user 1” having new identities added to it each time a user logged in afterwards. This effectively meant “user 2” and “user 3” got logged into the “user 1” profile, once they finished authenticating.

Please see the simplified diagram below.

Once the change was reverted, authentication functioned as normal, however there was still leftover information in the “user 1” profiles and access tokens used during the incident.

We moved immediately into remediation and cleanup.

For context, a single user in Kinde may have multiple identities. For instance, a user may have an email identity, a Google identity and a Microsoft identity. Each of these has a unique user identity ID.

When adding the new custom OAuth connection, an adjustment was made to how the user identity ID was generated, allowing it to be dynamic to accomodate for the new custom connection type. The release had the following logic on the callback for a successful authentication to any OAuth connection:

User identity ID = connection_id + ':' + connection_config.id_key

This appended the key name, but not the value of the key to the generated ID. This resulted in a generic identity ID being assigned to the first logged in user. Subsequent user logins had this same generic identity ID generated, and were effectively treated as the same OAuth connection. When the incident started, “user 1” would log in and get the buggy generic user identity ID added to their account. Every subsequent user logging in would match this generic buggy user identity ID and then assume the profile.

The correct code should have been something like:

User identity ID = connection_id + ':' + profile[connection_config.id_key]

Note the profile ID key’s value is used in this example, which is unique for every identity.

After identifying the cause and reversing the issue, we set about remediating and cleaning up. Here are the actions we took immediately following the incident.

Identified and shared a list of “user 1” users for each connection type in each business

As a priority, we immediately combed logs and compiled a list of all affected users. We focused on identifying “user 1” who’s profile was shown to others, alongside identifying the other users who saw the wrong profiles. We then contacted the businesses these users belonged to, so they could take remediation actions on their side.

Invalidated admin sessions and refresh tokens, and revoked access tokens

We revoked the access and refresh tokens of all “user 1” users, which had the cascading effect of invalidating sessions for all the users who were signed into the “user 1” profile. Invalidating the server-side access and refresh tokens meant all the users had to re-authenticate. We then manually double-checked that all user profile information was appearing correctly.

We worked with customers who allow cached access tokens on the user’s browser, so they could force the invalidation of these access tokens as well.

We also invalidated all Kinde Admin sessions for users who signed in during the incident window.

Identified and shared a list of all users who logged in to businesses during the incident

We then compiled a complete list of users who signed in during the incident, to cross-check and identify anyone else who might have seen “user 1” profiles. The intent was to ensure that no accidental changes or malicious action was taken by this group of users. We did not find any evidence of profile tampering or unauthorized activity.

Fixed the identities assigned to the “user 1” users

The incident caused the profile for “user 1” to have an incorrect identity, which required the engineering team to fix. Based on feedback, we quickly and successfully restored users whose profiles were impacted, deleted, or otherwise incorrectly modified during the incident.

Assisted customers with restoring auth settings that were changed

Some customers, who disabled their OAuth2 connections in response to the incident, had trouble re-enabling their auth and config exactly how it was. The Kinde engineering team used audit logs to asses what had changed, and provided pre-incident setting information to assist customers.

Note from Ross Chaldecott, CEO

 

I personally, along with the Kinde team, take this incident incredibly seriously. The safeguards and actions outlined below are really just the beginning and we'll be doing everything in our power to make sure an incident like this doesn't happen again, and to support our customers and their users at this time. Our team has been working around the clock to remedy things, but if your business is affected and you need additional support, please reach out to us by email, or in our Slack and Discord communities.

We have prioritized a full review of our testing procedures

We released a critical bug that passed all our automated and manual testing. This is unacceptable. Testing showed everything worked fine. Even though it didn’t.

We will update our testing scripts to validate all user IDs throughout the auth flow, to ensure the expected profile is always loaded correctly. We will also map out the more edge case auth flow variations we’ve become aware of, and incorporate this into testing routines.

We will adopt a defensive coding stance

The bug passed a faulty user identity ID back from a completed authentication flow, which meant it landed on an empty default identifier stuck on the first user who authenticated on that OAuth connection. We’re going to review the OAuth flow, as well as other flows such as email, username and enterprise connections, to include more defensive checks that would prevent a scenario like this from happening again.

Create emergency procedures that can quickly block authentication per region

During the incident we found that we couldn’t quickly block all authentication at a moments notice. We have 2 key tools that can assist with this, the cloud infrastructure WAF and our own application throttles. While we can block authentication using both, neither were pre-setup to do this and required additional time to build the appropriate rules and logic.

To assist in future incidents and minimize fallout, we will set up appropriate mechanisms where incident responders can quickly block authentication.

Expand security incident exercises to include scenarios like this incident

We plan to incorporate additional database and log analysis practices into security testing scenarios. This will help us build more efficiency in how we report incident impact and scope, and speed up data queries to reduce incident remediation time.

Fortunately, we frequently test our point-in-time data restoration, which gave us full confidence in tracking activity during the incident. Expanding our exercises to include more assumed breach scenarios will further improve our response speed.

More detailed audit log for customer changes

We plan to review what our debug audit logs have in them and start exposing the relevant information to the Kinde Admin, so that customers can easily view changes to business-related settings. We will add more detail to log events and broaden log scope across our platform. This will assist larger teams in understanding changes without having to wait for Kinde support to retrieve the information.

We were too specific with incident information before we were 100% sure

We made the mistake of making some assumptions early on during the incident about the exact timing of events and the number of impacted users.

We recognize that we must be more careful with preliminary information and be clear about what we’re completely confident with, and what is still to be determined. We don’t want to communicate incorrect information, but at the same time we don’t want to hold information back.

We will work to improve our judgement calls and communication timing in these situations.

What happened was entirely on us and we are genuinely sorry for the disruption and any worry this caused you and your customers. Our plan is to continue building a world with more founders, but we definitely want to build one where founders can rely on us.

This incident has shone a light on several areas for improvement, and we are committed to ensuring an incident like this is impossible to repeat.