Northflank reports AI key abuse and asks customers to rotate secrets

Northflank says customers reported unauthorized AI API key use after a security incident. It requests replacement of all stored secrets. Its October 30 notice directs users to revoke credentials with their issuers and update replacements in Northflank.

On October 31, it opened MySQL and Redis password rotation to all customers, alongside audit-log access for free accounts. It requests AI-provider usage and billing checks.

Environment variables carry settings into application containers. Northflank’s documentation describes values stored on individual services and jobs, or inherited through shared secret groups. A value set directly on a service or job takes precedence over an identically named value inherited from a group. Consequently, a shared group and an individual workload can supply different credentials.

Database rotation has different completion steps

The rotation API documents distinct MySQL and Redis behavior. MySQL’s default mode changes user-facing passwords while preserving usernames. Full mode also changes internal passwords and restarts the addon. Both modes retain old passwords until finalization. Redis restarts and completes rotation in one step, revoking old credentials; clients must reconnect with the new password. Its rotation requires persistence, and Sentinel configurations need at least three replicas.

A separate finalization endpoint permanently removes old credentials after an in-progress rotation. For MySQL, user-facing finalization disconnects affected users’ sessions; full finalization restarts the addon. Redis finishes automatically and does not support that endpoint. The distinction means generating replacement database credentials and invalidating the previous set can be separate events, depending on the database and rotation mode.

Applying a replacement to a running application has its own timing. The secret-group guide says saving group configuration does not restart workloads. Updated runtime variables take effect through service restarts or new job runs. Its restart-dependents control redeploys services and jobs inheriting the group with the new values. Groups can supply build arguments, runtime variables or both, so their configured scope also determines when applications receive them.

For automated changes, Northflank’s API guide says submitted variable maps replace the complete existing map. Sending only one replacement entry through PATCH can therefore remove other entries, rather than merge that entry into the existing collection. The documented procedure reads the current group, merges the intended change, submits the complete set, then reads it back to compare the remaining keys. The same rule applies to the CLI and JavaScript client. The guide also calls for a safe backup before bulk updates, because removed variables cannot be restored from Northflank. Other fields omitted from a request remain unchanged.

Audit logs record event type, triggering user, origin, timestamp and parent events. Configuration changes show before-and-after differences. Organisation, team, project and resource scopes allow review of actions originating from users, Northflank systems or version-control integrations. These records describe changes inside the hosting platform, while the AI provider’s usage and billing records describe activity processed by that provider. Date, event-type and origin filters can narrow that review to changes around credential replacement.

Wiz’s October 30 Black October report covers incidents at nine AI developer platforms. Two enterprise-credential abuse cases involved approximately 800,000 and 1.7 million Bedrock calls. Wiz estimated their combined usage above $100,000 at list prices. The report does not identify the compromised vendor in those two cases, establish actual paid losses, or tie all the platform incidents to a single vulnerability or attacker.