Quarkus GitHub App: Fixing Installation Deletion Errors

by ADMIN 56 views

Hey everyone! Let's dive into a common issue faced when using quarkus-github-app: errors triggered by installation.deleted events. This happens because the framework automatically tries to create an installation client, even after the installation has been deleted on GitHub. This article will walk you through the problem, the errors, the attempts made to fix it, and a potential solution.

Understanding the Problem

When working with quarkus-github-app, you might encounter errors when processing installation.deleted events. The root cause? The framework's automatic attempt to create an installation client. Before your event handler even gets a chance to run, the framework tries to fetch installation details from the GitHub API. But, if the installation is already deleted on GitHub's end, this results in a GHFileNotFoundException. This can be a real headache, especially when you're trying to keep your logs clean and focus on actual issues.

The problem stems from the framework's eagerness to create an installation client before calling the event handler. This is a design choice that aims to simplify access to GitHub resources within your event handlers. However, in the case of installation.deleted events, this behavior backfires, as the installation is, well, already gone! This leads to unnecessary error spam and can obscure other, more critical issues in your logs. Let's look closer at the error details to understand what's going on.

Think of it like this: you're trying to call someone who just disconnected their phone line. The framework is diligently trying to get through, but the line is dead, resulting in an error. In the context of GitHub Apps, the "phone line" is the installation, and the framework is trying to fetch its details. When an app is uninstalled, that "line" is disconnected, leading to the GHFileNotFoundException. This automatic client creation, while helpful in most scenarios, becomes a hindrance when dealing with deletion events. This is particularly frustrating because these events are a normal part of the application lifecycle, and you'd expect the framework to handle them gracefully.

Diving into the Error Details

Let's break down the error details to get a clearer picture of what's happening under the hood.

The Error Log

Here's a typical error log you might see:

{
  "loggerName": "io.quarkiverse.githubapp",
  "level": "ERROR",
  "message": "Error handling delivery e2e91ff2-aa61-11f0-9ff3-8b09c267156a\n› Event: installation.deleted\nException",
  "exception": {
    "exceptionType": "java.lang.IllegalStateException",
    "message": "Unable to create a GitHub token for the installation 90169271",
    "causedBy": {
      "exceptionType": "org.kohsuke.github.GHFileNotFoundException",
      "message": "https://api.github.com/app/installations/90169271 {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/apps/apps#get-an-installation-for-the-authenticated-app\",\"status\":\"404\"}"
    }
  }
}

As you can see, the log clearly indicates an error occurring while handling the installation.deleted event. The key part is the GHFileNotFoundException, which tells us that the framework couldn't find the installation on GitHub's side. This is exactly what we'd expect when an app is uninstalled, but the framework treats it as an error nonetheless.

Key Points in the Stack Trace

Looking at the stack trace, a few key points emerge:

  • The error originates in GitHubService.createInstallationClient() at line 181. This confirms that the issue lies within the framework's client creation logic.
  • The framework calls GHApp.getInstallationById(), which returns a 404 error. This is the direct cause of the GHFileNotFoundException. The framework is trying to fetch installation details, but the installation is no longer there.
  • Crucially, this happens before the actual event handler method is invoked. This means that even if your handler doesn't need the installation client, the framework still tries to create it, leading to the error.

This analysis confirms that the framework's automatic installation client creation is the culprit. It highlights the need for a way to either bypass this creation for specific events or handle the resulting GHFileNotFoundException more gracefully.

Use Case: Webhook Proxy/Forwarder

To better understand the impact of this issue, let's consider a specific use case: using quarkus-github-app as a webhook proxy/forwarder. In this scenario, the application acts as an intermediary, receiving GitHub App webhook events and forwarding them to a downstream microservice for processing. The flow typically looks like this:

  1. A gateway service receives GitHub App webhook events.
  2. The gateway forwards the raw payload to a downstream microservice.
  3. The downstream service processes the event.

In this setup, the gateway service doesn't need the installation client. It's simply responsible for receiving and forwarding the raw JSON payload. The actual processing of the event happens in the downstream service, which may or may not need to interact with the GitHub API. This is a common pattern in microservice architectures, where different services handle different aspects of the application logic.

This use case highlights the core problem: the framework's automatic installation client creation is unnecessary in the gateway service. Since the gateway only forwards the raw payload, it doesn't need to interact with the GitHub API directly. The framework's attempt to create a client is not only redundant but also causes errors when handling installation.deleted events. This is a clear example of a situation where the framework's default behavior doesn't align with the application's needs.

In essence, the gateway service is like a mail carrier: it receives the letter (webhook event) and delivers it to the recipient (downstream service) without needing to know the contents of the letter or the sender's personal information (installation details). The framework, however, is insisting on opening the letter and reading the sender's address before delivering it, even when it's not necessary and can lead to errors if the address is no longer valid.

Attempts to Solve the Issue

Several approaches were attempted to address this issue, but unfortunately, none provided a complete solution.

Attempt 1: Using @Installation.Deleted Annotation

The first attempt involved using the standard @Installation.Deleted annotation, which is the typical way to handle installation deletion events in quarkus-github-app:

void onInstallationDeleted(@Installation.Deleted GHEventPayload.Installation installationPayload) {
  // Forward to downstream service
  targetService.onInstallationDeleted(installationPayload);
}

The expectation was that this would allow the handler to process the installation.deleted event. However, this approach failed. The framework still attempted to create the installation client before the handler was called, resulting in the same 404 error.

Result: ❌ The framework attempts to create the installation client before the handler is called, resulting in the 404 error.

This attempt highlighted that the issue wasn't specific to the event handling logic itself, but rather the framework's initialization process. Even when using the standard annotation, the framework's eagerness to create the client remained a problem.

Attempt 2: Using @RawEvent Annotation

Next, the @RawEvent annotation was tried. According to the documentation, @RawEvent is designed for cases where you don't need the installation client. This seemed like a promising solution, as it should bypass the client creation process:

void onInstallationDeleted(@RawEvent(event = "installation", action = "deleted") GitHubEvent gitHubEvent) throws Exception {
  GHEventPayload.Installation payload = objectMapper.readValue(
      gitHubEvent.getPayload(), GHEventPayload.Installation.class);
  // Forward to downstream service
  targetService.onInstallationDeleted(payload);
}

The idea was to receive the raw event payload and manually deserialize it, avoiding the framework's automatic client creation. However, this approach also failed. The framework still threw the same error, indicating that it was still attempting to create the installation client even for @RawEvent handlers.

Result: ❌ Still throws the same error. The framework appears to attempt installation client creation even for @RawEvent handlers.

This result was particularly surprising and frustrating. It suggested a discrepancy between the documentation's promise and the framework's actual behavior. The @RawEvent annotation, which should have bypassed client creation, didn't seem to have the intended effect.

Attempt 3: Configuration Property

Finally, a configuration property was tried to disable the installation token validity check:

quarkus.github-app.check-installation-token-validity=false

The hope was that disabling the token validity check would prevent the framework from attempting to fetch the installation details. However, this approach also failed.

Result: ❌ Still throws the same error. This property only affects token validation, not the initial installation client creation.

This attempt clarified that the issue wasn't related to token validation but rather to the initial client creation process itself. The configuration property, while useful in other scenarios, didn't address the core problem.

These attempts highlight the persistence of the issue and the need for a more fundamental solution within the framework itself.

Expected vs. Actual Behavior

To summarize the problem, let's clearly define the expected and actual behaviors of the quarkus-github-app framework in this scenario.

Expected Behavior

For webhook proxy/forwarder use cases, the framework should behave in one of the following ways:

  1. Option A (Preferred): When using @RawEvent, skip the installation client creation entirely. This aligns with the documentation's suggestion that @RawEvent is intended for cases where the client is not needed.
  2. Option B: Provide a configuration option to disable automatic installation client creation for specific events. This would allow developers to opt-out of the default behavior for events like installation.deleted.
  3. Option C: Handle GHFileNotFoundException gracefully for deletion events. Since this exception is expected when an installation is deleted, the framework should suppress the error or log it at a lower level.

These options would provide developers with the flexibility to handle installation.deleted events without unnecessary error spam.

Actual Behavior

Currently, the framework always attempts to create an installation client by fetching installation details from the GitHub API, regardless of:

  • Whether @RawEvent is used.
  • Whether the installation still exists.
  • Whether the handler actually needs the client.

This causes error spam in logs for a completely normal and expected event (installation deletion). The framework's behavior is inflexible and doesn't account for use cases where the installation client is not required.

This discrepancy between expected and actual behavior highlights a key area for improvement in the quarkus-github-app framework. Addressing this issue would significantly improve the developer experience and reduce unnecessary noise in application logs.

Configuration and Environment

To provide a complete picture of the context, here's the configuration and environment used during the troubleshooting process.

Configuration

The following properties were used in the application.properties file:

quarkus.github-app.webhook-url-path=/v1/github-app/webhook
quarkus.github-app.app-id=${GITHUB_APP_APP_ID}
quarkus.github-app.webhook-secret=${GITHUB_APP_WEBHOOK_SECRET}
quarkus.github-app.private-key=${GITHUB_APP_PRIVATE_KEY}

These properties configure the basic settings for the GitHub App, such as the webhook URL path, application ID, webhook secret, and private key. These are standard settings for a quarkus-github-app application.

Environment

The application was deployed in the following environment:

  • Quarkus Version: 3.28.3
  • quarkus-github-app Version: 2.12.1
  • Deployment: Kubernetes cluster
  • Java Version: 21

This information is important for understanding the specific versions of the framework and related technologies used during the investigation. It also provides context for any potential compatibility issues or known bugs in those versions.

The Need for a Workaround (and the Lack Thereof)

Currently, there is no working workaround for this issue. The error spam occurs every time a user uninstalls the GitHub App, which is a normal part of the application lifecycle. This is a significant problem because it clutters the logs with unnecessary errors, making it harder to identify genuine issues.

The absence of a workaround underscores the need for a fix within the framework itself. Developers are left with no way to prevent these errors from occurring, which can be frustrating and time-consuming.

The ideal workaround would allow developers to either skip the installation client creation for installation.deleted events or handle the resulting GHFileNotFoundException gracefully. However, as demonstrated by the attempts described earlier, none of the existing mechanisms in quarkus-github-app provide this functionality.

This lack of a workaround further emphasizes the importance of the suggested solutions outlined below. A fix within the framework is the only way to truly address this issue and provide a better developer experience.

Suggested Solutions

To address this issue, here are some suggested solutions that could be implemented in the quarkus-github-app framework:

  1. Preferred: Make @RawEvent truly bypass installation client creation as the documentation implies. This would align the framework's behavior with the documentation and provide a straightforward way to handle events without the need for an installation client. This is the most elegant solution, as it leverages an existing mechanism in the framework.
  2. Add a configuration property like quarkus.github-app.skip-installation-client-for-events=deleted,suspended to explicitly skip client creation for specific events. This would provide developers with fine-grained control over client creation and allow them to optimize the framework's behavior for their specific use cases. This solution offers flexibility and customization.
  3. Add intelligent error handling that recognizes deletion events and suppresses the expected 404 error. This would prevent the error spam in logs without requiring any code changes in the application. This solution is pragmatic and focuses on reducing noise.
  4. Provide a way to inject GitHubEvent with the raw payload without triggering installation client creation. This would offer an alternative to @RawEvent that is specifically designed for scenarios where the client is not needed. This solution provides an alternative approach that is more explicit and less reliant on implicit behavior.

These solutions offer a range of options for addressing the issue, from leveraging existing mechanisms to introducing new features. The preferred solution is to make @RawEvent work as intended, as this aligns with the documentation and provides a simple and intuitive way to handle events without client creation.

Related Documentation

The following documentation provides relevant information about the quarkus-github-app framework and webhook events:

The documentation states: "Webhook requests don’t provide an installation id so we can’t initialize an installation GitHub client nor a GraphQL client." and suggests using @RawEvent for this purpose. This statement is particularly relevant, as it highlights the intended use case for @RawEvent and the discrepancy between the documentation and the framework's actual behavior.

This documentation reinforces the expectation that @RawEvent should bypass installation client creation. The fact that it doesn't in the case of installation.deleted events is a bug that needs to be addressed.

Conclusion

The issue of errors caused by installation.deleted events in quarkus-github-app is a significant problem, particularly for applications that act as webhook proxies or forwarders. The framework's automatic attempt to create an installation client, even after the installation has been deleted, leads to unnecessary error spam and obscures genuine issues in the logs.

The attempts to solve the problem using existing mechanisms like @Installation.Deleted, @RawEvent, and configuration properties have been unsuccessful. This highlights the need for a fix within the framework itself.

The suggested solutions, particularly making @RawEvent work as intended, offer a path forward. Addressing this issue will significantly improve the developer experience and make quarkus-github-app an even more powerful tool for building GitHub Apps.

Hopefully, this article has shed light on the issue and provided a clear understanding of the problem, its impact, and potential solutions. Let's hope the Quarkus team addresses this soon!