"Smudge error" in GIT LFS with bearer token on Bitbucket Data Center

Summary

When performing a git operation in Bitbucket Data Center using NGINX, it fails with a Smudge error.

Environment

Bitbucket 8.x and above running behind NGINX acting as a reverse proxy.

Diagnosis

Error is related to LFS objects and personal access token

This error happens when both:

1. There are LFS objects involved (for example, they are stored in the repository and are being cloned or pulled, or if LFS objects are to be pushed).

2. You are using a personal access token (or HTTP access token) with git (by specifying the -c http.extraHeader='Authorization: Bearer <token>' command line option in git).

It does not happen if you provide your username and your token as the password to git.

Steps to reproduce

1. Attempt to clone the repository using the command:

   git clone https://xxx.git -c http.extraheader="Authorization: Bearer token xxx"

2. Observe the error message received during the LFS object fetch.

   > GET /rest/git-lfs/storage/LFS/newlfs/9d8453505bdc6f269678e16b3e56c2a2948a41f2c792617cc9611ed363c95b63 HTTP/1.1 > Host: instenv-509821-s0hm.instenv.internal.atlassian.com > Authorization: JWT xxxx > Authorization: Bearer BBDC-xxxx > User-Agent: git-lfs/3.6.1 (GitHub; linux amd64; go 1.23.3; git ea47a34b) > 08:44:27.529556 trace git-lfs: HTTP: 400 < HTTP/2.0 400 Bad Request < Content-Length: 122

Cause

Nginx drops JWT auth, causing LFS smudge errors with access tokens

When using an HTTP Access Token with git on repositories that have LFS enabled, the LFS download fails if there is a reverse proxy in front of the Bitbucket server.

This only happens if the -c http.extraHeader command line option is used in git to specify the HTTP Access Token for authentication via a Bearer token.

If nginx is used, it fails with a similar "Smudge error" because, unlike Apache, nginx does not merge the headers but discards one of the two headers and only forwards the HTTP Access token instead of the JWT to Bitbucket, so authentication fails for the LFS operation.

Solution

One possible workaround to solve the JWT header being discarded is to pass the bearer token via a "custom header" and have nginx reuse that into an auth header for non-LFS endpoints. This means the Authorization header won't be duplicated and nginx won't discard it.

Nginx reverse proxy server config changes

1. Test sending the bearer token using a custom header in the git clone request as below:

   GIT_TRACE=1 GIT_CURL_VERBOSE=1 git -c "http.extraheader=X-Auth-Token: BBDC-xxxx" clone http://test-atl.com:8990/bitbucket/scm/new/newlfs.git

2. Next, configure nginx to transform that into the Auth bearer header if it's a non-LFS endpoint. Here, the two blocks below map the incoming `X-Auth-Token` header to a variable `$auth_token_prefixed`. For requests matching `/rest/git-lfs/storage/`, use the incoming `Authorization` header and for all other requests, use the `$auth_token_prefixed` value.

   # Map the incoming `X-Auth-Token` header to a variable `$auth_token_prefixed` # If `X-Auth-Token` is empty, set `$auth_token_prefixed` to an empty string. # Otherwise, prefix the token with "Bearer ". map $http_x_auth_token $auth_token_prefixed { "" ""; default "Bearer $http_x_auth_token"; } # Map the `$auth_header_mapped` variable based on the URI of the request. # For requests matching `/rest/git-lfs/storage/`, use the incoming `Authorization` header. # For all other requests, use the `$auth_token_prefixed` value. map $uri $auth_header_mapped { ~^/rest/git-lfs/storage/ $http_authorization; default $auth_token_prefixed; } # Main location block location / { # Set the `Authorization` header for the upstream server using the `$auth_header_mapped` value proxy_set_header Authorization $auth_header_mapped; # Other existing config goes here... }

3. After this, the LFS objects should be downloaded successfully using the NGINX proxy server over HTTP.

4. The map directive may not be natively supported in the Ingress resource for Bitbucket running on Kubernetes. But this will be possible using annotations or by extending the NGINX configuration via a ConfigMap.

Nginx Ingress Controller Changes

1. First, enable the "allow-snippet-annotations". The default ConfigMap is usually named ingress-nginx-controller (or similar) and located in the ingress-nginx namespace. Edit the ingress config map to allow for manual override of helm chart values:

   kubectl edit configmap ingress-nginx-controller -n ingress-nginx

2. Add the following keys under data for the annotations:

   data: allow-snippet-annotations: "true" annotations-risk-level: Critical

3. Next, edit the ingress object (not the ingressclass object):

   ubuntu@ip-10-229-128-26:~$ kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE bitbucket-ingress nginx * 10.97.211.88 80 34m kubectl edit ingress bitbucket-ingress

4. Add the below annotations to the list of other annotations that may be there:

   annotations: nginx.ingress.kubernetes.io/server-snippet: | map $http_x_auth_token $auth_token_prefixed { "" ""; default "Bearer $http_x_auth_token"; } map $uri $auth_header_mapped { ~^/rest/git-lfs/storage/ $http_authorization; default $auth_token_prefixed; } nginx.ingress.kubernetes.io/configuration-snippet: | proxy_set_header Authorization $auth_header_mapped;

5. Below is the test result after applying the above changes. This time, when you invoke the git clone, you need to pass the bearer token in a custom header as " http.extraheader=X-Auth-Token" as below:

   This custom header is only part of this workaround and not a supported header or part of the product. The actual name of the header “X-Auth-Token” can be customised as per your choice, as it is entirely custom.

   GIT_TRACE=1 GIT_CURL_VERBOSE=1 git -c "http.extraheader=X-Auth-Token: BBDC-NzU2NjE5MDcyNDcwOv0iWEzCpv9YX5CTJIYn+3tEAA+j" clone https://k8s-101987.prod.atl-cd.net/scm/at/jira.git

Response