starred/dehydrated
1
0
Fork 0
mirror of https://github.com/dehydrated-io/dehydrated.git synced 2026-01-13 15:13:33 +01:00
Code Issues 79 Packages Projects Releases 10 Wiki Activity

dehydrated does not work with DigiCert ACMEv2 API - token value confusion between challenge types #479

New Issue
Closed
opened 2025-12-29 01:25:57 +01:00 by adam · 5 comments
adam commented 2025-12-29 01:25:57 +01:00
Owner
Copy Link

Originally created by @colin-stubbs on GitHub (May 28, 2020).

The current code within sign_csr() that tries to determine the token to use with the selected challenge type seems to fail to pick the correct token.

If dehydrated is configured to perform dns-01 validation, but the ACME API returns all three challenge types (dns-01, tls-alpn-01, http-01) BUT with different token values for each, the token value from the last entry in the list is used.

e.g. the token value specific to the http-01 challenge is used by dehydrated as part of dns-01 (hook deploys DNS record), and the DigiCert ACMEv2 API when asked to verify that token (it's the http-01 NOT the dns-01 value) it goes ahead and correctly performs a check for http-01 token via HTTP instead of dns-01 via DNS.

The issue is in sign_csr() when it iterates over the list of challenges return in the JSON payload from this response,

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 27 May 2020 23:07:02 GMT
Content-Type: application/json
Content-Length: 892
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Cache-Control: public, max-age=0, no-cache
Link: <https://acme.digicert.com/v2/acme/account/SECRET_UNIQUE_ACCOUNT_VALUE>;rel="up"
Location: https://acme.digicert.com/v2/acme/authz/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE
Replay-Nonce: NONCE_VALUE
X-Correlation-Id: F19D8DE2-CC90-52BC-0422-3BD27D01069F
X-Hostname: enrollme05.slc.digicert.com
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000

{
	"challenges": [
		{
			"status": "pending",
			"token": "kea-gtCl0nnJqDHu05N3JD3s-zwQNiXU",
			"type": "dns-01",
			"url": "https://acme.digicert.com/v2/acme/challenge/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE/q2quuRXWzp8LhnLj"
		},
		{
			"status": "pending",
			"token": "4tUeO_9MxeWFl8pGflNQMxnBQV1xpT-2",
			"type": "tls-alpn-01",
			"url": "https://acme.digicert.com/v2/acme/challenge/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE/DilxB7yuGKKjXii4"
		},
		{
			"status": "pending",
			"token": "4ZDSajMX7nzWR2diVSs6alSCHQVlxuKJ",
			"type": "http-01",
			"url": "https://acme.digicert.com/v2/acme/challenge/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE/iH1RuwXGm4tHWU62"
		}
	],
	"expires": "2020-05-29T17:07:00-06:00",
	"identifier": {
		"type": "dns",
		"value": "sub.domain.tld"
	},
	"status": "pending",
	"wildcard": false
}
Originally created by @colin-stubbs on GitHub (May 28, 2020). The current code within sign_csr() that tries to determine the token to use with the selected challenge type seems to fail to pick the correct token. If dehydrated is configured to perform dns-01 validation, but the ACME API returns all three challenge types (dns-01, tls-alpn-01, http-01) BUT with different token values for each, the token value from the last entry in the list is used. e.g. the token value specific to the http-01 challenge is used by dehydrated as part of dns-01 (hook deploys DNS record), and the DigiCert ACMEv2 API when asked to verify that token (it's the http-01 NOT the dns-01 value) it goes ahead and correctly performs a check for http-01 token via HTTP instead of dns-01 via DNS. The issue is in sign_csr() when it iterates over the list of challenges return in the JSON payload from this response, ``` HTTP/1.1 200 OK Server: nginx Date: Wed, 27 May 2020 23:07:02 GMT Content-Type: application/json Content-Length: 892 Connection: keep-alive Cache-Control: public, max-age=0, no-cache Cache-Control: public, max-age=0, no-cache Link: <https://acme.digicert.com/v2/acme/account/SECRET_UNIQUE_ACCOUNT_VALUE>;rel="up" Location: https://acme.digicert.com/v2/acme/authz/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE Replay-Nonce: NONCE_VALUE X-Correlation-Id: F19D8DE2-CC90-52BC-0422-3BD27D01069F X-Hostname: enrollme05.slc.digicert.com X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff Strict-Transport-Security: max-age=31536000 { "challenges": [ { "status": "pending", "token": "kea-gtCl0nnJqDHu05N3JD3s-zwQNiXU", "type": "dns-01", "url": "https://acme.digicert.com/v2/acme/challenge/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE/q2quuRXWzp8LhnLj" }, { "status": "pending", "token": "4tUeO_9MxeWFl8pGflNQMxnBQV1xpT-2", "type": "tls-alpn-01", "url": "https://acme.digicert.com/v2/acme/challenge/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE/DilxB7yuGKKjXii4" }, { "status": "pending", "token": "4ZDSajMX7nzWR2diVSs6alSCHQVlxuKJ", "type": "http-01", "url": "https://acme.digicert.com/v2/acme/challenge/SECRET_UNIQUE_ACCOUNT_VALUE/UNIQUE_REQUEST_VALUE/iH1RuwXGm4tHWU62" } ], "expires": "2020-05-29T17:07:00-06:00", "identifier": { "type": "dns", "value": "sub.domain.tld" }, "status": "pending", "wildcard": false } ```
adam closed this issue 2025-12-29 01:25:57 +01:00
adam commented 2025-12-29 01:25:58 +01:00
Author
Owner
Copy Link

@colin-stubbs commented on GitHub (May 28, 2020):

Example from Let's Encrypt, where token values are all the same... so the assumption that the last token value can be used still works.

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 27 May 2020 23:27:55 GMT
Content-Type: application/json
Content-Length: 818
Connection: keep-alive
Boulder-Requester: 13894319
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: NONCE_VALUE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "test1234.domain.tld"
  },
  "status": "pending",
  "expires": "2020-06-03T23:27:54Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/ACCOUNT_ID/3YO5Sg",
      "token": "B1Nsb3u9zGdG5acKyZwcce7c8ID53jT-pPHMitntIkM"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/59753864/f-0OxA",
      "token": "B1Nsb3u9zGdG5acKyZwcce7c8ID53jT-pPHMitntIkM"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/59753864/OyavkQ",
      "token": "B1Nsb3u9zGdG5acKyZwcce7c8ID53jT-pPHMitntIkM"
    }
  ]
}
@colin-stubbs commented on GitHub (May 28, 2020): Example from Let's Encrypt, where token values are all the same... so the assumption that the last token value can be used still works. ``` HTTP/1.1 200 OK Server: nginx Date: Wed, 27 May 2020 23:27:55 GMT Content-Type: application/json Content-Length: 818 Connection: keep-alive Boulder-Requester: 13894319 Cache-Control: public, max-age=0, no-cache Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index" Replay-Nonce: NONCE_VALUE X-Frame-Options: DENY Strict-Transport-Security: max-age=604800 { "identifier": { "type": "dns", "value": "test1234.domain.tld" }, "status": "pending", "expires": "2020-06-03T23:27:54Z", "challenges": [ { "type": "http-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/ACCOUNT_ID/3YO5Sg", "token": "B1Nsb3u9zGdG5acKyZwcce7c8ID53jT-pPHMitntIkM" }, { "type": "dns-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/59753864/f-0OxA", "token": "B1Nsb3u9zGdG5acKyZwcce7c8ID53jT-pPHMitntIkM" }, { "type": "tls-alpn-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/59753864/OyavkQ", "token": "B1Nsb3u9zGdG5acKyZwcce7c8ID53jT-pPHMitntIkM" } ] } ```
adam commented 2025-12-29 01:25:58 +01:00
Author
Owner
Copy Link

@colin-stubbs commented on GitHub (May 28, 2020):

Summary of issues found in DigiCert ACME API endpoint:

  1. TAB characters included with spaces in body of response, e.g. inside JSON formatted content, have to update clean_json() function to replaces any instance of space and/or tab with single space
  2. challenges was being extracted from JSON response using weird sed regex match when it should be using the standard get_json_array_value function
  3. Even though challenge has been completed successfully, DigiCert ACMEv2 API may respond with "processing" instead of "pending", where Let's Encrypt will always respond with "pending"... "processing" is a valid status according to RFC8555 and seems to indicate the same thing... that the ACME client should back off and try again shortly.
@colin-stubbs commented on GitHub (May 28, 2020): Summary of issues found in DigiCert ACME API endpoint: 1. TAB characters included with spaces in body of response, e.g. inside JSON formatted content, have to update clean_json() function to replaces any instance of space and/or tab with single space 2. challenges was being extracted from JSON response using weird sed regex match when it should be using the standard get_json_array_value function 2. Even though challenge has been completed successfully, DigiCert ACMEv2 API may respond with "processing" instead of "pending", where Let's Encrypt will always respond with "pending"... "processing" is a valid status according to RFC8555 and seems to indicate the same thing... that the ACME client should back off and try again shortly.
adam commented 2025-12-29 01:25:59 +01:00
Author
Owner
Copy Link

@colin-stubbs commented on GitHub (May 28, 2020):

Tested against both Let's Encrypt and DigiCert... DigiCert failing for somewhat expected reason now, e.g. no funds to use for cert.

$ diff -u /usr/bin/dehydrated.latest /usr/bin/dehydrated
--- /usr/bin/dehydrated.latest	2020-05-28 13:34:00.485242030 +1000
+++ /usr/bin/dehydrated	2020-05-28 14:00:18.774128866 +1000
@@ -453,7 +453,7 @@
 
 # Remove newlines and whitespace from json
 clean_json() {
-  tr -d '\r\n' | _sed -e 's/ +/ /g' -e 's/\{ /{/g' -e 's/ \}/}/g' -e 's/\[ /[/g' -e 's/ \]/]/g'
+  tr -d '\r\n' | _sed -e 's/[ \t]+/ /g' -e 's/\{ /{/g' -e 's/ \}/}/g' -e 's/\[ /[/g' -e 's/ \]/]/g'
 }
 
 # Encode data as url-safe formatted base64
@@ -744,7 +744,7 @@
     fi
 
     # Find challenge in authorization
-    challenges="$(echo "${response}" | _sed 's/.*"challenges": \[(\{.*\})\].*/\1/')"
+    challenges="$(echo "${response}" | get_json_array_value challenges)"
     challenge="$(<<<"${challenges}" _sed -e 's/^[^\[]+\[(.+)\]$/\1/' -e 's/\}(, (\{)|(\]))/}\'$'\n''\2/g' | grep \""${CHALLENGETYPE}"\" || true)"
     if [ -z "${challenge}" ]; then
       allowed_validations="$(grep -Eo '"type": "[^"]+"' <<< "${challenges}" | grep -Eo ' "[^"]+"' | _sed -e 's/"//g' -e 's/^ //g')"
@@ -817,7 +817,7 @@
 
     reqstatus="$(printf '%s\n' "${result}" | get_json_string_value status)"
 
-    while [[ "${reqstatus}" = "pending" ]]; do
+    while [[ "${reqstatus}" = "pending" ]] || [[ "${reqstatus}" = "processing" ]]; do
       sleep 1
       if [[ "${API}" -eq 2 ]]; then
         result="$(signed_request "${challenge_uris[${idx}]}" "")"
$

$ rm -rf /etc/dehydrated/accounts/* /etc/dehydrated/certs/* /etc/dehydrated/chains/* /etc/dehydrated/archive/*

$ dehydrated --cron --accept-terms --config /etc/dehydrated/config_letsencrypt --hook /etc/dehydrated/hooks/dehydrated-lexicon 
# INFO: Using main config file /etc/dehydrated/config_letsencrypt
# INFO: Using additional config file /etc/dehydrated/conf.d/local.sh
+ Generating account key...
+ Registering account key with ACME server...
+ Fetching account URL...
Processing test1234.domain.tld
 + Creating new directory /etc/dehydrated/certs/test1234.domain.tld ...
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for test1234.domain.tld
 + 1 pending challenge(s)
 + Deploying challenge tokens...
deploy_challenge called: test1234.domain.tld, DcGPJRRvAZa3Z_3lX2sv-Clog4RyNLwU2ru_zc2SWUQ, YjbusyGcy3ofi2Ye-kEmDrkLSlWRHIsGFoTpn8PU6YA
 + Responding to challenge for test1234.domain.tld authorization...
 + Challenge is valid!
 + Cleaning challenge tokens...
clean_challenge called: test1234.domain.tld, DcGPJRRvAZa3Z_3lX2sv-Clog4RyNLwU2ru_zc2SWUQ, YjbusyGcy3ofi2Ye-kEmDrkLSlWRHIsGFoTpn8PU6YA
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
 + Done!
$


$ dehydrated --cron --accept-terms --config /etc/dehydrated/config_digicert --hook /etc/dehydrated/hooks/dehydrated-lexicon 
# INFO: Using main config file /etc/dehydrated/config_digicert
# INFO: Using additional config file /etc/dehydrated/conf.d/local.sh
+ Generating account key...
+ Registering account key with ACME server...
+ Fetching account URL...
Processing test1234.domain.tld
 + Creating new directory /etc/dehydrated/certs/test1234.domain.tld ...
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for test1234.domain.tld
 + 1 pending challenge(s)
 + Deploying challenge tokens...
deploy_challenge called: test1234.domain.tld, 49G7ClNPmL0fxEeg-z_pocUtuoU8T_kl, _aeTvrwANeM-0J2i48LhRjqZMfT4XY75_mL21j41Yf0
 + Responding to challenge for test1234.domain.tld authorization...
 + Challenge is valid!
 + Cleaning challenge tokens...
clean_challenge called: test1234.domain.tld, 49G7ClNPmL0fxEeg-z_pocUtuoU8T_kl, _aeTvrwANeM-0J2i48LhRjqZMfT4XY75_mL21j41Yf0
 + Requesting certificate...
  + ERROR: An error occurred while sending post-request to https://acme.digicert.com/v2/acme/finalize/UNIQUE_VALUE/UNIQUE_VALUE (Status 500)

Details:
HTTP/1.1 100 Continue

HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Thu, 28 May 2020 04:07:12 GMT
Content-Type: application/problem+json
Content-Length: 274
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Replay-Nonce: NONCE_WAS_HERE
X-Correlation-Id: 5E3CDD4E-0370-10BC-4D80-2371F8A3D370
X-Hostname: enrollme03.slc.digicert.com
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000

{"detail":"failed to request certificate, {\"errors\":[{\"code\":\"insufficient_funds\",\"message\":\"There are insufficient funds available in your account to complete this request.  Please deposit additional funds and try again.\"}]}","status":500,"type":"serverInternal"}

$
@colin-stubbs commented on GitHub (May 28, 2020): Tested against both Let's Encrypt and DigiCert... DigiCert failing for somewhat expected reason now, e.g. no funds to use for cert. ``` $ diff -u /usr/bin/dehydrated.latest /usr/bin/dehydrated --- /usr/bin/dehydrated.latest 2020-05-28 13:34:00.485242030 +1000 +++ /usr/bin/dehydrated 2020-05-28 14:00:18.774128866 +1000 @@ -453,7 +453,7 @@ # Remove newlines and whitespace from json clean_json() { - tr -d '\r\n' | _sed -e 's/ +/ /g' -e 's/\{ /{/g' -e 's/ \}/}/g' -e 's/\[ /[/g' -e 's/ \]/]/g' + tr -d '\r\n' | _sed -e 's/[ \t]+/ /g' -e 's/\{ /{/g' -e 's/ \}/}/g' -e 's/\[ /[/g' -e 's/ \]/]/g' } # Encode data as url-safe formatted base64 @@ -744,7 +744,7 @@ fi # Find challenge in authorization - challenges="$(echo "${response}" | _sed 's/.*"challenges": \[(\{.*\})\].*/\1/')" + challenges="$(echo "${response}" | get_json_array_value challenges)" challenge="$(<<<"${challenges}" _sed -e 's/^[^\[]+\[(.+)\]$/\1/' -e 's/\}(, (\{)|(\]))/}\'$'\n''\2/g' | grep \""${CHALLENGETYPE}"\" || true)" if [ -z "${challenge}" ]; then allowed_validations="$(grep -Eo '"type": "[^"]+"' <<< "${challenges}" | grep -Eo ' "[^"]+"' | _sed -e 's/"//g' -e 's/^ //g')" @@ -817,7 +817,7 @@ reqstatus="$(printf '%s\n' "${result}" | get_json_string_value status)" - while [[ "${reqstatus}" = "pending" ]]; do + while [[ "${reqstatus}" = "pending" ]] || [[ "${reqstatus}" = "processing" ]]; do sleep 1 if [[ "${API}" -eq 2 ]]; then result="$(signed_request "${challenge_uris[${idx}]}" "")" $ $ rm -rf /etc/dehydrated/accounts/* /etc/dehydrated/certs/* /etc/dehydrated/chains/* /etc/dehydrated/archive/* $ dehydrated --cron --accept-terms --config /etc/dehydrated/config_letsencrypt --hook /etc/dehydrated/hooks/dehydrated-lexicon # INFO: Using main config file /etc/dehydrated/config_letsencrypt # INFO: Using additional config file /etc/dehydrated/conf.d/local.sh + Generating account key... + Registering account key with ACME server... + Fetching account URL... Processing test1234.domain.tld + Creating new directory /etc/dehydrated/certs/test1234.domain.tld ... + Signing domains... + Generating private key... + Generating signing request... + Requesting new certificate order from CA... + Received 1 authorizations URLs from the CA + Handling authorization for test1234.domain.tld + 1 pending challenge(s) + Deploying challenge tokens... deploy_challenge called: test1234.domain.tld, DcGPJRRvAZa3Z_3lX2sv-Clog4RyNLwU2ru_zc2SWUQ, YjbusyGcy3ofi2Ye-kEmDrkLSlWRHIsGFoTpn8PU6YA + Responding to challenge for test1234.domain.tld authorization... + Challenge is valid! + Cleaning challenge tokens... clean_challenge called: test1234.domain.tld, DcGPJRRvAZa3Z_3lX2sv-Clog4RyNLwU2ru_zc2SWUQ, YjbusyGcy3ofi2Ye-kEmDrkLSlWRHIsGFoTpn8PU6YA + Requesting certificate... + Checking certificate... + Done! + Creating fullchain.pem... + Done! $ $ dehydrated --cron --accept-terms --config /etc/dehydrated/config_digicert --hook /etc/dehydrated/hooks/dehydrated-lexicon # INFO: Using main config file /etc/dehydrated/config_digicert # INFO: Using additional config file /etc/dehydrated/conf.d/local.sh + Generating account key... + Registering account key with ACME server... + Fetching account URL... Processing test1234.domain.tld + Creating new directory /etc/dehydrated/certs/test1234.domain.tld ... + Signing domains... + Generating private key... + Generating signing request... + Requesting new certificate order from CA... + Received 1 authorizations URLs from the CA + Handling authorization for test1234.domain.tld + 1 pending challenge(s) + Deploying challenge tokens... deploy_challenge called: test1234.domain.tld, 49G7ClNPmL0fxEeg-z_pocUtuoU8T_kl, _aeTvrwANeM-0J2i48LhRjqZMfT4XY75_mL21j41Yf0 + Responding to challenge for test1234.domain.tld authorization... + Challenge is valid! + Cleaning challenge tokens... clean_challenge called: test1234.domain.tld, 49G7ClNPmL0fxEeg-z_pocUtuoU8T_kl, _aeTvrwANeM-0J2i48LhRjqZMfT4XY75_mL21j41Yf0 + Requesting certificate... + ERROR: An error occurred while sending post-request to https://acme.digicert.com/v2/acme/finalize/UNIQUE_VALUE/UNIQUE_VALUE (Status 500) Details: HTTP/1.1 100 Continue HTTP/1.1 500 Internal Server Error Server: nginx Date: Thu, 28 May 2020 04:07:12 GMT Content-Type: application/problem+json Content-Length: 274 Connection: keep-alive Cache-Control: public, max-age=0, no-cache Replay-Nonce: NONCE_WAS_HERE X-Correlation-Id: 5E3CDD4E-0370-10BC-4D80-2371F8A3D370 X-Hostname: enrollme03.slc.digicert.com X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff Strict-Transport-Security: max-age=31536000 {"detail":"failed to request certificate, {\"errors\":[{\"code\":\"insufficient_funds\",\"message\":\"There are insufficient funds available in your account to complete this request. Please deposit additional funds and try again.\"}]}","status":500,"type":"serverInternal"} $ ```
adam commented 2025-12-29 01:25:59 +01:00
Author
Owner
Copy Link

@danimo commented on GitHub (Sep 2, 2020):

@colin-stubbs Can you check my pull request? It's essentially a rebase of your patch to master (which uses json.sh).

@danimo commented on GitHub (Sep 2, 2020): @colin-stubbs Can you check my pull request? It's essentially a rebase of your patch to master (which uses json.sh).
adam commented 2025-12-29 01:25:59 +01:00
Author
Owner
Copy Link

@lukas2511 commented on GitHub (Dec 10, 2020):

I think this should be fixed by now, otherwise please comment or feel free to open a new issue.

@lukas2511 commented on GitHub (Dec 10, 2020): I think this should be fixed by now, otherwise please comment or feel free to open a new issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Complicated
bug
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
 suggestions wanted
testing
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adam
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/dehydrated#479
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?