HARICA ACME DNS-01 order gets stuck with Post "": unsupported protocol scheme ""

baltersar

Hello,

I am currently testing Cert Warden with the HARICA Stage ACME service and I am experiencing a reproducible problem when issuing certificates using DNS-01.

Environment

Cert Warden: 0.29.x
ACME CA: HARICA Stage
Challenge type: DNS-01
DNS provider: IONOS
Domain used for testing: apache.lehrer-bremen.de

The DNS provisioning itself works correctly.

A new order initially proceeds normally:

orders: fulfilling worker 1: ordering order id 6
(certificate name: apache-test, subject: apache.lehrer-bremen.de)

challenges: success on any needed provision action
for domain apache.lehrer-bremen.de

challenges: waiting to validate apache.lehrer-bremen.de

After the propagation delay, the following error occurs:

failed to save response replay nonce
(nonce manager: cannot save empty nonce)

challenges: status: 0;
type: urn:ietf:params:acme:error:malformed;
detail: challenge or authorization must be in pending state

All subsequent retries fail with:

challenges: Post "": unsupported protocol scheme ""

Cert Warden then remains in this retry loop and the order does not complete.

I tested different DNS propagation delays, including 300 seconds and 30 seconds, and the behavior is identical.

Using Cert Warden's Debug POST-as-GET functionality, I queried the authorization while Cert Warden was stuck.

HARICA reports:

{
"identifier": {
"type": "dns",
"value": "apache.lehrer-bremen.de"
},
"status": "valid",
"challenges": [
{
"type": "http-01",
"status": "pending"
},
{
"type": "dns-01",
"status": "valid"
}
]
}

The POST-as-GET response from HARICA also contains a Replay-Nonce response header.

So at this point HARICA considers both the DNS-01 challenge and the authorization valid, while the Cert Warden challenge solver continues retrying with:

Post "": unsupported protocol scheme ""

There is another interesting behavior:

If I restart the Cert Warden container while the order is stuck, Cert Warden detects the incomplete order after startup:

orders: 1 incomplete orders added to order queue

It then retries the same existing order:

orders: fulfilling worker 0: ordering order id 4

and only a few seconds later successfully completes it:

orders: fulfilling worker 0:
order id 4 completed with status valid

orders: fulfilling worker 0: order 4 done

No new DNS challenge is required after the restart.

This behavior has been reproduced with multiple HARICA Stage orders.

It therefore appears that HARICA successfully validates the DNS-01 authorization, but the running Cert Warden challenge solver does not correctly handle the authorization changing to valid. After receiving challenge or authorization must be in pending state, the solver also appears to lose the URL used for subsequent requests, resulting in Post "".

Could this be a compatibility issue between HARICA's ACME implementation and Cert Warden's challenge/nonce handling, or an issue with handling an authorization that has already transitioned to valid?

I can provide additional debug logs and POST-as-GET responses if needed.

baltersar

Hi Greg,
just following up on this issue.

I was able to reproduce it multiple times with HARICA Stage. The important observation is that HARICA already reports the DNS-01 challenge and authorization as valid via Cert Warden's Debug POST-as-GET, while Cert Warden remains stuck in the Post "": unsupported protocol scheme "" retry loop.

Restarting Cert Warden causes the same incomplete order to complete successfully within a few seconds.

If useful, I can provide the complete debug logs and POST-as-GET response.

Thanks!

gregtwallace

Can you post the relevant json for your order, authorization objects?

In the json posted I don't see a "status" for the authorization itself (and I'm curious what the order and auth look like in full).

What's going on is its trying to POST to the challenge(s) but since the challenges don't have a url, that field is blank, hence the post to "".

I'd need to see the complete objects to get an idea of how to fix the logic to account for this. A full debug log of an attempted order would be best.

(I made a harica account but I’m not readily seeing an acme test environment I can test with.)

baltersar

Thanks. I enabled debug logging and reproduced the issue with a new order. I think the debug log now shows exactly where the empty URL appears.

The order initially contains a valid authorization URL, and the authorization initially contains valid URLs for both the http-01 and dns-01 challenges.

Cert Warden successfully provisions the DNS-01 challenge and, after the configured 300 second propagation wait, sends the expected POST with {} to the HARICA dns-01 challenge URL:

2026-08-27T10:15:16.113+0200
sending acme signed post ... to:
https://acme-stg-v02.harica.gr/.../chall/.../bf43...
unencoded payload: {}

A few seconds later Cert Warden performs another POST-as-GET to the same challenge URL:

2026-08-27T10:15:23.375+0200
sending acme signed post ... to:
https://acme-stg-v02.harica.gr/.../chall/.../bf43...
unencoded payload: ""

HARICA then responds:

type: urn:ietf:params:acme:error:malformed
detail: challenge or authorization must be in pending state

Immediately afterwards there is also:

failed to save response replay nonce
(nonce manager: cannot save empty nonce)

From the next retry onwards, Cert Warden's debug log shows that the target URL has become empty:

sending acme signed post ... to: ; unencoded payload: ""
challenges: Post "": unsupported protocol scheme ""

This continues until the solver times out after about 30 minutes.

I also queried the authorization manually using Cert Warden's Debug POST-as-GET afterwards. HARICA reports the authorization itself as:

"status": "valid"

and the challenges as:

{
"type": "http-01",
"status": "pending",
"url": "https://..."
},
{
"type": "dns-01",
"status": "valid",
"url": "https://...",
"validated": "2026-08-27T11:15:17.393+03:00"
}

So the DNS-01 validation itself appears to succeed at HARICA.

The order object queried afterwards is still:

{
"status": "pending",
"identifiers": [
{
"type": "dns",
"value": "apache.lehrer-bremen.de"
}
],
"authorizations": [
"https://acme-stg-v02.harica.gr/.../authz/..."
],
"finalize": "https://acme-stg-v02.harica.gr/.../finalize/..."
}

So it looks like the empty challenge URL does not originate from the initial authorization object. The URL is present initially and Cert Warden successfully uses it. It seems to become empty only after HARICA returns the challenge or authorization must be in pending state response.

I can provide the complete debug log for this order if useful.

baltersar

Hi Greg,

just following up on this issue again.

It's been about two weeks since I posted the additional debug information you requested. Were you able to take a look at it?

The issue is still reproducible with HARICA Stage. The DNS-01 authorization becomes valid on HARICA's side, but Cert Warden gets stuck after the challenge or authorization must be in pending state response and subsequently starts trying to POST to an empty URL.

Restarting Cert Warden still allows the same incomplete order to complete successfully without creating a new DNS challenge.

If it would help with troubleshooting, I'm happy to provide the complete debug log of an affected order.

Thanks!

gregtwallace

I believe the issue is two-fold:
1) The challenge URL gets overwritten upon error while trying to check status (leading to the unsupported protocol scheme error)
2) Contrary to every other ACME server I've dealt with, your provider may not support a direct PaG of a challenge URL (guessing based on the malformed error on subsequent attempt).

I believe I've fixed #1 in this branch: https://github.com/gregtwallace/certwarden-backend/tree/challenge-fix
If you're able to build the binary and confirm, that would be great (since I can't get a HARICA staging URL to test myself).

As for #2, can you do a PaG of the solved challenge URL? i.e., for

{
"type": "dns-01",
"status": "valid",
"url": "https://someurlhere.example.com/abc123",
"validated": "2026-08-27T11:15:17.393+03:00"
}

do a debug pag to https://someurlhere.example.com/abc123 and tell me the result.

baltersar

Hi Greg,

I tested the challenge-fix branch (commit b51265b) against the HARICA staging ACME endpoint.

The fix for the overwritten challenge URL works as expected. During the entire test, Cert Warden continued to use the original challenge URL. The previous error

Post "": unsupported protocol scheme ""

did not occur anymore.

However, the HARICA challenge POST-as-GET behavior is still reproducible. After the DNS-01 challenge had been submitted, Cert Warden repeatedly tried to query the challenge URL and HARICA responded with:

HTTP 400
urn:ietf:params:acme:error:malformed
challenge or authorization must be in pending state

Cert Warden also logged:

failed to save response replay nonce
(nonce manager: cannot save empty nonce)

Cert Warden kept retrying the same (correct) challenge URL until the first attempt eventually timed out with:

challenges: solving failed:
challenge failed to move to final state (timeout)

status: 0
type: urn:ietf:params:acme:error:malformed
detail: challenge or authorization must be in pending state

Interestingly, the order was picked up again later by Cert Warden's auto-ordering.

On this second attempt, Cert Warden queried the authorization URL. HARICA then reported:

authorization status: valid
dns-01 challenge status: valid

The order subsequently changed to ready, Cert Warden submitted the CSR to the finalize endpoint, HARICA changed the order to processing, and shortly afterwards the order became:

status: valid
certificate: https://.../cert/...

So the certificate was ultimately issued successfully.

In summary:

The challenge-fix successfully fixes the empty/overwritten challenge URL issue.
HARICA still returns 400 malformed when Cert Warden performs POST-as-GET directly against the already processed DNS-01 challenge URL.
The first Cert Warden run therefore times out.
HARICA nevertheless validates the DNS-01 authorization in the background.
When Cert Warden retries the incomplete order via auto-ordering, it sees the authorization as valid and successfully finalizes and retrieves the certificate.

I also tested a direct POST-as-GET against a previously solved HARICA DNS-01 challenge as requested. The authorization itself was valid, but the direct challenge request returned a HARICA malformed response rather than the challenge object.

So from my testing, the URL overwrite issue is fixed, while HARICA's behavior when querying the challenge resource appears to be the remaining interoperability issue.

If useful, I can provide the relevant debug log section from the complete order flow as well.

Thanks!

gregtwallace

Thanks for confirming. No further testing needed as this result aligns with my suspicions.

Basically Cert Warden follows the objects Order -> Auth -> Challenge and waits for the Challenge to validate. The protocol doesn't explicitly seem to require the direct Post-As-Get of a challenge to function, but every other server seems to implement this. Once CW gets to the Challenge level, it keeps trying to check it for it to become valid and then moves back up. Once it times out, the next attempt starts at the top again, so when it gets to Auth, it sees valid and doesn't continue down to the Challenge (hence why the 2nd time through finishes the job).

In the spec, there is a section:

   Usually, the validation process will take some time, so the client
   will need to poll the authorization resource to see when it is
   finalized.  For challenges where the client can tell when the server
   has validated the challenge (e.g., by seeing an HTTP or DNS request
   from the server), the client SHOULD NOT begin polling until it has
   seen the validation request from the server.

   To check on the status of an authorization, the client sends a POST-
   as-GET request to the authorization URL, and the server responds with
   the current authorization object.  In responding to poll requests
   while the validation is still in progress, the server MUST return a
   200 (OK) response and MAY include a Retry-After header field to
   suggest a polling interval to the client.

So while this detail isn't written in such a way to demand it be done this way, it is the suggested way, which is contrary to CW's current architecture (of polling the challenge instead of the authorization).

I will work to fix this soon-ish, but this particular issue will require a decent amount of rework to accomodate (as opposed to the simple change I already made). I'm not sure when I'll be able to get to this particular issue as my current available bandwidth for this project is already being consumed by some efforts to make things more stable and maintainable over time, but I'll see what I can do!

gregtwallace

I was actually able to make a relatively minor (but ugly) change to address the issue. I'll just work on a better refactor later.

Please rebuild the same branch and give it a try and let me know how it goes.