The Case of the Password That Rang a Bell
A short detective story about a database that couldn't read its own config, a password that looked perfect in every tool that touched it, and one byte that had been hiding in plain sight since long before anyone typed it.
┌──────────────────────────────────────────────────────────────┐ │ INVESTIGATION BOARD │ ├──────────────────────────────────────────────────────────────┤ │ │ │ [ analytics DB crash-loops: "invalid token" ] │ │ │ │ │ ┌─────────────────┼─────────────────┐ │ │ ▼ ▼ ▼ │ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ │ │ SUSPECT │ │ SUSPECT │ │ SUSPECT │ │ │ │ the YAML │ │ port 9009 │ │ the Helm │ │ │ │ we wrote │ │ conflict │ │ chart │ │ │ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │ │ │ innocent │ innocent │ innocent │ │ ▼ ▼ ▼ │ │ ┌──────────────────────────────┐ │ │ │ ACTUAL CULPRIT │ │ │ │ one unprintable byte in │ │ │ │ the password — invisible │ │ │ │ to every tool that read it │ │ │ └──────────────────────────────┘ │ │ │ │ evidence: the same password, printed, looked correct │ │ ...and re-encoding it produced a different string │ └──────────────────────────────────────────────────────────────┘
The Complaint
The Company runs a columnar analytics database on Kubernetes, deployed by GitOps out of a repository that describes the QA environment. It stores time-series metrics. A dashboarding service reads from it with a dedicated read-only account, and the account's password comes out of AWS Secrets Manager, delivered into the cluster by External Secrets Operator and handed to the database as an environment variable.
Routine. Roughly forty other services in that repository work the same way.
The complaint was that the database pod would not start. It crash-looped, and it did so with the most unpromising kind of error — a config parse failure:
Failed to preprocess config '/etc/clickhouse-server/users.yaml': SAXParseException: Invalid token in line 1 column 13
But there was a detail attached that made the whole thing much more interesting than a typo. A teammate had already tried something: replace the password reference with a hardcoded literal string, and the database boots perfectly. Put the secret reference back, and it dies.
The config file was byte-identical in both cases apart from that one value. So this wasn't a YAML mistake. Something about fetching the password was killing the database.
Chapter 1 — Three Confessions and a Corpse
The log was mostly noise, and the noise was the loud kind:
Poco::Exception ... Not found: tmp_path Poco::Exception ... Not found: user_files_path Warning: Effective user of the process does not match the owner of the data Warning: Listen [0.0.0.0]:9009 failed: Address already in use Error: Failed to preprocess config '.../users.yaml': SAXParseException ...
Three of those are decoys, and it's worth naming them because they cost real time in incidents like this.
The Not found: lines come from a helper the container's entrypoint runs before the server starts. They appear on healthy pods too.
"Address already in use" is the good decoy. It looks like a genuine, actionable networking problem — a port collision, a duplicated Service, an IPv6 misconfiguration. It is none of those. In a single-container pod, the only thing that can be holding a port is the previous copy of the same process, still shutting down from the last crash. It's a symptom of the crash loop. It would vanish the moment the real problem was fixed.
That's a rule worth keeping: in a tight restart loop, most of the log is the wreckage of the last attempt, and the wreckage is often more photogenic than the cause.
The corpse was the last line. And it had something odd about it.
The file is called users.yaml. The error is an XML parser error. SAXParseException is not what a YAML parser says when it's unhappy. Something in this pipeline had turned our YAML into XML and then choked on it.
Chapter 2 — The Suspect With Three Alibis
The natural first suspect was the config we'd written. The password reference used a slightly exotic syntax — this database's config format lets a YAML key beginning with @ become an XML attribute, which is how you say "take this value from an environment variable":
dashboard_reader:
password:
- '@from_env': DASHBOARD_DB_PASSWORD
That looks like the kind of thing that's subtly wrong. A list where a scalar was expected, maybe, or the wrong quoting on the @.
Except the same file defined three other accounts using the identical shape, pointing at three different environment variables, and all three had been working for weeks. Whatever was wrong, it wasn't the syntax.
So the structure was innocent, the chart was innocent — it turned out to be a thin passthrough that just copies your values into a custom resource for an operator to render, with no templating to blame — and we were left with the one thing that differed between the working case and the broken one: the value itself.
Time to read the source.
Chapter 3 — The Mechanism, In One Line
Open-source has a wonderful property during an incident: you can go and look.
The database's config processor implements environment-variable substitution like this:
env_document = dom_parser.parseString("<from_env>" + std::string{env_val} + "</from_env>");
Read it slowly, because everything follows from it.
To inject an environment variable into its configuration, the server builds a small XML document by gluing strings together, and re-parses it. There is no escaping step. Whatever bytes are in that environment variable get spliced directly into XML source and handed to an XML parser.
Which means the constraint is not on your config. The constraint is on your secret. A password containing & or < is not a password as far as this code path is concerned — it's malformed XML, and the server refuses to start.
It also explains the hardcoded-value clue perfectly. When the value is written literally in the YAML, a completely different code path handles it: the YAML parser builds the config node through a document API, where escaping is automatic and nothing is ever re-parsed. Only the substitution path concatenates strings.
That asymmetry is a genuinely useful diagnostic, and it generalizes far past this database:
Replace the indirection with a literal. If hardcoding works and the lookup fails, the problem is the value's bytes or the lookup mechanism — not your config structure.
Chapter 4 — Where I Got It Wrong
Now I had a mechanism, and I got greedy.
The error said line 1, column 13. The injected wrapper <from_env> is exactly ten characters. So the password starts at column 11, and with a little arithmetic I could tell the engineer exactly which character of their password to look at. I even reproduced it — driving the same XML parser the database uses, from Python, feeding it fabricated passwords until the error matched:
import xml.parsers.expat as x
p = x.ParserCreate()
try: p.Parse("<from_env>" + pw + "</from_env>", True)
except x.ExpatError as e:
print(x.ErrorString(e.code), p.ErrorLineNumber, p.ErrorColumnNumber)
Out came a confident, specific fingerprint: the second character of your password is & or <, and the third is a digit or a symbol. Character two. I said so plainly.
It was wrong, and it was wrong for the dullest possible reason.
Column numbers don't have a universal starting point. The underlying parser counts columns from zero. The C++ library wrapping it passes that number straight through without converting it to the one-based convention every text editor uses. I had assumed the conversion happened. That single off-by-one moved the accusation to a different character, and every confident detail I derived on top of it inherited the error.
The arithmetic, once actually measured instead of assumed:
character index = reported column − (length of the injected prefix) + 1
Which puts the culprit at character four, not two.
Wrong turns are cheap when you check them. This one was cheap because the engineer went and looked at the real value instead of taking my fingerprint at face value — and what they brought back was much better than my answer.
Chapter 5 — The Byte That Rang a Bell
They pulled the stored secret and decoded it. Here's what that looks like, with fabricated values that behave exactly like the real ones did:
$ printf %s 'cXR6Bzh2YXVsdDNNblBr' | base64 -d qtz8vault3MnPk
Fourteen characters. Looks like a password. Looks correct.
Then they re-encoded what they'd just seen, to check:
$ printf %s 'qtz8vault3MnPk' | base64 cXR6OHZhdWx0M01uUGs=
A different string. Decode a value, re-encode it, get something else. That's impossible — unless what you saw wasn't what was there.
$ printf %s 'cXR6Bzh2YXVsdDNNblBr' | base64 -d | od -An -c q t z \a 8 v a u l t 3 M n P k
There it is. \a — byte 0x07, ASCII BEL, the terminal bell. Character four.
It never appeared on screen because BEL is not a character, it's an instruction. Printing it doesn't draw anything; it asks the terminal to make a noise. The password had fifteen bytes and displayed fourteen. Every tool in the chain had been perfectly honest and completely useless: the secret store held it, the operator delivered it, the shell printed it, and none of them had any reason to mention it.
XML 1.0 forbids control characters in content — only tab, newline and carriage return are legal. So when the database glued this password into its little XML fragment, it produced a document that no conforming parser would accept. One inaudible byte, and a database that couldn't read its own configuration.
And the position was character four. Exactly where the corrected arithmetic said to look.
Chapter 6 — The Twist: The Encoding Was Never Broken
Here's the part that turns a one-off into a lesson.
The obvious story is "the base64 got corrupted." It didn't. Line up the two strings in the four-character groups base64 actually works in:
stored: cXR6 Bzh2 YXVs dDNN blBr -> 15 bytes
intended: cXR6 OHZh dWx0 M01u UGs= -> 14 bytes
^^^^ ^^^^
same everything after here differs
A typo inside a base64 string changes one group in place, or fails to decode at all. This is different: the alignment diverges after the very first group, and the byte counts don't match. That's the signature of a correct encoding of the wrong input.
In other words the stray byte was already in the plaintext before anyone base64-encoded it. The encoding did its job faithfully. Whatever generated or pasted that password produced the bad byte, and re-typing the secret fixed this instance and nothing at all about the cause.
Which is the uncomfortable part. Nobody knows how a BEL got into a password. A copy-paste from somewhere with a control character in it, a generator script, a clipboard round-trip through a tool that meant well. It's unresolved, which means it can happen again, silently, in exactly the same way.
One structural detail made it worse. The External Secrets configuration used a base64 decoding strategy, meaning the secret store deliberately holds encoded data and the operator decodes it on the way in. So the real bytes were invisible in the cloud console (encoded), invisible in the Kubernetes Secret (encoded again), and invisible in a terminal (unprintable). There was no layer at which a human could have seen this without deliberately going looking with the right tool.
Chapter 7 — The Fix, and the Fix That Would Have Been Worse
The repair was to re-encode the password correctly and store it again. No code changed. No manifest changed. The database came up on the next sync.
But there was a tempting alternative that deserves a warning label, because it's the fix most people would reach for on the other version of this bug — the one where the password legitimately contains an ampersand.
If XML is the problem, escape it. Store & instead of &. The database's XML parser unescapes it back to &, the account gets the password you intended, everything works.
Everything except the dashboarding service, which reads the same secret key and does no unescaping whatsoever. It would send the literal &. The database would hold one password, the dashboard would send another, and the failure would surface later as an authentication error with no visible connection to the encoding change that caused it.
That's the trap: escaping at rest converts a loud startup crash into a silent mismatch. A secret shared between a strict consumer and a relaxed one cannot carry either one's escaping. The store holds one canonical value; each reader does its own quoting.
The options that actually work:
- Constrain the value to what every consumer accepts — no
&, no<, no control bytes. Fixes the instance, enforces nothing. - Check the bytes at write time, before they ever reach the secret store. Fixes the class.
- Give the strict consumer a different representation. This database accepts a SHA-256 hash of the password instead of the password. Hex output cannot contain a hostile byte, so no value could ever break the config again. It costs a second secret key kept in sync with the plaintext the dashboard still needs — which is why it's the right answer and also the one that didn't get adopted.
Option 2 is a two-line check and it's the one I'd actually push for:
printf %s "$B64" | base64 -d | LC_ALL=C grep -qP '[^\x20-\x7E]' \ && echo 'REJECT: non-printable byte' || echo OK
Worth noting where it can't live, though: the repositories in this story have no CI at all — the deployment controller reads the main branch directly. There's no pipeline to hang a check on. That's a gap with a growing queue of things that want it.
Hints for the Reader
base64 -dpiped to your terminal is not a validation step. Control bytes don't render. BEL rings a bell, backspace eats the previous character, carriage return moves the cursor to the start of the line and lets the rest overwrite what you already read. Useod -c, or grep for anything outside printable ASCII. What you see is a rendering, not the data.- Valid base64 does not mean correct plaintext. Encoding integrity and content integrity are different properties, and only one of them gets checked anywhere. If the decoded byte count or the group alignment differs from what you expect, the corruption predates the encoding — and rotating the value fixes today without touching whatever produced it.
- Read the conditional clause in the source, not the reassuring sentence in the docs. "Takes the value from an environment variable" is true. "Concatenates it into XML and re-parses it, unescaped" is the part that decides which passwords are legal. One line of open-source answered a question that no amount of config-staring would have.
- Line and column numbers have no universal base. Parsers, bindings and wrappers each pick one, and they don't agree. Before deriving a character position from an error message, verify the convention empirically — otherwise you'll accuse the wrong character with total confidence, which is worse than saying you don't know.
- Replace the indirection with a literal to split "bad value" from "bad wiring." One deploy, and the answer is unambiguous in either direction. Make the literal obviously fake, and leave a note in the same diff about what to revert to — in this story the diagnostic placeholder and its comment were still sitting in the repository after the real cause had been found and fixed somewhere else entirely.
- You can often reproduce a compiled service's parser bug without the service. No cluster access and no container runtime here — so I found the exact call in upstream source, identified the underlying library, and drove that same library from Python with the exact string the server builds. Matching the message and the line and the column is what makes it evidence instead of a plausible theory.
- In a crash loop, read bottom-up to the first real error, then forward from the start of that attempt. Everything else belongs to a previous run. A port conflict inside a single-container pod is the corpse of the last process, not a networking problem.
- Secrets shared across consumers can't carry one consumer's escaping. If one reader needs special characters encoded, that encoding belongs at that reader's boundary — or give it a hash, or give it its own key. Never bake it into the stored value, or you've traded a crash for a silence.