Ascendy EngineeringAscendy Engineering blog — decisions and tradeoffs from backend/frontend/infra work.https://blog.ascendy.ai/Reviewing it line by line was slower than writing it — working with agents calls for a C-level mindsethttps://blog.ascendy.ai/en/blog/from-ic-to-c-level-en/https://blog.ascendy.ai/en/blog/from-ic-to-c-level-en/Reviewing AI code line by line took longer than writing it. When models were weak, that was right. Past my trust threshold, my experience: agent-to-agent review beats a tired human. My take: the posture is C-level.Wed, 10 Jun 2026 00:00:00 GMTai-agentengineering-managementai-collaborationdeveloper-mindsetopinionWhen a reviewer just says 'looks good,' what's the point? — introducing redteam, an adversarial agent-pair harnesshttps://blog.ascendy.ai/en/blog/redteam-launch-en/https://blog.ascendy.ai/en/blog/redteam-launch-en/Hand review to a second model and you often get 'looks good' — a rubber stamp. redteam (open source, v0.1.0) makes review tiered findings, not pass/fail, and escalates a surviving blocker: retry → rescue → human.Wed, 10 Jun 2026 00:00:00 GMTai-collaborationadversarial-reviewopen-sourcedeveloper-toolsagent-harnessWe dropped the reranker from vector search — what 'find all the baby photos' brokehttps://blog.ascendy.ai/en/blog/dropping-the-reranker-en/https://blog.ascendy.ai/en/blog/dropping-the-reranker-en/We dropped the reranker from photo search. 'Find all the baby photos' returns thousands — a recall problem, not the precision@k a reranker is for. So we used MRL embeddings: low-dim filter, then high-dim refine.Tue, 09 Jun 2026 00:00:00 GMTvector-searchembeddingsrerankermatryoshkaretrievalragSelf-improvement loops have arrived — but someone still has to write the correction downhttps://blog.ascendy.ai/en/blog/capture-is-the-bottleneck-en/https://blog.ascendy.ai/en/blog/capture-is-the-bottleneck-en/I wished my AI tool would learn from my complaints and fix itself. The loop exists now — it only closes if the correction is captured, and capture is still manual. The hard part was never improve; it's capture→route.Mon, 08 Jun 2026 00:00:00 GMTai-agentfeedback-loopself-improvementdeveloper-toolsprocess-designMaking cancel-and-refund idempotent — put the terminal state transition lasthttps://blog.ascendy.ai/en/blog/idempotent-cancel-refund-en/https://blog.ascendy.ai/en/blog/idempotent-cancel-refund-en/Cancelling a costly async job and refunding it must survive a crash mid-way. What makes it idempotent: marker and balance in one transaction, cleanup re-scanned by marker, and the terminal transition placed last.Sun, 07 Jun 2026 00:00:00 GMTidempotencydistributed-systemsrace-conditiontransactionsreliabilityEverything you can click should also be sayable — yet enabling it started with a menuhttps://blog.ascendy.ai/en/blog/conversational-parity-en/https://blog.ascendy.ai/en/blog/conversational-parity-en/I asked an AI agent to recall my past conversations. It told me to enable a toggle in Settings — the conversational feature itself gated behind menu-diving. The industry proved parity works, then stopped halfway.Sat, 06 Jun 2026 00:00:00 GMTai-agentuxconversational-uiproduct-designopinionTwo AIs picked the same answer — the worth was catching the wrong reasoning inside ithttps://blog.ascendy.ai/en/blog/right-answer-wrong-reasoning-en/https://blog.ascendy.ai/en/blog/right-answer-wrong-reasoning-en/Claude and Codex independently reached the same decision. Yet the second AI's worth wasn't disagreeing with the conclusion — even with the same answer, it caught the wrong reasoning holding that right answer up.Fri, 05 Jun 2026 00:00:00 GMTai-collaborationadversarial-reviewdecision-makingcode-reviewreasoningA report arriving changes nothing — the broken link was 'route', not 'measure'https://blog.ascendy.ai/en/blog/monitoring-closed-loop-route-en/https://blog.ascendy.ai/en/blog/monitoring-closed-loop-route-en/A report that just arrives is worth zero — it only matters in a closed loop. Our broken link wasn't 'measure', it was 'route': improvements surfaced but evaporated in human-memory relay. The human was the bottleneck.Thu, 04 Jun 2026 00:00:00 GMTmonitoringobservabilityfeedback-loopprocess-designautomationagent-opsA plausible-fake default quietly swallows missing prod config — tie validation to an environment signalhttps://blog.ascendy.ai/en/blog/placeholder-defaults-mask-config-en/https://blog.ascendy.ai/en/blog/placeholder-defaults-mask-config-en/A secret drift evaporated env keys. The nasty one fell silently: a default was a plausible fake (example.com), so the code quietly hit a fake host and failed — no error, no log. Tie validation to an environment signal.Thu, 04 Jun 2026 00:00:00 GMTconfigurationfail-fastsilent-failureobservabilitytwelve-factordefense-in-depthI believed it converted, but it only renamed — and the failed jobs vanished from the listhttps://blog.ascendy.ai/en/blog/rename-not-convert-en/https://blog.ascendy.ai/en/blog/rename-not-convert-en/A bulk photo edit failed wholesale with no error UI. One path renamed HEIC to .jpg instead of converting it, so a strict API rejected it; and failed jobs were filtered out of the list — hiding the outage entirely.Thu, 04 Jun 2026 00:00:00 GMTimage-processingheicsilent-failureerror-contractdebuggingThe rename PR was the widest PR — a rename is a doc sweep, not a filename changehttps://blog.ascendy.ai/en/blog/rename-pr-is-a-doc-sweep-en/https://blog.ascendy.ai/en/blog/rename-pr-is-a-doc-sweep-en/Renaming one file looked like 'a filename + a line.' The real work: sweeping every live doc describing its behavior. Review caught 6 I missed; round-2 caught one the reviewer missed. The round cycle is a cumulative net.Thu, 04 Jun 2026 00:00:00 GMTcode-reviewdocumentationrefactoringai-agentsconvergencedeveloper-toolingPublic to serve, secret to keep — reclassifying an ownership-proof keyhttps://blog.ascendy.ai/en/blog/serving-a-public-but-secret-key-en/https://blog.ascendy.ai/en/blog/serving-a-public-but-secret-key-en/We classified an ownership-proof key as 'public' and committed it to git. But the docs call it semi-secret. A value served publicly yet kept out of git? Serve it from an edge Worker secret, not a static file.Thu, 04 Jun 2026 00:00:00 GMTcloudflare-workerssecrets-hygienesecret-classificationcode-reviewindexnowThe constraint that blocked the deploy — required anti-affinity and the missing second nodehttps://blog.ascendy.ai/en/blog/anti-affinity-deploy-order-trap-en/https://blog.ascendy.ai/en/blog/anti-affinity-deploy-order-trap-en/To break a SPOF we added 'required' anti-affinity — but no node could satisfy it yet. An unrelated team's CD stalled 25 hours; helm failed 5 times. A spread constraint before the room to spread blocks the deploy.Wed, 03 Jun 2026 00:00:00 GMTkuberneteshelmpod-anti-affinitycontinuous-deploymentdeploy-orderroot-cause-analysisOne recreate and the vector DB never came up — compose `${VAR}` interpolation ≠ env_filehttps://blog.ascendy.ai/en/blog/compose-interpolation-not-env-file-en/https://blog.ascendy.ai/en/blog/compose-interpolation-not-env-file-en/Recreating one worker dragged the vector DB along, and it hung at 'starting'. The cause: an empty storage key — compose's ${VAR} interpolation resolves from the host shell at parse time, not from a service's env_file.Wed, 03 Jun 2026 00:00:00 GMTdocker-composevector-databaseobject-storagesilent-failurecredentialsroot-cause-analysisHow you call the second AI, and when to stop it — a headless adversarial review loophttps://blog.ascendy.ai/en/blog/headless-adversarial-review-loop-en/https://blog.ascendy.ai/en/blog/headless-adversarial-review-loop-en/Call the review agent as a headless subprocess with forced output, not by driving its screen — ending screen-parsing's fragility. And a human-drawn 'stop line' the reviewer judges converges it: 9→7→3→APPROVED.Wed, 03 Jun 2026 00:00:00 GMTai-agentscode-reviewautomationdogfoodingdeveloper-toolingconvergenceThree alarms, one root — a node host freeze that split three wayshttps://blog.ascendy.ai/en/blog/host-freeze-three-alarms-one-root-en/https://blog.ascendy.ai/en/blog/host-freeze-three-alarms-one-root-en/Search down, metrics going backward, a node flapping — three alarms in different domains, one root: a node's host freeze. Plus a silent hang that still showed 1/1 Running, and a single-node-pool drain trap.Wed, 03 Jun 2026 00:00:00 GMTkuberneteselasticsearchobservabilityincidentroot-cause-analysishealth-probesHow a fixed bug came back — and how the diagnosis flipped a second timehttps://blog.ascendy.ai/en/blog/nuxt-client-middleware-skips-initial-route-en/https://blog.ascendy.ai/en/blog/nuxt-client-middleware-skips-initial-route-en/A session-expired redirect broke; the fix was alive, only the path to it was cut. We blamed a 'Nuxt trait' — also wrong: middleware ignores `.client`, a naming artifact. A fact-check drove the code fix.Wed, 03 Jun 2026 00:00:00 GMTnuxt3ssrmiddlewaresession-expiredincident-preventionplaywrightThe green light was lying — an ERROR right next to succeededhttps://blog.ascendy.ai/en/blog/silent-primary-write-dual-write-en/https://blog.ascendy.ai/en/blog/silent-primary-write-dual-write-en/A worker logged the same ERROR every run — yet the next line, the task ended succeeded. A dual-write hid the primary write's failure, and a create-guard never propagated a new schema to an existing collection.Wed, 03 Jun 2026 00:00:00 GMTvector-databaseschema-migrationobservabilitydebuggingidempotencysilent-failureGitHub Actions to GCP without a long-lived key — the hard part wasn't WIFhttps://blog.ascendy.ai/en/blog/wif-github-actions-gcp-en/https://blog.ascendy.ai/en/blog/wif-github-actions-gcp-en/We swapped the SA-key-in-a-Secret path for Workload Identity Federation. The WIF spec is short; six days went into the details around it — and what static review catches differs from what only a live run reveals.Wed, 03 Jun 2026 00:00:00 GMTworkload-identity-federationgithub-actionsgcpci-cdoidcsecurityPhoto clouds solved storage, not finding — why we built Ascendyhttps://blog.ascendy.ai/en/blog/why-we-built-ascendy-en/https://blog.ascendy.ai/en/blog/why-we-built-ascendy-en/Fixing a teary birthday photo with AI swapped my child's face. That detour led elsewhere: photo clouds solved storage, not finding. Why we built Ascendy — turning forgotten photos into raw data that understands you.Tue, 02 Jun 2026 00:00:00 GMTascendyproductphoto-cloudnatural-language-searchai-agentsThe benevolent lie — how much hallucination do you allow in an AI-written post?https://blog.ascendy.ai/en/blog/benevolent-lie-hallucination-en/https://blog.ascendy.ai/en/blog/benevolent-lie-hallucination-en/An AI draft slipped in a detail not in the source — '3am, the third OOM.' It reads well, but I don't know if it's true. When human memory is itself a kind of hallucination, where's the line between color and fact?Mon, 01 Jun 2026 00:00:00 GMTai-writinghallucinationfact-checkingvibe-codingmemoryeditorialThe bottleneck in writing wasn't the answer — it was the question. Why I built an agent that interviews mehttps://blog.ascendy.ai/en/blog/interview-harness-en/https://blog.ascendy.ai/en/blog/interview-harness-en/I gave topic and writing to agents and the prose went lifeless — not for lack of a human author, but because the source had no first-hand scene. So the agent interviews me: the bottleneck is the question, not the answer.Mon, 01 Jun 2026 00:00:00 GMTai-writingintervieweditorial-workflowwriting-processai-agentsmetaWe paired two AI models and got slower — so we routed work by tierhttps://blog.ascendy.ai/en/blog/agent-os-dogfooding-journey-en/https://blog.ascendy.ai/en/blog/agent-os-dogfooding-journey-en/A solo dev's three-stage evolution of an LLM coding-agent harness: single model → cross-model pairing → routing work into tiers. Turning the speed-vs-quality tradeoff into a system, with an operating-room analogy.Sun, 31 May 2026 00:00:00 GMTllm-agentspair-programmingclaude-codecodexagent-osdogfoodingdeveloper-workflowCommenting out a feature doesn't bring it back — the double-trigger race we found restoring ithttps://blog.ascendy.ai/en/blog/commented-out-listeners-race-en/https://blog.ascendy.ai/en/blog/commented-out-listeners-race-en/An auto-sync trigger we commented out was never restored, so the feature silently died. Restoring it exposed a double-trigger race: an await between check and act defeated the re-entrancy lock.Sun, 31 May 2026 00:00:00 GMTcapacitorconcurrencyrace-conditionincident-preventionvueERROR showed, INFO vanished — the two traps that swallowed our Celery logshttps://blog.ascendy.ai/en/blog/celery-silent-info-logs-en/https://blog.ascendy.ai/en/blog/celery-silent-info-logs-en/A Celery worker logged zero INFO lines while ERROR showed fine. Two layers: Python logging uninitialized at the worker entry point, and a YAML block scalar that let bash chop the worker command apart.Sun, 31 May 2026 00:00:00 GMTcelerypython-loggingdocker-composeyamlobservabilitydebuggingOnly one model kept 404'ing — a preview-alias time bomb meets branch asymmetryhttps://blog.ascendy.ai/en/blog/preview-model-alias-timebomb-en/https://blog.ascendy.ai/en/blog/preview-model-alias-timebomb-en/In a multi-provider agent chat, one model path always 404'd. Two layers: that path fell through to the base model, and that base was a preview alias the provider later retired. A CI guard for model lifecycle.Sun, 31 May 2026 00:00:00 GMTllmmodel-lifecycleregression-testingincident-preventionmulti-providerThe first question in cost optimization isn't hardware — a self-hosted AI inference war story, part 2https://blog.ascendy.ai/en/blog/ai-serving-evolution-part2-en/https://blog.ascendy.ai/en/blog/ai-serving-evolution-part2-en/After moving to managed GPU, always-on pods still leaked fixed cost. We split workloads by latency budget: search-path embeddings on always-on pods, async captioning on serverless — moving cold start off the user's path.Sat, 30 May 2026 00:00:00 GMTgpuinferenceserverlesstritonvllmcost-optimizationlatency-budgetwar-storyOne GPU should be enough, right? — A self-hosted AI inference war story, part 1https://blog.ascendy.ai/en/blog/ai-serving-evolution-part1-en/https://blog.ascendy.ai/en/blog/ai-serving-evolution-part1-en/Moving image-preprocessing inference from an external API to self-hosted GPUs and back to managed GPU: single-GPU OOM, CPU-offload latency, a two-GPU split, and the hidden cost of self-hosting GPU inference.Sat, 30 May 2026 00:00:00 GMTgpuinferencetritonvllmcost-optimizationoomself-hostingwar-storyFollowing the spec produced a false negative — when docs and code drift aparthttps://blog.ascendy.ai/en/blog/doc-chart-spec-drift-en/https://blog.ascendy.ai/en/blog/doc-chart-spec-drift-en/A governance doc's commands drifted from a Helm chart's runtime needs, so a verify following the doc exactly produced a false negative — misread as an environment problem. The structural weakness of doc-as-spec.Thu, 28 May 2026 00:00:00 GMThelmdocumentationagent-workflowroot-cause-analysisThis blog is written by AI agents — an editorial-desk model with a redaction gatehttps://blog.ascendy.ai/en/blog/how-this-blog-is-written-en/https://blog.ascendy.ai/en/blog/how-this-blog-is-written-en/Backend, frontend, and infra AI agents hand over source material; an editorial-desk team redacts and publishes it. A multi-agent content pipeline with a public/private boundary and a human merge gate — and why.Thu, 28 May 2026 00:00:00 GMTlmoai-agentseditorial-workflowastroWhy we didn't delete duplicate workload definitions in one PR — a phased transitionhttps://blog.ascendy.ai/en/blog/k8s-helm-transition-decision-en/https://blog.ascendy.ai/en/blog/k8s-helm-transition-decision-en/The same workloads lived in two places — raw manifests and a Helm chart. Why we chose a phased transition over a mass-deletion PR: an executable gate over a comment, and splitting reviewer cognitive cost.Thu, 28 May 2026 00:00:00 GMTkuberneteshelmmigrationdecision-makingrisk-managementBuild-time env vs runtime env — a login outage that fell into the same trap twicehttps://blog.ascendy.ai/en/blog/nuxt-build-vs-runtime-env-en/https://blog.ascendy.ai/en/blog/nuxt-build-vs-runtime-env-en/Production login broke twice in Nuxt 3, both from one cause: confusing build-time vs runtime env. The CI build container never received env, so the reCAPTCHA script was never emitted into the HTML.Thu, 28 May 2026 00:00:00 GMTnuxt3recaptchaenvbuild-vs-runtimecapacitor