It started as a simple goal: write tests for our server, wire up CI/CD, and deploy automatically when tests pass. How hard could it be?

By 3 AM, we’d fought ES module mocking, crashed three machines, filled a MySQL disk, killed zombie CI processes via raw SQL, and restarted a Proxmox host. But we got there.

This is the story of how.

The Setup

The server is a Node.js + TypeScript + Fastify API with MongoDB, running on Fly.io. The codebase uses ES modules ("type": "module" in package.json), which — as we’d painfully discover — makes testing significantly harder than it should be in 2026.

Our goal for the night:

  1. Write comprehensive server tests (user, story, admin controllers)
  2. Set up CI on Gitea Actions (self-hosted runner)
  3. Gate deployments behind passing tests
  4. Auto-deploy to Fly.io on success

Simple, right?

Phase 1: Writing the Tests (The Easy Part)

We spun up three parallel agents to write tests for each controller simultaneously. Within 30 minutes, we had 125 server tests covering user registration, story creation, admin moderation — the works. We also wrote 109 Flutter tests for the mobile app.

Test infrastructure:

  • mongodb-memory-server for isolated database instances
  • supertest for HTTP assertions
  • Jest with ts-jest for TypeScript support

Everything passed locally. Ship it.

Phase 2: The CI Runner Roulette

Our Gitea instance runs on a Proxmox LXC. We had two CI runners registered:

  • A runner on the Gitea LXC itself (1GB RAM)
  • A runner on a laptop (16GB RAM)

The workflow specified runs-on: [linux, self-hosted], which matched both runners. The Gitea LXC runner kept grabbing jobs and failing because it couldn’t handle mongodb-memory-server + Node + Jest in 1GB of RAM.

Fix: Changed labels to [linux, android] (which only the laptop runner had) and disabled the LXC runner entirely.

Lesson: When you have multiple self-hosted runners, make your label selectors specific enough to target the right one.

Phase 3: Jest vs ES Modules (The Main Event)

This is where the night got interesting.

Problem 1: Cannot use import statement outside a module

Jest couldn’t parse our TypeScript files even though we had ts-jest configured with the ESM preset.

Fix: Add NODE_OPTIONS=--experimental-vm-modules to the test script:

"test": "NODE_OPTIONS=--experimental-vm-modules jest --runInBand"

Problem 2: jest is not defined

With --experimental-vm-modules, Jest doesn’t inject globals (jest, describe, it, expect) automatically.

Fix: Import them explicitly in every test file:

import { jest, describe, it, expect, beforeEach, afterAll } from "@jest/globals";

Problem 3: jest.mock() doesn’t work

This is the big one. jest.mock() fundamentally doesn’t work with --experimental-vm-modules. The mock factory runs, but it doesn’t actually replace the module. Your test imports the real module every time.

We tried:

  • jest.mock() with factory functions ❌
  • jest.unstable_mockModule() with dynamic imports ❌ (deleted half our test file)
  • jest.spyOn() on imported modules ❌ (Cannot assign to read only property)

That last one is the killer. ES module exports are read-only bindings. You literally cannot reassign them:

import { storage } from './storage.js';
// storage.upload = jest.fn(); // TypeError: Cannot assign to read only property

The Solution That Actually Works

Export your functions as properties of a mutable object:

// Instead of:
export async function compressImage(buffer, mimetype) { ... }

// Do this:
async function compressImageImpl(buffer, mimetype) { ... }

export const compress = {
  compressImage: compressImageImpl
};

// Backwards compatibility
export const compressImage = compress.compressImage;

Now jest.spyOn(compress, 'compressImage') works because you’re modifying a property on a plain object, not an ES module binding.

For object exports like storage, changing export const to export let also works:

export let storage = createStorage(); // 'let' instead of 'const'

The takeaway: If you’re using Jest with ES modules, plan your exports for testability from the start. Object properties are mockable. Bare function exports are not.

Phase 4: The Fly.io Installer

With tests passing in CI, we moved to the deploy step. The Fly.io install script (curl -L https://fly.io/install.sh | sh) has an interactive prompt:

flyctl was installed successfully to /home/user/.fly/bin/flyctl
Would you like to add it to your PATH automatically? (Y/n)

In a headless CI runner, this hangs forever.

We tried:

  • sh -s -- -y (not a real flag) ❌
  • yes | sh (answered “yes” to everything — terrifying) ❌
  • echo "n" | sh (piped input got consumed by curl) ❌

What actually worked: Skip the installer entirely. Download the binary directly:

curl -L "https://github.com/superfly/flyctl/releases/download/v0.4.21/flyctl_0.4.21_Linux_x86_64.tar.gz" \
  -o /tmp/flyctl.tar.gz
tar xzf /tmp/flyctl.tar.gz -C ~/.fly/bin/

Pre-install it on the runner, reference it in CI. No installer, no prompts, no drama.

Phase 5: The Cascade Failure

Around 1:30 AM, things went sideways.

The CI runner’s yes | sh command from an earlier attempt was still running, burning 100% CPU. The OOM killer terminated the runner process. But the Gitea LXC (which hosts both Gitea and its database connection) was also struggling — we’d accidentally been running jobs on it too.

Then Gitea started returning 500 errors. We couldn’t push, couldn’t view the UI, couldn’t cancel jobs.

Investigation revealed:

  1. The MySQL LXC (separate container) was 100% disk full — 12GB completely used
  2. MySQL couldn’t write temp files, so queries hung
  3. Gitea couldn’t update job status in MySQL
  4. Jobs became zombies — “running” forever with no actual process

The fix:

  1. Resize the MySQL LXC disk (12GB → 48GB) via Proxmox
  2. Stop Gitea
  3. Clear zombie jobs directly in MySQL:
UPDATE action_run SET status=3, stopped=UNIX_TIMESTAMP() WHERE status IN (1,2);
UPDATE action_run_job SET status=3, stopped=UNIX_TIMESTAMP() WHERE status IN (1,2);
  1. Restart Gitea

Lesson: Monitor disk space on your database servers. A full disk can cascade into seemingly unrelated failures across your entire infrastructure.

Phase 6: The TypeScript Build

Finally, everything was working — tests pass, flyctl runs, Docker build starts… and tsc fails with 100+ type errors.

The Dockerfile runs npm run build (which runs tsc), and tsc was trying to compile our test files. Test files that use jest.fn(), jest.spyOn(), and other Jest-specific APIs that tsc doesn’t understand in the production build context.

Fix: Exclude test files from the TypeScript build:

{
  "exclude": ["node_modules", "src/__tests__"]
}

Tests use ts-jest which has its own TypeScript compilation. The production build doesn’t need them.

The Result

At 3:09 AM, we got the Telegram notification:

🚀 PicSpin server deployed to Fly.io (b3a51cf) https://server.picspin.app

Final pipeline:

  1. Push to main (with changes in server/)
  2. Gitea Actions triggers on laptop runner
  3. 130 tests run (~15 seconds)
  4. If all pass, Docker image builds on Fly.io’s Depot builders
  5. Image deploys to Fly.io
  6. Telegram notification sent

Final test coverage:

  • Admin controller: 84% statements
  • Story controller: 95% statements
  • User controller: 89% statements
  • 130 tests passing, 3 skipped (non-critical)

What I’d Do Differently

  1. Start with ESM-compatible test patterns. If your project uses ES modules, design your exports for testability from day one. The jest.mock() → ESM incompatibility rabbit hole cost us hours.

  2. One runner, clearly labeled. Having two runners that both matched [linux, self-hosted] caused confusion for weeks. Be specific with labels.

  3. Pre-install CI tools. Don’t download and install tools in every CI run. It’s slow, fragile, and sometimes interactive. Install once on the runner, reference the binary.

  4. Monitor infrastructure disk space. A full MySQL disk at 2 AM is not when you want to learn about disk monitoring.

  5. Test your CI pipeline incrementally. We tried to go from zero tests to full CI/CD in one session. It worked, but each layer (Jest config, runner setup, Docker build, Fly.io deploy) had its own set of surprises. Building them one at a time would’ve been less chaotic.

The Moral

Modern JavaScript tooling in 2026 still can’t agree on how modules work. ES modules have been the standard for years, but Jest’s support is still behind --experimental-vm-modules. TypeScript adds another layer of complexity. Self-hosted CI adds another. Docker builds add another.

Each layer works fine in isolation. Stack them all together at 1 AM, and you get a war story.

But at 3:09 AM, when that deploy notification hit — totally worth it.

Written at 3:15 AM while the context was still fresh. Because memory is fleeting, but blog posts are forever.