Tracking changes made vs upstream's main branch to enable self hosting #1

Open

GrocerPublishAgent wants to merge 24 commits from self-host into upstream-main

pull from: self-host

merge into: comrades:upstream-main

comrades:upstream-main

Conversation 0 Commits 24 Files changed 90 +7037 -2503

GrocerPublishAgent commented 2026-04-12 01:18:43 +00:00

Owner

Copy link

No description provided.

GrocerPublishAgent added 5 commits 2026-04-12 01:18:46 +00:00

Add self-hosting infrastructure and make server fully configurable ... ccc49a5b0e

## Server: replace hardcoded URLs with runtime-configurable values

Previously `server-origin`, `dashboard-origin`, `s3-bucket-name`, and
`cloudfront-s3-bucket-url` were all resolved at startup by switching on
the `:env` value (`:prod` / `:staging` / `:dev`), making every value
hardcoded to Instant's own infrastructure.

Now each is driven by an env var or config.edn key with no fallback to
Instant-specific defaults:

- `SERVER_BASE_URL` / `:server-base-url`   → replaces `server-origin`
- `DASHBOARD_BASE_URL` / `:dashboard-base-url` → replaces `dashboard-origin`
- `S3_BUCKET_NAME` / `:s3-bucket-name`     → replaces hardcoded bucket names
- `CLOUDFRONT_S3_BUCKET_URL` / `:cloudfront-s3-bucket-url`

A `normalize-base-url` helper strips trailing slashes so concatenation
with path segments works regardless of whether the operator wrote
`example.com/api` or `example.com/api/`.

OAuth redirect URLs in `dash/routes` and `runtime/routes` were changed
from `def` (resolved once at load time) to `defn` (resolved per
request) so they pick up the configured `server-base-url`.

## Server: support URL path prefixes for dashboard CORS

The dashboard can now be served at a path prefix
(e.g. `https://example.com/dashboard`) rather than requiring a bare
domain.  `allow-cors-origin?` previously compared the `Origin` header
directly against `dashboard-origin`; that breaks when the origin
includes a path.  A new `dashboard-cors-origin` helper extracts just
the `scheme://host[:port]` portion via `java.net.URI` so the comparison
works correctly in both cases.

## Server: Minio as a drop-in S3 replacement

Four new env vars control the S3 client:

- `S3_ENDPOINT`      — custom endpoint URL (e.g. `https://minio.corp.example.com`)
- `S3_REGION`        — region string passed to the SDK
- `S3_PATH_STYLE`    — set to `true` to use path-style addressing (`host/bucket/key`)
                       instead of virtual-hosted (`bucket.host/key`), required by Minio
- `S3_USE_AWS_PRIVATE_APIS` — opt-in to the AWS CRT async client; when false (default)
                              the standard SDK async builder is used so Minio works

The sync `S3Client`, async `S3AsyncClient`, and presign-credentials
builder all apply these overrides. `S3_ACCESS_KEY` / `S3_SECRET_KEY`
can now also be provided as env vars (in addition to config.edn) to
supply static credentials without AWS IAM.

The presigned URL generator (`aws_signature.clj`) was updated to use
the endpoint's scheme and authority when path-style is active, so
generated URLs point at the Minio host rather than `s3.amazonaws.com`.

## Self-hosting: guided setup and deployment scripts

`self-hosting/setup-local-dev` is a Nushell script that walks through
every prerequisite for a working dev environment and reports exactly
what needs to be fixed:

- Checks Java 22+, Clojure CLI, Node.js, pnpm, golang-migrate
- Validates the `.env` file and reports any missing required keys
  (`DATABASE_URL`, `S3_ENDPOINT`, `S3_REGION`, `S3_BUCKET_NAME`,
  `S3_ACCESS_KEY`, `S3_SECRET_KEY`) with an env-file template if absent
- Checks optional URL config (`SERVER_BASE_URL`, `DASHBOARD_BASE_URL`,
  `S3_PATH_STYLE`) and warns if unset
- Connects to PostgreSQL and verifies version (≥14), `wal_level=logical`,
  `max_replication_slots`, `max_wal_senders`, `pg_hint_plan` in
  `shared_preload_libraries`, `wal2json` plugin, and that migrations
  have been applied
- Checks Minio health endpoint and (via `mc`) verifies the bucket exists

Supporting Nushell libraries under `self-hosting/lib/`:
- `dotenv.nu` — parses `.env` / `env.local` files into records
- `pg.nu`     — PostgreSQL connectivity and config checks
- `s3.nu`     — Minio/S3 health and bucket checks
- `ui.nu`     — coloured ok/warn/fail/info output helpers

Additional scripts:
- `setup-remote-environment` — provisions a fresh remote host
- `run-server` / `run-dashboard` — thin wrappers to start each process
- `deploy.sh` + Nomad HCL files — Nomad-based deployment for server
  and dashboard

## Other

- Add new `@instantdb/qwik` package and Qwik City sandbox app
- Remove GitHub star-count feature from marketing site (StarBurst
  component, getGithubStars, starCountContext)
- Update agent/cursor/windsurf rules files

Disable paid plan check for self-hosted deployments ... b58bdc8d06

`plan-supports-members?` now always returns true, bypassing the Stripe
subscription type check. Self-hosted instances don't have Stripe billing,
so gating features by subscription tier doesn't apply.

Replace marketing landing page with simple self-hosted homepage ... ca667b2bef

Shows a minimal page with a dashboard link and repo fork footer,
replacing the full InstantDB marketing site for self-hosted deployments.

Update CI credentials and add Mac runner verification script ... 95b8ada72f

Rename test DB/user/password to instantdb_* naming convention, and
remove inline migrate binary download in favor of a pre-installed
global migrate binary. Add verify-mac-runner Nushell script to check
SSH-accessible Mac Mini runner prerequisites (tools, postgres, disk).

Merge branch 'upstream-main' into self-host ebb6c432e6

GrocerPublishAgent added 1 commit 2026-04-12 01:21:55 +00:00

ci: run clojure jobs on macos runner, drop postgres service container ... ce71bef5c4

Postgres is pre-installed on the mac-mini host (verified via
self-hosting/verify-mac-runner), so the docker service container
is unnecessary. All jobs now target the self-hosted macos runner.

GrocerPublishAgent added 1 commit 2026-04-12 15:09:11 +00:00

ci: replace github-script with bash for Forgejo compatibility ... b5c4a75e9b

actions/github-script@v7 is not available on data.forgejo.org, so
replace change detection with plain git diff shell scripts. Also drop
timings fetch/upload/download steps — tests now split randomly across
the 5 parallel nodes.

GrocerPublishAgent added 1 commit 2026-04-13 16:06:51 +00:00

ci: replace actions/checkout with manual git fetch for Forgejo compatibility ... 32b007698e

actions/checkout@v4 fails on self-hosted Forgejo because it can't authenticate
to the local instance. Replace with manual git init/fetch/checkout using the
forgejo-action token, matching the pattern used in other project workflows.

GrocerPublishAgent force-pushed self-host from 32b007698e to 5a9a852f90 2026-04-13 16:17:13 +00:00 Compare

GrocerPublishAgent added 1 commit 2026-04-16 14:25:50 +00:00

ci: add Forgejo workflow to publish @peoplesgrocers/instantdb-qwik to npm ... 5204a5735d

Rename qwik package to @peoplesgrocers/instantdb-qwik v1.0.5 (matching
upstream @instantdb/core) and add a .forgejo/workflows publish workflow
triggered by qwik-v* tags.

GrocerPublishAgent added 1 commit 2026-04-17 13:57:43 +00:00

ci: build workspace dependencies before qwik package ... f29d8099fa

The qwik build failed because @instantdb/core and @instantdb/version
weren't built first. Use pnpm's ...pkg filter to include dependencies.

GrocerPublishAgent added 1 commit 2026-04-17 14:05:25 +00:00

ci: explicitly build @instantdb/version and @instantdb/core before qwik 3501170d32

GrocerPublishAgent added 1 commit 2026-04-17 14:12:17 +00:00

ci: fix npm auth by writing .npmrc with NPM_TOKEN for publish 31f8854691

GrocerPublishAgent added 1 commit 2026-04-17 14:30:22 +00:00

ci: simplify js.yml for self-hosted Forgejo runner ... 424427a717

Fix checkout failure on PRs where github.ref_name resolved to just
the PR number. Use git fetch origin + checkout by SHA like the
detect_changes job.

Remove GitHub Actions setup steps (pnpm/action-setup, setup-node,
actions/cache) since the self-hosted runner already has node/pnpm.

GrocerPublishAgent added 1 commit 2026-04-17 14:46:23 +00:00

ci: consolidate all workflows under .github/workflows/ ... a1bdb44743

Forgejo ignores .github/workflows/ when .forgejo/workflows/ exists.
Move publish-qwik.yml to .github/workflows/ and remove .forgejo/ so
all workflows live in one place and are picked up by the runner.

GrocerPublishAgent added 1 commit 2026-04-17 14:57:38 +00:00

ci: fix prettier formatting and add eslint.config.js for qwik-city ... 70cb0ad47d

Add ESLint flat config for the qwik-city sandbox (required by ESLint 9).
Run prettier --write to fix formatting issues in 12 files.

GrocerPublishAgent added 1 commit 2026-04-17 15:44:18 +00:00

ci: fix lint errors, add workflow_dispatch, simplify clojure.yml ... 8f586d46b9

Fix eslint errors in qwik-city sandbox (replace any with proper types).
Add workflow_dispatch to both js.yml and clojure.yml so builds can be
triggered manually via ci trigger. Simplify clojure.yml for self-hosted
runner (remove setup-java, setup-clojure, cache actions).

GrocerPublishAgent added 2 commits 2026-04-18 14:43:39 +00:00

chore: add qwik package README, license, and publish metadata ... a1665bda44

- Add README.md with install/usage docs for @peoplesgrocers/instantdb-qwik
- Add files and license fields to qwik package.json for npm publish
- Gitignore secrets.nuon
- Add SPDX license header to ci script and fix nullable column access

ci: fix clojure step timeout, fix qwik-city lint errors ... fdca98ff9e

Remove timeout-minutes: 1 from Prepare database step — Forgejo runner
was leaking it to the subsequent clojure -P step, killing dep download
after 60s. Add workflow_dispatch to js.yml.

Fix eslint errors in qwik-city sandbox: replace any with proper types,
remove unused Signal import.

GrocerPublishAgent added 1 commit 2026-04-18 14:47:52 +00:00

ci: fix Prepare database step hanging after completion ... d28db6594b

Redirect stdin from /dev/null for migrate and psql to prevent them
from holding open a file descriptor that the runner waits on. Replace
until loop with single pg_isready call.

GrocerPublishAgent added 1 commit 2026-04-18 15:00:31 +00:00

ci: run clojure tests on single node to avoid port conflicts ... 2f12e6e67e

Single self-hosted runner can't parallelize tests that bind to the
same ports. Keep the matrix/NODE_COUNT structure so it's easy to
scale back up with multiple runners.

GrocerPublishAgent added 1 commit 2026-04-19 17:32:11 +00:00

fix: rewrite workspace deps to upstream versions on qwik publish 463263ce87

GrocerPublishAgent added 1 commit 2026-04-19 18:37:30 +00:00

fix: qwik publish workflow bbf9677a4e

GrocerPublishAgent added 1 commit 2026-04-19 21:19:01 +00:00

ci: limit runner concurrency to one job at a time 8cecc26ab4

GrocerPublishAgent added 1 commit 2026-04-23 16:19:09 +00:00

feat: result-oriented connection state, publish forked core ... f601bf6501

Fork @instantdb/core as @peoplesgrocers/instantdb-core to support
reconnectNow() and richer connection status in the qwik package.

Reactor: store reconnect timer ID, add reconnectNow() method.
Qwik: useConnectionStatus returns ConnectionState discriminated union
with type/error/retryAt instead of bare status string.
Unified publish workflow deploys both packages from a single v* tag.

Some checks failed

Hide all checks

JS CI / detect_changes (pull_request) Successful in 16s

Details

JS CI / detect_changes (push) Successful in 15s

Details

Clojure CI / detect_changes (push) Successful in 35s

Details

Publish @peoplesgrocers/instantdb-core + instantdb-qwik / publish (push) Failing after 2m6s

Details

JS CI / Check formatting (pull_request) Failing after 2m23s

Details

JS CI / Build & Test packages (pull_request) Failing after 2m25s

Details

JS CI / Check formatting (push) Failing after 1m43s

Details

JS CI / Build & Test packages (push) Failing after 1m46s

Details

Clojure CI / lint (push) Successful in 22s

Details

Clojure CI / clj-check (0) (push) Failing after 3m8s

Details

Clojure CI / clj-build-check (push) Successful in 2m18s

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin self-host:self-host

git switch self-host

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch upstream-main

git merge --no-ff self-host

git switch self-host

git rebase upstream-main

git switch upstream-main

git merge --ff-only self-host

git switch self-host

git rebase upstream-main

git switch upstream-main

git merge --no-ff self-host

git switch upstream-main

git merge --squash self-host

git switch upstream-main

git merge --ff-only self-host

git switch upstream-main

git merge self-host

git push origin upstream-main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

comrades/self-hosted-instantdb!1

No description provided.