Felitendo
2c0356317b
`prisma db push` is how both containers reconcile the database at start, and it stops the moment a change could destroy data — a dropped column, a dropped table, a unique index over rows it has not seen. It stops for the whole push, so the reaction table and the `upvotes` column left a deployment restart- looping on "Use the --accept-data-loss flag", with the identity columns and the download token stuck behind them. `prisma/pre-push.sql` is every one of those changes written out in full and applied first, so the push that follows only ever sees work it considers safe. It is idempotent, a no-op on a fresh database, and it carries the whole series since local IPA storage, so a deployment that skipped four releases catches up in one boot. Both containers run it; an advisory lock makes the second one wait rather than race the first. Not `--accept-data-loss`: that flag would wave through the next drop nobody meant to make. A destructive change stays something somebody wrote down.
317 lines
10 KiB
Markdown
317 lines
10 KiB
Markdown
# Contributing to FTStore Admin
|
|
|
|
This covers `web/` only — the Next.js dashboard and feed server. The iOS app
|
|
lives in [`ios/`](../ios) and has its own guide in
|
|
[CONTRIBUTING.md](../CONTRIBUTING.md) at the repo root. Every command below is
|
|
run from `web/`.
|
|
|
|
## Prerequisites
|
|
|
|
- **Node.js** 20+
|
|
- **PostgreSQL** 16+ (or Docker)
|
|
- **npm** 10+
|
|
|
|
Optional (for full pipeline):
|
|
- Telegram API credentials ([my.telegram.org](https://my.telegram.org))
|
|
- GitHub personal access token with `repo` scope
|
|
|
|
---
|
|
|
|
## Running as a User (Docker)
|
|
|
|
The fastest way to get everything running.
|
|
|
|
### 1. Clone and configure
|
|
|
|
```bash
|
|
git clone https://github.com/FTStuff/FTStore.git
|
|
cd FTStore/web
|
|
cp .env.example .env
|
|
```
|
|
|
|
Edit `.env` with your credentials:
|
|
|
|
```env
|
|
TELEGRAM_API_ID=12345
|
|
TELEGRAM_API_HASH=abc123...
|
|
TELEGRAM_PHONE=+1234567890
|
|
TELEGRAM_CHANNELS=channel1,channel2
|
|
STORAGE_DIR=/app/storage
|
|
```
|
|
|
|
### 2. Start all services
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
This starts three containers:
|
|
|
|
| Service | Port | Description |
|
|
|---------|------|-------------|
|
|
| `db` | 5432 | PostgreSQL database |
|
|
| `app` | 3000 | Next.js dashboard |
|
|
| `worker` | — | Telegram scanner (background) |
|
|
|
|
The app container runs database migrations automatically on startup.
|
|
|
|
### 3. Open the dashboard
|
|
|
|
Visit [http://localhost:3000](http://localhost:3000). You'll land on the Dashboard page.
|
|
|
|
### 4. Seed default settings (first run)
|
|
|
|
```bash
|
|
docker compose exec app npx prisma db seed
|
|
```
|
|
|
|
### 5. Volume Permissions
|
|
|
|
Each container runs as a specific non-root user. If you encounter permission errors, fix ownership on the Docker volume data:
|
|
|
|
| Volume | Owner (UID:GID) | Error symptom |
|
|
|--------|-----------------|---------------|
|
|
| `tdlib-data` | 1001:1001 | `EACCES: permission denied, mkdir '/app/tdlib-data/db'` |
|
|
| `ipa-storage` | 1001:1001 | `Storage directory /app/storage is not writable` at startup |
|
|
| `temp-downloads` | 1001:1001 | `EACCES: permission denied` on `/tmp/ftstore` |
|
|
| `pgdata` | 70:70 | `FATAL: could not open file "global/pg_filenode.map": Permission denied` |
|
|
|
|
For named volumes:
|
|
|
|
```bash
|
|
# Fix ownership
|
|
sudo chown -R 1001:1001 /var/lib/docker/volumes/ftstore_tdlib-data/_data
|
|
sudo chown -R 1001:1001 /var/lib/docker/volumes/ftstore_ipa-storage/_data
|
|
sudo chown -R 1001:1001 /var/lib/docker/volumes/ftstore_temp-downloads/_data
|
|
sudo chown -R 70:70 /var/lib/docker/volumes/ftstore_pgdata/_data
|
|
```
|
|
|
|
If you're using bind mounts (e.g. at `/opt/docker/ftstore`):
|
|
|
|
```bash
|
|
sudo chown -R 1001:1001 /opt/docker/ftstore/tdlib-data
|
|
sudo chown -R 1001:1001 /opt/docker/ftstore/storage
|
|
sudo chown -R 1001:1001 /opt/docker/ftstore/temp-downloads
|
|
sudo chown -R 70:70 /opt/docker/ftstore/pgdata
|
|
```
|
|
|
|
`ipa-storage` must be the **same** volume (or host path) in the app and worker
|
|
services: the worker writes what the scraper downloads, the app serves it and
|
|
writes dashboard uploads.
|
|
|
|
Alternatively, you can remove the volumes and let them be recreated with correct permissions (⚠️ deletes database and session data):
|
|
|
|
```bash
|
|
docker compose down
|
|
docker volume rm ftstore_tdlib-data ftstore_ipa-storage ftstore_temp-downloads ftstore_pgdata
|
|
docker compose up -d
|
|
```
|
|
|
|
### 6. Updating / Redeploying
|
|
|
|
The Telegram session is preserved across redeployments. Both services use `stop_grace_period: 30s` so TDLib can close its database cleanly, and the worker retries connecting with backoff after a restart.
|
|
|
|
```bash
|
|
docker compose pull # pull latest images
|
|
docker compose up -d # recreate containers (session preserved)
|
|
```
|
|
|
|
> **Important:** Do not use `docker compose down -v` unless you want to delete the Telegram session and database. Use plain `docker compose down` to keep volume data.
|
|
|
|
### 7. Stopping
|
|
|
|
```bash
|
|
docker compose down # stop containers (keeps data)
|
|
docker compose down -v # stop and delete all data
|
|
```
|
|
|
|
---
|
|
|
|
## Running as a Developer
|
|
|
|
### 1. Clone and install
|
|
|
|
```bash
|
|
git clone https://github.com/FTStuff/FTStore.git
|
|
cd FTStore/web
|
|
npm install
|
|
```
|
|
|
|
### 2. Set up the database
|
|
|
|
Start a local PostgreSQL instance. With Docker:
|
|
|
|
```bash
|
|
docker run -d --name ftstore-db \
|
|
-e POSTGRES_USER=ftstore \
|
|
-e POSTGRES_PASSWORD=ftstore \
|
|
-e POSTGRES_DB=ftstore \
|
|
-p 5432:5432 \
|
|
postgres:16-alpine
|
|
```
|
|
|
|
### 3. Configure environment
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
```
|
|
|
|
Set at minimum:
|
|
|
|
```env
|
|
DATABASE_URL="postgresql://ftstore:ftstore@localhost:5432/ftstore?schema=public"
|
|
```
|
|
|
|
Add Telegram/GitHub credentials if you want to test the full pipeline.
|
|
|
|
### 4. Initialize the database
|
|
|
|
```bash
|
|
npx prisma generate # generate the Prisma client
|
|
npx prisma migrate dev # create tables and apply migrations
|
|
npm run db:seed # populate default settings
|
|
```
|
|
|
|
If you're iterating on the schema without needing migrations:
|
|
|
|
```bash
|
|
npx prisma db push # sync schema to DB (no migration files)
|
|
```
|
|
|
|
### 5. Start the dev server
|
|
|
|
```bash
|
|
npm run dev
|
|
```
|
|
|
|
Open [http://localhost:3000](http://localhost:3000).
|
|
|
|
### 6. Start the worker (optional)
|
|
|
|
The worker requires valid Telegram credentials and TDLib. In a separate terminal:
|
|
|
|
```bash
|
|
npm run worker
|
|
```
|
|
|
|
This connects to Telegram, scans configured channels, and processes the download queue.
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
src/
|
|
├── middleware.ts # Gates /admin, the feed and downloads; everything
|
|
│ # else is answered as a dead host
|
|
├── app/
|
|
│ ├── admin/ # The dashboard — ftstore.app/admin
|
|
│ │ ├── (auth)/ # login, register
|
|
│ │ └── (dashboard)/ # dashboard, metrics, library, queue, logs, settings
|
|
│ ├── api/ # REST endpoints (session-gated, except auth/health)
|
|
│ ├── ftstore.json/ # The AltStore feed the iOS app reads
|
|
│ ├── d/ # /d/<token>/<name>.ipa — public IPA downloads
|
|
│ └── media/icons/ # Tweak icons referenced by the feed
|
|
├── components/
|
|
│ ├── ui/ # shadcn/ui primitives
|
|
│ ├── layout/ # Sidebar, header, page container
|
|
│ └── ... # Feature-specific components
|
|
├── lib/
|
|
│ ├── ipa/ # IPA extraction and analysis
|
|
│ ├── telegram/ # TDLib client, scanner, downloader
|
|
│ ├── github/ # Inbound scanning of third-party release sources
|
|
│ ├── storage/ # Local IPA/icon store, retention, quota, HTTP serving
|
|
│ ├── appstore/ # iTunes lookup with caching
|
|
│ ├── json/ # Feed generation and serving
|
|
│ ├── pipeline/ # Queue, orchestrator, ingest
|
|
│ └── cleanup/ # Log and cache cleanup
|
|
├── hooks/ # React hooks (polling, etc.)
|
|
└── types/ # Shared TypeScript types
|
|
worker/
|
|
└── index.ts # Standalone Telegram scanner process
|
|
```
|
|
|
|
---
|
|
|
|
## Common Tasks
|
|
|
|
### Adding a shadcn/ui component
|
|
|
|
```bash
|
|
npx shadcn@latest add <component-name>
|
|
```
|
|
|
|
### Updating the database schema
|
|
|
|
1. Edit `prisma/schema.prisma`
|
|
2. Run `npx prisma migrate dev --name describe-your-change`
|
|
3. The Prisma client regenerates automatically
|
|
4. If the change drops a column or a table, or adds a unique index over rows
|
|
that already exist, add it to `prisma/pre-push.sql` as well
|
|
|
|
Step 4 is what keeps deployments booting. Production reconciles with
|
|
`prisma db push`, which refuses changes like those, and refuses the *whole*
|
|
push along with them; `pre-push.sql` applies them explicitly before it runs.
|
|
See [Upgrading an existing install](README.md#upgrading-an-existing-install).
|
|
|
|
### Running a production build locally
|
|
|
|
```bash
|
|
npm run build
|
|
npm start
|
|
```
|
|
|
|
### Regenerating JSON feeds manually
|
|
|
|
Hit the API endpoint:
|
|
|
|
```bash
|
|
curl -X POST http://localhost:3000/api/actions/generate-json
|
|
```
|
|
|
|
Or use the Settings page > Admin Actions > Generate JSON.
|
|
|
|
### Triggering a channel scan
|
|
|
|
```bash
|
|
curl -X POST http://localhost:3000/api/actions/scan
|
|
```
|
|
|
|
### Checking system health
|
|
|
|
```bash
|
|
curl http://localhost:3000/api/health
|
|
```
|
|
|
|
---
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Required | Default | Description |
|
|
|----------|----------|---------|-------------|
|
|
| `DATABASE_URL` | Yes | — | PostgreSQL connection string |
|
|
| `TELEGRAM_API_ID` | For worker | — | Telegram API ID |
|
|
| `TELEGRAM_API_HASH` | For worker | — | Telegram API hash |
|
|
| `TELEGRAM_PHONE` | For worker | — | Phone number for Telegram auth |
|
|
| `TELEGRAM_CHANNELS` | For worker | — | Comma-separated channel usernames |
|
|
| `GITHUB_TOKEN` | No | — | Fallback PAT for the FTStore Manager proxy only |
|
|
| `APPSTORE_COUNTRY` | No | `us` | App Store region for lookups |
|
|
| `SCAN_INTERVAL_MINUTES` | No | `30` | How often the worker scans channels |
|
|
| `JSON_REGEN_INTERVAL_MINUTES` | No | `60` | How often JSON feeds regenerate |
|
|
| `CLEANUP_INTERVAL_HOURS` | No | `24` | How often old versions and orphaned files are cleaned |
|
|
| `MAX_VERSIONS_PER_APP` | No | `5` | Versions kept per app in JSON feeds |
|
|
| `TEMP_DIR` | No | `/tmp/ftstore` | Temporary download directory |
|
|
| `STORAGE_DIR` | No | `/app/storage` | Where IPAs and tweak icons are stored |
|
|
| `MAX_STORAGE_GB` | No | `0` | Storage cap in GiB; the scraper stops there (0 = unlimited) |
|
|
| `LOG_RETENTION_DAYS` | No | `30` | Days before old logs are deleted |
|
|
|
|
---
|
|
|
|
## Architecture Notes
|
|
|
|
- The **worker** runs as a separate process because TDLib maintains a persistent stateful connection to Telegram that doesn't fit the Next.js request lifecycle.
|
|
- The worker and web app communicate exclusively through the **shared PostgreSQL database** — there is no direct IPC or message queue.
|
|
- **IPA files, the JSON feed and tweak icons are all served by this server.** The feed is rendered from the database per request (`src/lib/json/serve-feed.ts`) rather than published anywhere; downloads go through `/d/<token>/<name>.ipa`, guarded by a random capability token and, optionally, the FTStore User-Agent.
|
|
- GitHub is only used for *inbound* scraping of third-party release repositories (`src/lib/github/`, each source carrying its own token) and for the FTStore Manager proxy.
|
|
- The storage cap is enforced as **queue policy**, not in the ingest step: the worker stops claiming work while the cap is reached, so manual dashboard uploads still succeed.
|