base on Web app to combine feeds from social networks and RSS and to help read more meaningful and deep content # Slow Reader Web app to combine feeds from social networks and RSS and to help read more meaningful and deep content. Right now, it is just a _prototype_. We plan to have features: - Combine all feeds (social media, RSS) in a single app. - Track how each subscription is useful for you. - Split subscriptions to **slow** (something useful, deep) and **fast-food** (fun and small). - Spend more time on slow content by blocking fast in the evening, etc. Pre-alpha prototype: [`dev.slowreader.app`](https://dev.slowreader.app/) **[↬ How to contribute and join the team](./CONTRIBUTING.md)** To ask any question: **[h+h lab Discord](https://discord.gg/TyFTp6mAZT)** --- - [License](#license) - [Principles](#principles) - [Local-First: Clients Do All the Job](#local-first-clients-do-all-the-job) - [Zero-Knowledge Synchronization](#zero-knowledge-synchronization) - [Event-Sourcing and CRDT](#event-sourcing-and-crdt) - [Client Core: Reusing All Logic Between Different Platforms](#client-core-reusing-all-logic-between-different-platforms) - [Project Structure](#project-structure) - [Tools](#tools) - [Scripts](#scripts) - [Synchronization Protocol](#synchronization-protocol) - [Client Storage](#client-storage) - [Test Strategy](#test-strategy) - [Visual Language](#visual-language) - [Dependencies](#dependencies) - [Guides](#guides) ## License Slow Reader is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License (version 3 or any later). For details, see the [`LICENSE.md`](./LICENSE.md) file in this directory. ## Principles ### Local-First: Clients Do All the Job Local-first means the client stores all data locally and does most of the job. Even if we decide to close the cloud, you can still use the Slow Reader client. In our case, it means that: - Client stores all feeds and posts. You can read posts offline. - Client checks feeds for new posts. - You need the cloud to sync data between clients. [Local-first manifest](https://www.inkandswitch.com/local-first/) tells more philosophy behind that idea. ### Zero-Knowledge Synchronization Clients use end-to-end encryption during cloud sync. It means the cloud can’t know what feeds you are reading or what posts you like. You can check [encryption source code](https://github.com/logux/client/blob/main/encrypt-actions/index.js). We have a proxy to make HTTP requests to other servers from the website. But this proxy is only for development, bypassing government censorship, or the initial test of the app. The web client should mostly use an upcoming web extension to bypass the CORS limit. Upcoming native clients will use direct HTTP requests since they don’t have a CORS limitation. ### Event-Sourcing and CRDT The source of truth in the client is a list of changes (action log). An action is a JSON object like: ```json { "type": "feeds/changed", "id": "kc4VfXpvw3vZZBu_ugGlC", "fields": { "reading": "slow" } } ``` To render the UI, the client reduces actions from the log into the state (objects of feeds, posts, etc.). We store the state cache in [Nano Stores](https://github.com/nanostores/nanostores). The log simplifies synchronization. We just need to track the last synchronized action and send all actions after that one. We use [Logux](https://logux.org/) to work with log and synchronization. ### Client Core: Reusing All Logic Between Different Platforms For now, we have only a web client. But we want to have native clients for different platforms. To make client porting easier, we separate core logic and UI. Core logic is the same for every client. The client just needs to bind this logic to the UI using native components. We write logic in TypeScript as smart stores in the [Nano Stores](https://github.com/nanostores/nanostores) state manager. The client needs to subscribe to stores and render UI according to the store. We try to move to the store as much as possible: app routing, validations, and UI helpers. The client should be as thin as possible. The ideal client is just a UI renderer. Core depends on the platform environment (like storage to store settings). Before using any store, the client must call [`setEnvironment()`](./core/environment.ts) to define how the core should interact with the platform. ## Project Structure Slow Reader is a local-first app. Clients do most of the work, and the server just syncs data between users’ devices (with end-to-end encryption). - **Clients.** - [`core/`](./core/): client’s logic and i18n translations. Clients for specific platforms is just a UI around this core to simplify porting - See **[`core/README.md`](./core/README.md)** for core architecture. - [`web/`](./web/): the client to be run in the browser. Both for desktop and mobile. - See **[`web/README.md`](./web/README.md)** for web client architecture. - [`server/`](./server/): a small server that syncs data between users’ devices. - [`proxy/`](./proxy/): HTTP proxy server to bypass censorship or to try web clients before they install the upcoming extensions (to bypass the CORS limit of the web apps). - [`extension/`](./extension/): browser’s extension to avoid CORS limits in web client. - [`api/`](./api/): types and constants shared between clients and server. - [`docs/`](./docs/): guides for developers. - [`scripts/`](./scripts/): scripts to test project and configure Google Cloud. Check the script’s descriptions for further details. - [`loader-tests/`](./loader-tests/): integration tests for each social network or news format. - [`.devcontainer`](./.devcontainer/): `Dockerfile` and configs to run project in Docker/Podman image on developer’s machine. It increases security (malicious dependency will not have access to the whole machine) and simplify onboarding. We have configs for Docker and [Podman](https://podman.io) (more secure version of Docker). - [`.github/`](./.github/): scripts to test projects on CI. - [`.husky/`](./.husky/): scripts to call on `git commit` command to avoid popular errors. - [`.vscode/`](./.vscode/): VS Code settings to reduce code format errors for new contributors. We are using [pnpm monorepo](https://pnpm.io/workspaces). Each project has its dependencies, tools, and configs. Read `README.md` in each project for project’s files and architecture. ## Tools Global development tools: - [Dev Container](https://containers.dev) to use the same environment for all developers and isolate project from developer’s machine. - [Prettier](./.prettierrc) to use the same code style formatting. - [TypeScript](./tsconfig.json) for strict type checking. - [ESLint](./eslint.config.js) to check for popular mistakes in JavaScript. - [remark](./.remarkrc) to find mistakes in `.md` files. Each project has its own tools, too. ## Scripts - `pnpm test`: run all tests. - `pnpm offline`: run all tests expect which need Internet. It is useful for developing in airplane/train. - `pnpm start`: run proxy and web client development server. - `pnpm format`: fix code style in all files. - `pnpm clean`: remove all temporary files. - `pnpm check-opml`: test loaders with user’s OPML RSS export. - `pnpm test-loaders`: test loaders with different blogging platforms. - `pnpm update-env`: check for Node.js and pnpm updates. We use pnpm feature to run scripts in parallel, having scripts like `test:types` and `test:audit`. Then, we run all scripts in all projects by `test:*` prefix. ## Synchronization Protocol We use [Logux WebSocket protocol](https://logux.org/protocols/ws/spec/) to synchronize actions between clients and server. Clients keep a list of changes (action log) as the source of truth and then send new actions to the server. The server then sends all new actions to other clients. The server doesn’t see those actions because clients encrypt them before sending and decrypt them upon receiving. The server sees only actions like: ```js // Add encrypted action to the server log { "type": "0", "d": "encrypted data", "iv": "random vector used together with password in encryption" } ``` ```js // Remove action from the server log { "type": "0/clean", "id": "action ID" } ``` ## Client Storage The clients store a list of changes (action log). During the start, the client reduces all necessary actions from the log to the [Logux SyncMap stores](https://logux.org/web-api/#globals-syncmaptemplate). For simple things like client local settings, we use [Nano Store Persistent](https://github.com/nanostores/persistent). The web client uses IndexedDB to store log and `localStorage` for the client’s settings. ## Test Strategy If any mistake happens a few times, we should add an automatic tool to **prevent mistakes** in the future. Possible strategies: 1. Types. 2. [Scripts](./scripts/), custom ESLint or Stylelint plugins. 3. Unit-tests. 4. [Pull request checklist](./docs/pull_request_template.md). Any **code-style** rule should be implemented as a `pre-commit` hook or linter’s rule. Types should try to use precise **types** and explain data relations with them: ```diff - { type: string, error?: Error } + { type: 'ok' } | { type: 'error', error: Error } ``` We are using unit tests for **client core**. We mock network requests and the platform environment, but emulate user interaction and test the composition of all stores. For the platform’s clients, we mostly use **visual tests**. But they could be complex and test the whole pages with mocking core’s stores. ## Visual Language We prefer the platform native look and feel where possible. Where not possible, we use old-style 3D with rich visual feedback and a z-axis. The slow mode should always use a yellow newspaper-like background (on color screens). We are using [Material Design Icons](https://pictogrammers.com/library/mdi/) icons. We prefer [natural non-linear shadows](https://shadows.brumm.af) for shadows more than 3 pixels. On desktops, we care not only about mouse UX but also about keyboard UX. Our keyboard UX rules: - Create a path: what keys can the user press to do some action? Try to make the path shorter. - Make hotkeys and non-standard keys visible to the user. - Think about focus. If the user starts to interact with the keyboard, move the focus to the next control. - <kbd>Esc</kbd> should work in as many cases as possible. - Don’t use only <kbd>Tab</kbd> to navigate. Mix it with arrows and hotkeys for list items. ## Dependencies How we choose dependencies: 1. Always checking alternatives from npm search, not just take the most popular one. 2. By project activity looking at their repository/issues/PR. 3. By JS bundle size for web client dependency. 4. By `node_modules` size and number of subdependencies. You can use [pkg-size.dev](https://pkg-size.dev) to get bundle, `node_modules`, and subdependencies. After adding a web client dependency, do not forget to call `cd web && pnpm size` to check the real size of dependency in our JS bundle. We put to `dependencies` only dependencies we need for production deploy. All other dependencies you should put to `devDependencies`. During production deploy we will use `pnpm install --prod` to reduce security risks of having malicious code in some dependency. We are using in `package.json` `1.0.0` version requirement and not `^1.0.0` to not get unexpected dependencies updates (at least, direct dependencies) if we will break the pnpm lock file. The `./scripts/check-versions.ts` in `pre-commit` hook will check that you do not forget this rule. To update specific dependency use: ```sh pnpm update DEPENDENCY ``` To update all dependencies: ```sh pnpm update --interactive --latest -r --include-workspace-root pnpm update -r --include-workspace-root ``` We can update all dependencies at least once per week. ## Guides - [How to Add New Page to Web Client?](./docs/new_page.md) ", Assign "at most 3 tags" to the expected json: {"id":"8794","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"