LeenkxTeam/Kmake
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Upload Kmake

Browse Source
This commit is contained in:
Gorochu
2026-05-26 23:36:42 -07:00
parent ba051b2f74
commit 555ec72358
41615 changed files with 13344630 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

391
doc/contributing/maintaining/maintaining-V8.md Normal file
View File

@ -0,0 +1,391 @@
# Maintaining V8 in Node.js
## Background
V8 follows the Chromium release schedule. The support horizon for Chromium is
different compared to the support horizon for Node.js. As a result, Node.js
needs to support multiple versions of V8 longer than what upstream needs
to support. V8 branches in Node.js lack of an official maintenance process due
to a missing LTS supported branch.
This document attempts to outline the current maintenance processes, proposes
a workflow for maintaining the V8 branches in both Node.js LTS and current
releases, and discusses how the Node.js and V8 teams at Google can help.
## V8 release schedule
V8 and Chromium follow a
[roughly 4-week release cadence][ChromiumReleaseCalendar]. At any given time
there are several V8 branches that are **active**, see the
[V8 release process](https://v8.dev/docs/release-process). All older branches
are abandoned and are not maintained by the V8 team.
### V8 merge process overview
The process for backporting bug fixes to active branches is officially
documented [on the V8 wiki][V8MergingPatching]. The summary of the process is:
* V8 only supports active branches. There is no testing done on any branches
older than the current stable/beta/master.
* A fix needing backport is tagged w/ _merge-request-x.x_ tag. This can be done
by anyone interested in getting the fix backported. Issues with this tag are
reviewed by the V8 team regularly as candidates for backporting.
* Fixes need some 'baking time' before they can be approved for backporting.
This means waiting a few days to ensure that no issues are detected on the
canary/beta builds.
* Once ready, the issue is tagged w/ _merge-approved-x.x_ and one can do the
actual merge by using the scripts on the [wiki page][V8MergingPatching].
* Merge requests to an abandoned branch will be rejected.
* Only bug fixes are accepted for backporting.
## Node.js support requirements
At any given time Node.js needs to be maintaining a few different V8 branches
for the various Current, LTS, and nightly releases.
The versions of V8 used in Node.js may have already been
abandoned by upstream V8. However, Node.js needs to continue supporting
these branches for many months (Current branches) or several
years (LTS branches).
## Maintenance process
Once a bug in Node.js has been identified to be caused by V8, the first step is
to identify the versions of Node.js and V8 affected. The bug may be present in
multiple different locations, each of which follows a slightly different
process.
* Unfixed bugs. The bug exists in the V8 master branch.
* Fixed, but needs backport. The bug may need porting to one or more branches.
* Backporting to active branches.
* Backporting to abandoned branches.
* Backports identified by the V8 team. Bugs identified by upstream V8 that we
haven't encountered in Node.js yet.
### Unfixed upstream bugs
If the bug can be reproduced on the [Node.js `canary` branch][], Chromium
canary, or V8 tip-of-tree, and the test case is valid, then the bug needs to be
fixed upstream first.
* Start by opening a bug upstream using [this template][V8TemplateUpstreamBug].
* Make sure to include a link to the corresponding Node.js issue
(if one exists).
* If the fix is simple enough, you may fix it yourself;
[contributions][V8Contributing] are welcome.
* V8's build waterfall tests your change.
* Once the bug is fixed it may still need backporting, if it exists in other V8
branches that are still active or are branches that Node.js cares about.
Follow the process for backporting below.
### Backporting to active branches
If the bug exists in any of the active V8 branches, we may need to get the fix
backported. At any given time, there are [two active branches][V8ActiveBranches]
(beta and stable) in addition to master. The following steps are needed to
backport the fix:
* Identify which version of V8 the bug was fixed in.
* Identify if any active V8 branches still contain the bug:
* A tracking bug is needed to request a backport.
* If there isn't already a V8 bug tracking the fix, open a new merge request
bug using this [Node.js specific template][V8TemplateMergeRequest].
* If a bug already exists
* Add a reference to the GitHub issue.
* Attach _merge-request-x.x_ labels to the bug for any active branches
that still contain the bug.
* Once the merge has been approved, it should be merged using the
[merge script documented in the V8 wiki][V8MergingPatching]. Merging requires
commit access to the V8 repository. If you don't have commit access you can
indicate someone on the V8 team can do the merge for you.
* It is possible that the merge request may not get approved, for example if it
is considered to be a feature or otherwise too risky for V8 stable. In such
cases we float the patch on the Node.js side. See the process on 'Backporting
to Abandoned branches'.
* Once the fix has been merged upstream, it can be picked up during an update of
the V8 branch (see below).
### Backporting to abandoned branches
Abandoned V8 branches are supported in the Node.js repository. The fix needs
to be cherry-picked in the Node.js repository and V8-CI must test the change.
As an example for how to backport changes, consider the bug
[RegExp show inconsistent result with other browsers](https://crbug.com/v8/5199).
From the bug we can see that it was merged by V8 into 5.2 and 5.3, and not into
V8 5.1 (since it was already abandoned). Since Node.js `v6.x` uses V8 5.1, the
fix needed to be backported.
#### Backporting with `git-node` (recommended)
You can use [`git-node`][] to help you backport patches. This removes
some manual steps and is recommended.
Here are the steps for the bug mentioned above:
1. Install `git-node` by installing [`@node-core/utils`][].
2. Install the prerequisites for [`git-node-v8`][].
3. Find the commit hash linked-to in the issue (in this case a51f429).
4. Checkout a branch off the appropriate _vY.x-staging_ branch (e.g.
_v6.x-staging_ to fix an issue in V8 5.1).
5. Run `git node v8 backport a51f429`.
6. If there are conflicts, `git-node` will wait for you to resolve them:
```console
$ git node v8 backport a51f429
✔ Update local V8 clone
❯ V8 commit backport
✔ Get current V8 version
✔ Generate patches
❯ Apply and commit patches to deps/v8
❯ Commit a51f429772d1
⠏ Apply patch
◼ Increment embedder version number
◼ Commit patch
? Resolve merge conflicts and enter 'RESOLVED' ‣
```
Resolve conflicts, stage the files (you may need to open another terminal or use
a GUI git client), then return to the terminal running `git-node`, type
`RESOLVED`, and hit <kbd>Enter</kbd>.
7. After you resolve conflicts (or if there are no conflicts), the
output should look like this:
```console
$ git node v8 backport a51f429
✔ Update local V8 clone
✔ V8 commit backport
```
8. Open a PR against the v6.x-staging branch in the Node.js repository.
Launch the normal and [V8 CI][] using the Node.js CI system. We only
needed to backport to v6.x as the other LTS branches weren't affected
by this bug.
See [`git-node-v8-backport`][] for more documentation and additional options.
#### Backporting manually
* For each abandoned V8 branch corresponding to an LTS branch that is affected
by the bug:
* Checkout a branch off the appropriate _vY.x-staging_ branch (e.g.
_v6.x-staging_ to fix an issue in V8 5.1).
* Cherry-pick the commit(s) from the V8 repository.
* Increase the `v8_embedder_string` number in `common.gypi`.
* In some cases the patch may require extra effort to merge in case V8 has
changed substantially. For important issues, we may be able to lean on the
V8 team to get help with reimplementing the patch.
* Open a cherry-pick pull request on `nodejs/node` targeting the
_vY.x-staging_ branch and notify the `@nodejs/v8` team.
* Run the Node.js [V8 CI][] in addition to the [Node.js CI][].
The CI uses the `test-v8` target in the `Makefile`, which uses
`tools/make-v8.sh` to reconstruct a git tree in the `deps/v8` directory to
run V8 tests.[^1]
Here are the steps for the bug mentioned above:
* Download and apply the commit linked-to in the issue (in this case a51f429):
```bash
curl -L https://github.com/v8/v8/commit/a51f429.patch | git am -3 --directory=deps/v8
```
If the branches have diverged significantly, this may not apply cleanly. It
may help to try to cherry-pick the merge to the oldest branch that was done
upstream in V8. In this example, this would be the patch from the merge to
5.2. The hope is that this would be closer to the V8 5.1, and has a better
chance of applying cleanly.
* Modify the commit message to match the format we use for V8 backports and
replace yourself as the author. `git commit --amend --reset-author`. You may
want to add extra description if necessary to indicate the impact of the fix
on Node.js. In this case the original issue was descriptive enough. Example:
```console
deps: cherry-pick a51f429 from V8 upstream
Original commit message:
[regexp] Fix case-insensitive matching for one-byte subjects.
The bug occurs because we do not canonicalize character class ranges
before adding case equivalents. While adding case equivalents, we abort
early for one-byte subject strings, assuming that the ranges are sorted.
Which they are not.
R=marja@chromium.org
BUG=v8:5199
Review-Url: https://codereview.chromium.org/2159683002
Cr-Commit-Position: refs/heads/master@{#37833}
Refs: https://github.com/v8/v8/commit/a51f429772d1e796744244128c9feeab4c26a854
PR-URL: https://github.com/nodejs/node/pull/7833
```
* Increase the `v8_embedder_string` number in `common.gypi`.
* Open a PR against the `v6.x-staging` branch in the Node.js repository. Launch
the normal and [V8 CI][] using the Node.js CI system. We only needed to
backport to `v6.x` as the other LTS branches weren't affected by this bug.
### Backports identified by the V8 team
For bugs found through the browser or other channels, the V8 team marks bugs
that might be applicable to the abandoned branches in use by Node.js. This is
done through manual tagging by the V8 team and through an automated process that
tags any fix that gets backported to the stable branch (as it is likely
candidate for backporting further).
Such fixes are tagged with the following labels in the V8 issue tracker:
* `NodeJS-Backport-Review` ([V8][NodeJS-Backport-Review-V8],
[Chromium][NodeJS-Backport-Review-Chromium]): to be reviewed if this is
applicable to abandoned branches in use by Node.js. This list if regularly
reviewed by the Node.js team at Google to determine applicability to Node.js.
* `NodeJS-Backport-Approved` ([V8][NodeJS-Backport-Approved-V8],
[Chromium][NodeJS-Backport-Approved-Chromium]): marks bugs that are deemed
relevant to Node.js and should be backported.
* `NodeJS-Backport-Done` ([V8][NodeJS-Backport-Done-V8],
[Chromium][NodeJS-Backport-Done-Chromium]): Backport for Node.js has been
performed already.
* `NodeJS-Backport-Rejected` ([V8][NodeJS-Backport-Rejected-V8],
[Chromium][NodeJS-Backport-Rejected-Chromium]): Backport for Node.js is not
desired.
The backlog of issues with such is regularly reviewed by the node-team at Google
to shepherd through the backport process. External contributors are welcome to
collaborate on the backport process as well. Some of the bugs may be
security issues and will not be visible to external collaborators.
## Updating V8
Node.js keeps a vendored copy of V8 inside of the deps/ directory. In addition,
Node.js may need to float patches that do not exist upstream. This means that
some care may need to be taken to update the vendored copy of V8.
V8 builds against the version of ICU supplied by Node.js,
see [maintaining-icu.md](./maintaining-icu.md) for special considerations.
Specifically, a V8 update may necessitate an ICU update.
### Minor updates (patch level)
Because there may be floating patches on the version of V8 in Node.js, it is
safest to apply the patch level updates as a patch. For example, imagine that
upstream V8 is at 5.0.71.47 and Node.js is at 5.0.71.32. It would be best to
compute the diff between these tags on the V8 repository, and then apply that
patch on the copy of V8 in Node.js. This should preserve the patches/backports
that Node.js may be floating (or else cause a merge conflict).
#### Applying minor updates with `git-node` (recommended)
1. Install [`git-node`][] by installing [`@node-core/utils`][].
2. Install the prerequisites for [`git-node-v8`][].
3. Run `git node v8 minor` to apply a minor update.
See [`git-node-v8-minor`][] for more documentation and additional options.
#### Applying minor updates manually
The rough outline of the process is:
```bash
# Assuming your fork of Node.js is checked out in $NODE_DIR
# and you want to update the Node.js main branch.
# Find the current (OLD) version in
# $NODE_DIR/deps/v8/include/v8-version.h
cd $NODE_DIR
git checkout main
git merge --ff-only origin/main
git checkout -b V8_NEW_VERSION
curl -L https://github.com/v8/v8/compare/${V8_OLD_VERSION}...${V8_NEW_VERSION}.patch | git apply --directory=deps/v8
# You may want to amend the commit message to describe the nature of the update
```
V8 also keeps tags of the form _5.4-lkgr_ which point to the _Last Known Good
Revision_ from the 5.4 branch that can be useful in the update process above.
### Major updates
We upgrade the version of V8 in Node.js `main` whenever a V8 release goes stable
upstream, that is, whenever a new release of Chrome comes out.
Upgrading major versions would be much harder to do with the patch mechanism
above. A better strategy is to
1. Audit the current `main` branch and look at the patches that have been
floated since the last major V8 update.
2. Replace the copy of V8 in Node.js with a fresh checkout of the latest stable
V8 branch. Special care must be taken to recursively update the DEPS that V8
has a compile time dependency on (at the moment of this writing, these are
only trace\_event and gtest\_prod.h)
3. Reset the `v8_embedder_string` variable to "-node.0" in `common.gypi`.
4. Refloat (cherry-pick) all the patches from list computed in 1) as necessary.
Some of the patches may no longer be necessary.
To audit for floating patches:
```bash
git log --oneline deps/v8
```
To replace the copy of V8 in Node.js, use the [`git-node`][] tool. For example,
if you want to replace the copy of V8 in Node.js with the branch-head for V8 5.1
branch:
```bash
cd $NODE_DIR
git node v8 major --branch=5.1-lkgr
```
This should be followed up with manual refloating of all relevant patches.
## Proposal: Using a fork repository to track upstream V8
The fact that Node.js keeps a vendored, potentially edited copy of V8 in deps/
makes the above processes a bit complicated. An alternative proposal would be to
create a fork of V8 at `nodejs/v8` that would be used to maintain the V8
branches. This has several benefits:
* The process to update the version of V8 in Node.js could be automated to track
the tips of various V8 branches in `nodejs/v8`.
* It would simplify cherry-picking and porting of fixes between branches as the
version bumps in `v8-version.h` would happen as part of this update instead of
on every change.
* It would simplify the V8-CI and make it more automatable.
* The history of the V8 branch in `nodejs/v8` becomes purer and it would make it
easier to pull in the V8 team for help with reviewing.
* It would make it simpler to setup an automated build that tracks Node.js
`main` + V8 lkgr integration build.
This would require some tooling to:
* A script that would update the V8 in a specific Node.js branch with V8 from
upstream (dependent on branch abandoned vs. active).
* We need a script to bump V8 version numbers when a new version of V8 is
promoted from `nodejs/v8` to `nodejs/node`.
* Enabled the V8-CI build in Jenkins to build from the `nodejs/v8` fork.
[^1]: On macOS, the V8 tests require a full Xcode install, not just the "command
line tools" for Xcode.
[ChromiumReleaseCalendar]: https://www.chromium.org/developers/calendar
[Node.js CI]: https://ci.nodejs.org/job/node-test-pull-request/
[Node.js `canary` branch]: https://github.com/nodejs/node-v8/tree/canary
[NodeJS-Backport-Approved-Chromium]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3ANodeJS-Backport-Approved
[NodeJS-Backport-Approved-V8]: https://bugs.chromium.org/p/v8/issues/list?can=1&q=label%3ANodeJS-Backport-Approved
[NodeJS-Backport-Done-Chromium]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3ANodeJS-Backport-Done
[NodeJS-Backport-Done-V8]: https://bugs.chromium.org/p/v8/issues/list?can=1&q=label%3ANodeJS-Backport-Done
[NodeJS-Backport-Rejected-Chromium]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3ANodeJS-Backport-Rejected
[NodeJS-Backport-Rejected-V8]: https://bugs.chromium.org/p/v8/issues/list?can=1&q=label%3ANodeJS-Backport-Rejected
[NodeJS-Backport-Review-Chromium]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3ANodeJS-Backport-Review
[NodeJS-Backport-Review-V8]: https://bugs.chromium.org/p/v8/issues/list?can=1&q=label%3ANodeJS-Backport-Review
[V8 CI]: https://ci.nodejs.org/job/node-test-commit-v8-linux/
[V8ActiveBranches]: https://build.chromium.org/p/client.v8.branches/console
[V8Contributing]: https://v8.dev/docs/contribute
[V8MergingPatching]: https://v8.dev/docs/merge-patch
[V8TemplateMergeRequest]: https://bugs.chromium.org/p/v8/issues/entry?template=Node.js%20merge%20request
[V8TemplateUpstreamBug]: https://bugs.chromium.org/p/v8/issues/entry?template=Node.js%20upstream%20bug
[`@node-core/utils`]: https://github.com/nodejs/node-core-utils#Install
[`git-node-v8-backport`]: https://github.com/nodejs/node-core-utils/blob/main/docs/git-node.md#git-node-v8-backport-sha
[`git-node-v8-minor`]: https://github.com/nodejs/node-core-utils/blob/main/docs/git-node.md#git-node-v8-minor
[`git-node-v8`]: https://github.com/nodejs/node-core-utils/blob/HEAD/docs/git-node.md#git-node-v8
[`git-node`]: https://github.com/nodejs/node-core-utils/blob/HEAD/docs/git-node.md#git-node-v8

59
doc/contributing/maintaining/maintaining-cjs-module-lexer.md Normal file
View File

@ -0,0 +1,59 @@
# Maintaining cjs-module-lexer
The [cjs-module-lexer](https://github.com/nodejs/node/tree/HEAD/deps/cjs-module-lexer)
dependency is used within the Node.js ESM implementation to detect the
named exports of a CommonJS module.
It is used within
[`node:internal/modules/esm/translators`](https://github.com/nodejs/node/blob/HEAD/lib/internal/modules/esm/translators.js)
in which both `internal/deps/cjs-module-lexer/lexer` and
`internal/deps/cjs-module-lexer/dist/lexer` are required and used.
`internal/deps/cjs-module-lexer/lexer`
is a regular JavaScript implementation that is
used when WebAssembly is not available on a platform.
`internal/deps/cjs-module-lexer/dist/lexer` is a faster
implementation using WebAssembly which is generated from a
C based implementation. These two paths
resolve to the files in `deps/cjs-module-lexer` due to their
inclusion in the `deps_files` entry in
[node.gyp](https://github.com/nodejs/node/blob/main/node.gyp).
The two different versions of lexer.js are maintained in the
[nodejs/cjs-module-lexer][] project.
In order to update the Node.js dependencies to use to a newer version
of cjs-module-lexer, complete the following steps:
* Clone [nodejs/cjs-module-lexer][]
and check out the version that you want Node.js to use.
* Follow the WASM build steps outlined in
[wasm-build-steps](https://github.com/nodejs/cjs-module-lexer#wasm-build-steps).
This will generate the WASM based dist/lexer.js file.
* Preserving the same directory structure, copy the following files over
to `deps/cjs-module-lexer` directory where you have checked out Node.js.
```text
├── CHANGELOG.md
├── dist
│ ├── lexer.js
│ └── lexer.mjs
├── lexer.js
├── LICENSE
├── package.json
└── README.md
```
* Update the link to the cjs-module-lexer in the list at the end of
[doc/api/esm.md](../../api/esm.md)
to point to the updated version.
* Create a PR, adding the files in the deps/cjs-module-lexer that
were modified.
If updates are needed to cjs-module-lexer for Node.js, first PR
those updates into
[nodejs/cjs-module-lexer][],
request a release and then pull in the updated version once available.
[nodejs/cjs-module-lexer]: https://github.com/nodejs/cjs-module-lexer

438
doc/contributing/maintaining/maintaining-dependencies.md Normal file
View File

@ -0,0 +1,438 @@
# Maintaining Dependencies
Node.js depends on additional components beyond the Node.js code
itself. These dependencies provide both native and JavaScript code
and are built together with the code under the `src` and `lib`
directories to create the Node.js binaries.
All dependencies are located within the `deps` directory.
This a list of all the dependencies:
* [acorn][]
* [ada][]
* [amaro][]
* [base64][]
* [brotli][]
* [c-ares][]
* [cjs-module-lexer][]
* [corepack][]
* [googletest][]
* [histogram][]
* [icu-small][]
* [inspector\_protocol][inspector_protocol]
* [libuv][]
* [llhttp][]
* [minimatch][]
* [nghttp2][]
* [nghttp3][]
* [ngtcp2][]
* [npm][]
* [openssl][]
* [postject][]
* [simdjson][]
* [sqlite][]
* [undici][]
* [uvwasi][]
* [V8][]
* [zlib][]
* [zstd][]
Any code which meets one or more of these conditions should
be managed as a dependency:
* originates in an upstream project and is maintained
in that upstream project.
* is not built from the `preferred form of the work for
making modifications to it` (see
[GNU GPL v2, section 3.](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html)
when `make node` is run. A good example is
WASM code generated from C (the preferred form).
Typically generation is only supported on a subset of platforms, needs
additional tools, and is pre-built outside of the `make node`
step and then committed as a WASM binary in the directory
for the dependency under the `deps` directory.
By default all dependencies are bundled into the Node.js
binary, however, `configure` options should be available to
use an externalized version at runtime when:
* the dependency provides native code and is available as
a shared library in one or more of the common Node.js
distributions.
* the dependency provides JavaScript and is not built
from the `preferred form of the work for making modifications
to it` when `make node` is run.
Many distributions use externalized dependencies for one or
more of these reasons:
1. They have a requirement to build everything that they ship
from the `preferred form of the work for making
modifications to it`. This means that they need to
replace any pre-built components (for example WASM
binaries) with an equivalent that they have built.
2. They manage the dependency separately as it is used by
more applications than just Node.js. Linking against
a shared library allows them to manage updates and
CVE fixes against the library instead of having to
patch all of the individual applications.
3. They have a system wide configuration for the
dependency that all applications should respect.
## Supporting externalized dependencies with native code
Support for externalized dependencies with native code for which a
shared library is available can added by:
* adding options to `configure.py`. These are added to the
shared\_optgroup and include an options to:
* enable use of a shared library
* set the name of the shared library
* set the path to the directory with the includes for the
shared library
* set the path to where to find the shared library at
runtime
* add a call to configure\_library() to `configure.py` for the
library at the end of list of existing configure\_library() calls.
If there are additional libraries that are required it is
possible to list more than one with the `pkgname` option.
* in `node.gypi` guard the build for the dependency
with `node_shared_depname` so that it is only built if
the dependency is being bundled into Node.js itself. For example:
```text
[ 'node_shared_brotli=="false"', {
'dependencies': [ 'deps/brotli/brotli.gyp:brotli' ],
}],
```
## Supporting externalizable dependencies with JavaScript code
Support for an externalizable dependency with JavaScript code
can be added by:
* adding an entry to the `shareable_builtins` map in
`configure.py`. The path should correspond to the file
within the deps directory that is normally bundled into
Node.js. For example `deps/cjs-module-lexer/lexer.js`.
This will add a new option for building with that dependency
externalized. After adding the entry you can see
the new option by running `./configure --help`.
* adding a call to `AddExternalizedBuiltin` to the constructor
for BuiltinLoader in `src/node_builtins.cc` for the
dependency using the `NODE_SHARED_BUILTLIN` #define generated for
the dependency. After running `./configure` with the new
option you can find the #define in `config.gypi`. You can cut and
paste one of the existing entries and then update to match the
import name for the dependency and the #define generated.
* if the version of the dependency is reported in `process.versions`,
update `src/node_metadata.h` and `src/node_metadata.cc` so that the
version is not reported when the dependency is externalized.
Not reporting the version is better than incorrectly reporting
the version of the dependency bundled with Node.js, instead of the
version for the externalized dependency. Use one of the existing
externalized dependencies, like Undici, as an example of how to
update these files correctly. Make sure to run the tests with the
dependency externalized, as the tests will also need to be updated
to handle this properly.
## Supporting non-externalized dependencies with JavaScript code
If the dependency consists of JavaScript in the
`preferred form of the work for making modifications to it`, it
can be added as a non-externalizable dependency. In this case
simply add the path to the JavaScript file in the `deps_files`
list in the `node.gyp` file.
## Common approach for dependencies with WASM components
WASM components within dependencies are most often built
outside of the regular Node.js `make build` step. They also
require different tools.
It is important that the tools and their versions used to build
WASM components shipped within Node.js are well documented and
be available if needed to rebuild/update older Node.js versions.
In order to minimize the different number of tools and versions
used to build WASM components and to document and ensure future
availability, the project builds and maintains a common
[wasm-builder](https://github.com/nodejs/wasm-builder) container
that should be use to build WASM components in Node.js
dependencies.
The container provides a durable copy of the versions of the tools
used for a specific build which are under the control of the Node.js
project. In addition, the tools and verions are documented through metadata
within the container in the `/home/node/metadata directory`.
The available tools can be found by looking at the current version of the
[Dockerfile](https://github.com/nodejs/wasm-builder/blob/main/container-build-info/Dockerfile)
used to create the container.
If additional WASM tool are needed beyond those available in the
container, additions should be PR'd into the wasm-builder container.
Examples of using the container include:
* [build/wasm.js](https://github.com/nodejs/undici/blob/main/build/wasm.js) from undici
* [tools/build-wasm.js](https://github.com/nodejs/amaro/blob/main/tools/build-wasm.js) from amaro
In addition to using the container to build WASM components, the goal is also
for the WASM components and final files that are shipped with Node.js to be
built by the [dep-updaters](https://github.com/nodejs/node/tree/main/tools/dep_updaters)
that are run on a regular basis and that they use only the files available in the Node.js
repo for the dependency. For example, being able to rebuild the WASM and files that
we ship in Node.js using only the files in
[../deps/undici](https://github.com/nodejs/node/tree/main/deps/undici).
## Updating dependencies
Most dependencies are automatically updated by
[dependency-update-action][] that runs weekly.
However, it is possible to manually update a dependency by running
the corresponding script in `tools/update-deps`.
[OpenSSL](https://github.com/openssl/openssl) has its own update action:
[update-openssl-action][].
[npm-cli-bot](https://github.com/npm/cli/blob/latest/.github/workflows/create-node-pr.yml)
takes care of npm update, it is maintained by the npm team.
PRs for manual dependency updates should only be accepted if
the update cannot be generated by the automated tooling,
the reason is clearly documented and either the PR is
reviewed in detail or it is from an existing collaborator.
In general updates to dependencies should only be accepted
if they have already landed in the upstream. The TSC may
grant an exception on a case-by-case basis. This avoids
the project having to float patches for a long time and
ensures that tooling can generate updates automatically.
## Dependency list
### acorn
The [acorn](https://github.com/acornjs/acorn) dependency is a JavaScript parser.
[acorn-walk](https://github.com/acornjs/acorn/tree/master/acorn-walk) is
an abstract syntax tree walker for the ESTree format.
### ada
The [ada](https://github.com/ada-url/ada) dependency is a
fast and spec-compliant URL parser written in C++.
### amaro
The [amaro](https://www.npmjs.com/package/amaro) dependency is a wrapper around the
WebAssembly version of the SWC JavaScript/TypeScript parser.
### brotli
The [brotli](https://github.com/google/brotli) dependency is
used for the homonym generic-purpose lossless compression algorithm.
### c-ares
The [c-ares](https://github.com/c-ares/c-ares) is a C library
for asynchronous DNS requests.
### cjs-module-lexer
The [cjs-module-lexer](https://github.com/nodejs/node/tree/HEAD/deps/cjs-module-lexer)
dependency is used within the Node.js ESM implementation to detect the
named exports of a CommonJS module.
See [maintaining-cjs-module-lexer][] for more information.
### corepack
The [corepack](https://github.com/nodejs/corepack) dependency is a
zero-runtime-dependency Node.js script that acts as a bridge between
Node.js projects and the package managers they are intended to
be used with during development.
In practical terms, Corepack will let you use Yarn and pnpm without having to
install them - just like what currently happens with npm, which is shipped
by Node.js by default.
### googletest
The [googletest](https://github.com/google/googletest) dependency is Google’s
C++ testing and mocking framework.
### histogram
The [histogram](https://github.com/HdrHistogram/HdrHistogram_c) dependency is
a C port of High Dynamic Range (HDR) Histogram.
### ic
The [icu](http://site.icu-project.org) is widely used set of C/C++
and Java libraries providing Unicode and Globalization
support for software applications.
See [maintaining-icu][] for more information.
### inspector\_protocol
The [inspector\_protocol](https://chromium.googlesource.com/deps/inspector_protocol/)
is Chromium's of code generators and templates for the inspector protocol.
See [this doc](../../../tools/inspector_protocol/README.md) for more information.
### libuv
The [libuv](https://github.com/libuv/libuv) dependency is a
multi-platform support library with a focus on asynchronous I/O.
It was primarily developed for use by Node.js.
### llhttp
The [llhttp](https://github.com/nodejs/llhttp) dependency is
the http parser used by Node.js.
See [maintaining-http][] for more information.
### minimatch
The [minimatch](https://github.com/isaacs/minimatch) dependency is a
minimal matching utility.
### nghttp2
The [nghttp2](https://github.com/nghttp2/nghttp2) dependency is a C library
implementing HTTP/2 protocol.
See [maintaining-http][] for more information.
### nghttp3
The [nghttp3](https://github.com/ngtcp2/nghttp3) dependency is HTTP/3 library
written in C. See ngtcp2 for more information.
### ngtcp2
The ngtcp2 and nghttp3 dependencies provide the core functionality for
QUIC and HTTP/3.
The sources are pulled from:
* ngtcp2: <https://github.com/ngtcp2/ngtcp2>
* nghttp3: <https://github.com/ngtcp2/nghttp3>
In both the `ngtcp2` and `nghttp3` git repos, the active development occurs
in the default branch (currently named `main` in each). Tagged versions do not
always point to the default branch.
We only use a subset of the sources for each.
The `nghttp3` library depends on `ngtcp2`. Both should always be updated
together. From `ngtcp2` we only want the contents of the `lib` and `crypto`
directories; from `nghttp3` we only want the contents of the `lib` directory.
### npm
The [npm](https://github.com/npm/cli) dependency is
the package manager for JavaScript.
New pull requests should be opened when a "next" version of npm has
been released. Once the "next" version has been promoted to "latest"
the PR should be updated as necessary.
The specific Node.js release streams the new version will be able to land into
are at the discretion of the release and LTS teams.
This process only covers full updates to new versions of npm. Cherry-picked
changes can be reviewed and landed via the normal consensus seeking process.
### openssl
The [openssl](https://github.com/quictls/openssl) dependency is a
fork of OpenSSL to enable QUIC.
[OpenSSL](https://www.openssl.org/) is toolkit for general-purpose
cryptography and secure communication.
Node.js currently uses the quictls/openssl fork, which closely tracks
the main openssl/openssl releases with the addition of APIs to support
the QUIC protocol.
See [maintaining-openssl][] for more information.
### postject
The [postject](https://github.com/nodejs/postject) dependency is used for the
[Single Executable strategic initiative](https://github.com/nodejs/single-executable).
### simdjson
The [simdjson](https://github.com/simdjson/simdjson) dependency is
a C++ library for fast JSON parsing.
### sqlite
The [sqlite](https://github.com/sqlite/sqlite) dependency is
an embedded SQL database engine written in C.
### undici
The [undici](https://github.com/nodejs/undici) dependency is an HTTP/1.1 client,
written from scratch for Node.js..
See [maintaining-http][] for more information.
### uvwasi
The [uvwasi](https://github.com/nodejs/uvwasi) dependency implements
the WASI system call API, so that WebAssembly runtimes can easily
implement WASI calls.
Under the hood, uvwasi leverages libuv where possible for maximum portability.
See [maintaining-web-assembly][] for more information.
### V8
[V8](https://chromium.googlesource.com/v8/v8.git/) is Google's open source
high-performance JavaScript and WebAssembly engine, written in C++.
See [maintaining-V8][] for more information.
### zlib
The [zlib](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/zlib)
dependency lossless data-compression library,
it comes from the Chromium team's zlib fork which incorporated
performance improvements not currently available in standard zlib.
### zstd
The [zstd](https://github.com/facebook/zstd) dependency is used for compression
according to [RFC 8878](https://datatracker.ietf.org/doc/html/rfc8878).
[acorn]: #acorn
[ada]: #ada
[amaro]: #amaro
[base64]: #base64
[brotli]: #brotli
[c-ares]: #c-ares
[cjs-module-lexer]: #cjs-module-lexer
[corepack]: #corepack
[dependency-update-action]: ../../../.github/workflows/tools.yml
[googletest]: #googletest
[histogram]: #histogram
[icu-small]: #icu-small
[inspector_protocol]: #inspector_protocol
[libuv]: #libuv
[llhttp]: #llhttp
[maintaining-V8]: ./maintaining-V8.md
[maintaining-cjs-module-lexer]: ./maintaining-cjs-module-lexer.md
[maintaining-http]: ./maintaining-http.md
[maintaining-icu]: ./maintaining-icu.md
[maintaining-openssl]: ./maintaining-openssl.md
[maintaining-web-assembly]: ./maintaining-web-assembly.md
[minimatch]: #minimatch
[nghttp2]: #nghttp2
[nghttp3]: #nghttp3
[ngtcp2]: #ngtcp2
[npm]: #npm
[openssl]: #openssl
[postject]: #postject
[simdjson]: #simdjson
[sqlite]: #sqlite
[undici]: #undici
[update-openssl-action]: ../../../.github/workflows/update-openssl.yml
[uvwasi]: #uvwasi
[v8]: #v8
[zlib]: #zlib
[zstd]: #zstd

90
doc/contributing/maintaining/maintaining-http.md Normal file
View File

@ -0,0 +1,90 @@
# Maintaining HTTP
Support for HTTP is a key priority in terms of ensuring the continued success of
Node.js as captured in the project's
[technical priorities](https://github.com/nodejs/node/blob/HEAD/doc/contributing/technical-priorities.md).
The current high level strategy is based on the discussion in the
[Next-10](https://github.com/nodejs/next-10)
[mini-summit](https://github.com/nodejs/next-10/blob/main/meetings/summit-jan-2022.md)
on modern HTTP which was held on Jan 27 2022.
## High level strategy
The key elements of our strategy for future HTTP APIs are:
* APIs should be HTTP protocol independent (support HTTP1, HTTP2, etc.).
* APIs should be transport protocol independent (TCP, QUIC, etc.).
* APIs should provide good defaults that perform well.
* Client/server APIs should be consistent and allow easy integration.
* Common requirements like piping out from client API to server APIs should be
easy.
* For both the Client and Server there is a need for multiple APIs, with each
targeting a different level of abstraction.
Unfortunately our existing HTTP APIs (
[HTTP](https://nodejs.org/docs/latest/api/http.html),
[HTTPS](https://nodejs.org/docs/latest/api/https.html), and
[HTTP2](https://nodejs.org/docs/latest/api/http2.html))
do not align with our high level strategy. While these APIs
are widely used and we do not plan to deprecate or remove them,
they are not the focus of active development or performance improvements.
Bug fixes however are still important for all of these APIs.
With respect to the HTTP protocols themselves, our current assessment in
terms of priority within existing or new APIs is:
* HTTP 2 is in “maintenance mode” for both protocol and APIs.
* HTTP 1 is a stable protocol, but innovation is still possible with the APIs.
* HTTP 3 is an important protocol and we need to add support for it.
The current strategy is to build out 2 new client APIs and 2 new Server APIs
in line with the high level strategy above.
While transport-level APIs are important (e.g. socket level), the HTTP APIs
should not be tied to a specific transport-level API. Therefore,
transport-level APIs are out of the scope of our HTTP strategy/maintenance
information.
### Client APIs
For client APIs we want a high-level API and a low-level API when
more control is required. The current plan is for the following APIs:
* High-level API -
[Fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API)
based API built on top of [undici](https://www.npmjs.com/package/undici).
* Low-level API - a subset of the APIs exposed by
[undici](https://www.npmjs.com/package/undici). The exact shape and
set of these APIs is still to be worked out. The current plan is to
pull undici into Node.js core without exposing its APIs in the Node.js
API so that it can initially be used to support the higher level
Fetch-based API. As this gets worked out we will discuss which
APIs to expose in the Node.js API surface.
### Server APIs
For the server APIs we do not yet have a clear path, other than wanting
to align them with the APIs built for the client.
### HTTP, HTTPS
The low-level implementation of the
[HTTP](https://nodejs.org/docs/latest/api/http.html)
and [HTTPS](https://nodejs.org/docs/latest/api/https.html) APIs
are maintained in the [llhttp](https://github.com/nodejs/llhttp)
repository. Updates are pulled into Node.js under
[deps/llhttp](https://github.com/nodejs/node/tree/HEAD/deps/llhttp).
### HTTP2
The low-level implementation of
[HTTP2](https://nodejs.org/docs/latest/api/http2.html)
is based on [nghttp2](https://nghttp2.org/). Updates are pulled into Node.js
under [deps/nghttp2](https://github.com/nodejs/node/tree/HEAD/deps/nghttp2)
as needed.
The low-level implementation is made available in the Node.js API through
JavaScript code in the [lib](https://github.com/nodejs/node/tree/HEAD/lib)
directory and C++ code in the
[src](https://github.com/nodejs/node/tree/HEAD/src) directory

270
doc/contributing/maintaining/maintaining-icu.md Normal file
View File

@ -0,0 +1,270 @@
# Maintaining ICU in Node.js
## Background
International Components for Unicode ([ICU4C][ICU]) is used both by V8
and also by Node.js directly to provide internationalization
functionality. To quote from icu-project.org:
> ICU is a mature, widely used set of C/C++ and Java libraries providing
> Unicode and Globalization support for software applications. ICU is
> widely portable and gives applications the same results on all platforms
> and between C/C++ and Java software.
If Node.js is configured to use its built-in ICU,
it uses a strict subset of ICU which is in
[deps/icu-small](https://github.com/nodejs/node/tree/HEAD/deps/icu-small).
A good description of the different ways Node.js can be built with ICU
support is in [api/intl.html](https://nodejs.org/api/intl.html).
## Data dependencies
ICU consumes and includes:
* Extracted locale data from [CLDR][]
* Extracted [Unicode][] data.
* Time zone ([tz][]) data
The current versions of these items can be viewed for node with `node -p process.versions`:
```console
$ node -p process.versions
{
…
cldr: '35.1',
icu: '64.2',
tz: '2019a',
unicode: '12.1'
}
```
### Time zone data
Time zone data files are updated independently of ICU CLDR data. ICU and its
main data files do not need to be upgraded in order to apply time zone data file
fixes.
The [IANA tzdata][tz] project releases new versions and announces them on the
[`tz-announce`](https://mm.icann.org/pipermail/tz-announce/) mailing list.
The Unicode project takes new releases and publishes
[updated time zone data files](https://github.com/unicode-org/icu-data/tree/HEAD/tzdata/icunew)
in the icu/icu-data repository.
All modern versions of Node.js use the version 44 ABI of the time zone data
files.
#### Example: updating the ICU `.dat` file
* Decompress `deps/icu-small/source/data/in/icudt##l.dat.bz2`, where `##` is
the ICU major version number.
* Clone the icu/icu-data repository and copy the latest `tzdata` release `le`
files into the `source/data/in` directory.
* Follow the upstream [ICU instructions](https://unicode-org.github.io/icu/userguide/datetime/timezone/)
to patch the ICU `.dat` file:
> `for i in zoneinfo64.res windowsZones.res timezoneTypes.res metaZones.res;
> do icupkg -a $i icudt*l.dat`
* Optionally, verify that there is only one of the above files listed when using
`icupkg -l`.
* Optionally, extract each file using `icupkg -x` and verify the `shasum`
matches the desired value.
* Compress the `.dat` file with the same filename as in the first step.
* Build, test, verifying `process.versions.tz` matches the desired version.
* Create a new minor version release.
## Release schedule
ICU typically has >1 release a year, particularly coinciding with a major
release of [Unicode][]. The current release schedule is available on the [ICU][]
website on the left sidebar.
### V8 depends on ICU
V8 will aggressively upgrade to a new ICU version, due to requirements for
features/bugfixes needed for [Ecma402][] support. The minimum required version
of ICU is specified within the V8 source tree. If the ICU version is too old,
V8 will not compile.
```c
// deps/v8/src/objects/intl-objects.h
#define V8_MINIMUM_ICU_VERSION 65
```
V8 in Node.js depends on the ICU version supplied by Node.js.
The file `tools/icu/icu_versions.json` contains the current minimum
version of ICU that Node.js is known to work with. This should be
_at least_ the same version as V8, so that users will find out
earlier that their ICU is too old. A test case validates this when
Node.js is built.
## How to upgrade ICU
> The script `tools/dep_updaters/update-icu.sh` automates
> this process.
* Make sure your Node.js workspace is clean (`git status`
should be sufficient).
* Configure Node.js with the specific [ICU version](http://site.icu-project.org/download)
you want to upgrade to, for example:
```bash
./configure \
--with-intl=full-icu \
--with-icu-source=https://github.com/unicode-org/icu/releases/download/release-67-1/icu4c-67_1-src.tgz
make
```
> _Note_ in theory, the equivalent `vcbuild.bat` commands should work also,
> but the commands below are makefile-centric.
* If there are ICU version-specific changes needed, you may need to make changes
in `tools/icu/icu-generic.gyp` or add patch files to `tools/icu/patches`.
* Specifically, look for the lists in `sources!` in the `tools/icu/icu-generic.gyp` for
files to exclude.
* Verify the Node.js build works:
```bash
make test-ci
```
Also running
<!-- eslint-disable strict -->
```js
new Intl.DateTimeFormat('es', { month: 'long' }).format(new Date(9E8));
```
…Should return `enero` not `January`.
* Now, run the shrink tool to update `deps/icu-small` from `deps/icu`
> :warning: Do not modify any source code in `deps/icu-small` !
> See section below about floating patches to ICU.
```bash
python tools/icu/shrink-icu-src.py
```
* Now, do a clean rebuild of Node.js to test:
```bash
make -k distclean
./configure
make
```
* Test this newly default-generated Node.js
<!-- eslint-disable strict -->
```js
process.versions.icu;
new Intl.DateTimeFormat('es', { month: 'long' }).format(new Date(9E8));
```
(This should print your updated ICU version number, and also `enero` again.)
You are ready to check in (`git add`) the updated `deps/icu-small`.
> :warning: Do not modify any source code in `deps/icu-small` !
> See section below about floating patches to ICU.
* Now, rebuild the Node.js license.
```bash
# clean up - remove deps/icu
make clean
tools/license-builder.sh
```
* Update the URL and hash for the full ICU file in `tools/icu/current_ver.dep`.
It should match the ICU URL used in the first step. When this is done, the
following should build with small ICU.
```bash
# clean up
rm -rf out deps/icu deps/icu4c*
./configure --with-intl=small-icu --download=all
make
make test-ci
```
* Commit the change to the `deps/icu-small`, `tools/icu/current_ver.dep`
and `LICENSE` files.
## Floating patches to ICU
Floating patches are applied at `configure` time. The "patch" files
are used instead of the original source files. The patch files are
complete `.cpp` files replacing the original contents.
Patches are tied to a specific ICU version. They won't apply to a
future ICU version. We assume that you filed a bug against [ICU][] and
upstreamed the fix, so the patch won't be needed in a later ICU
version.
### Example
For example, to patch `source/tools/toolutil/pkg_genc.cpp` for
ICU version 63:
```bash
# go to your Node.js source directory
cd <node>
# create the floating patch directory
mkdir -p tools/icu/patches/63
# create the subdirectory for the file(s) to patch:
mkdir -p tools/icu/patches/63/source/tools/toolutil/
# copy the file to patch
cp deps/icu-small/source/tools/toolutil/pkg_genc.cpp \
tools/icu/patches/63/source/tools/toolutil/pkg_genc.cpp
# Make any changes to this file:
(edit tools/icu/patches/63/source/tools/toolutil/pkg_genc.cpp )
# test
make clean && ./configure && make
```
You should see a message such as:
```console
INFO: Using floating patch "tools/icu/patches/63/source/tools/toolutil/pkg_genc.cpp" from "tools/icu"
```
### Clean up
Any patches older than the minimum version given in `tools/icu/icu_versions.json`
ought to be deleted, because they will never be used.
### Why not just modify the ICU source directly?
Especially given the V8 dependencies above, there may be times when a floating
patch to ICU is required. Though it seems expedient to simply change a file in
`deps/icu-small`, this is not the right approach for the following reasons:
1. **Repeatability.** Given the complexity of merging in a new ICU version,
following the steps above in the prior section of this document ought to be
repeatable without concern for overriding a patch.
2. **Verifiability.** Given the number of files modified in an ICU PR,
a floating patch could easily be missed or dropped altogether next time
something is landed.
3. **Compatibility.** There are a number of ways that ICU can be loaded into
Node.js (see the top level README.md). Only modifying `icu-small` would cause
the patch not to be landed in case the user specifies the ICU source code
another way.
[CLDR]: http://cldr.unicode.org/
[Ecma402]: https://github.com/tc39/ecma402
[ICU]: http://site.icu-project.org/
[Unicode]: https://home.unicode.org/
[tz]: https://www.iana.org/time-zones

133
doc/contributing/maintaining/maintaining-openssl.md Normal file
View File

@ -0,0 +1,133 @@
# Maintaining OpenSSL
OpenSSL is automatically updated by the [update-openssl-action][].
There is also a script in `tools/dep_updaters` that can be used to update it.
This document describes how to manually update `deps/openssl/`.
## Requirements
* Linux environment.
* `perl` Only Perl version 5 is tested.
* `nasm` (<https://www.nasm.us/>) Version 2.11 or higher is needed.
* GNU `as` in binutils. Version 2.26 or higher is needed.
## 0. Check requirements
```console
% perl -v
This is perl 5, version 22, subversion 1 (v5.22.1) built for
x86_64-linux-gnu-thread-multi
(with 60 registered patches, see perl -V for more detail)
% as --version
GNU assembler (GNU Binutils for Ubuntu) 2.26.1
Copyright (C) 2015 Free Software Foundation, Inc.
...
% nasm -v
NASM version 2.11.08
```
## 1. Obtain and extract new OpenSSL sources
Get a new source from <https://github.com/openssl/openssl/tree/openssl-3.0.16>
and copy all files into `deps/openssl/openssl`. Then add all files and commit
them. (The link above, and the branch, will change with each new OpenSSL
release).
```bash
git clone https://github.com/openssl/openssl
cd openssl
cd ../node/deps/openssl
rm -rf openssl
cp -R ../../../openssl openssl
rm -rf openssl/.git*
git add --all openssl
git commit openssl
```
```text
deps: upgrade openssl sources to openssl-3.0.16
This updates all sources in deps/openssl/openssl by:
$ git clone git@github.com:openssl/openssl.git
$ cd openssl
$ git checkout openssl-3.0.16
$ cd ../node/deps/openssl
$ rm -rf openssl
$ cp -R ../../../openssl openssl
$ rm -rf openssl/.git*
$ git add --all openssl
$ git commit openssl
```
## 2. Execute `make` in `deps/openssl/config` directory
Use `make` to regenerate all platform dependent files in
`deps/openssl/config/archs/`:
```bash
# On non-Linux machines
make gen-openssl
# On Linux machines
make -C deps/openssl/config clean
make -C deps/openssl/config
```
Fix up 32-bit Windows assembler directives. This will allow the commits to be
cherry-picked to older release lines that still provide binaries on 32-bit Windows.
```bash
make -C deps/openssl/config clean
# Edit deps/openssl/openssl/crypto/perlasm/x86asm.pl changing
# #ifdef to %ifdef to make it compatible to nasm on 32-bit Windows.
# See: https://github.com/nodejs/node/pull/43603#issuecomment-1170670844
# Reference: https://github.com/openssl/openssl/issues/18459
```
## 3. Check diffs
Check diffs to ensure updates are right. Even if there are no updates in openssl
sources, `buildinf.h` files will be updated because they have timestamp
data in them.
```bash
git diff -- deps/openssl
```
_Note_: On Windows, OpenSSL Configure generates a `makefile` that can be
used for the `nmake` command. The `make` command in step 2 (above) uses
`Makefile_VC-WIN64A` and `Makefile_VC-WIN32` that are manually
created. When source files or build options are updated in Windows,
it needs to change these two Makefiles by hand. If you are not sure,
please ask @shigeki for details.
## 4. Commit and make test
Update all architecture dependent files. Do not forget to git add or remove
files if they are changed before committing:
```bash
git add deps/openssl/config/archs
git add deps/openssl/openssl
git commit
```
The commit message can be written as (with the openssl version set
to the relevant value):
```text
deps: update archs files for openssl-3.0.16
After an OpenSSL source update, all the config files need to be
regenerated and committed by:
$ make -C deps/openssl/config
$ git add deps/openssl/config/archs
$ git add deps/openssl/openssl
$ git commit
```
Finally, build Node.js and run the tests.
[update-openssl-action]: ../../../.github/workflows/update-openssl.yml

140
doc/contributing/maintaining/maintaining-root-certs.md Normal file
View File

@ -0,0 +1,140 @@
# Maintaining the root certificates
Node.js contains a compiled-in set of root certificates used as trust anchors
for TLS certificate validation.
The certificates come from Mozilla, specifically NSS's `certdata.txt` file.
The PEM encodings of the certificates are converted to C strings, and committed
in `src/node_root_certs.h`.
## When to update
Root certificates should be updated sometime after Mozilla makes an NSS release,
check the [NSS release schedule][].
## Process
The `tools/dep_updaters/update-root-certs.mjs` script automates the update of
the root certificates, including:
* Downloading `certdata.txt` from Mozilla's source control repository.
* Running `tools/mk-ca-bundle.pl` to convert the certificates and generate
`src/node_root_certs.h`.
* Using `git diff-files` to determine which certificate have been added and/or
removed.
Manual instructions are included in the following collapsed section.
<details>
Commands assume that the current working directory is the root of a checkout of
the nodejs/node repository.
1. Find NSS metadata for update.
The latest released NSS version, release date, Firefox version, and Firefox
release date can be found in the [NSS release schedule][].
The tag to fetch `certdata.txt` from is found by looking for the release
version in the [tag list][].
2. Update `certdata.txt` from the NSS release tag.
Update the tag in the commands below, and run:
```bash
cd tools/
./mk-ca-bundle.pl -v 2>_before
curl -O https://hg.mozilla.org/projects/nss/raw-file/NSS_3_41_RTM/lib/ckfw/builtins/certdata.txt
```
The `_before` file will be used later. Verify that running `mk-ca-bundle`
made no changes to `src/node_root_certs.h`. If it did, something went wrong
with the previous update. Seek help!
Update metadata in the message below, and commit `certdata.txt`:
```text
tools: update certdata.txt
This is the certdata.txt[0] from NSS 3.41, released on 2018-12-03.
This is the version of NSS that will ship in Firefox 65 on
2018-12-11.
[0] https://hg.mozilla.org/projects/nss/raw-file/NSS_3_41_RTM/lib/ckfw/builtins/certdata.txt
```
3. Update `node_root_certs.h` from `certdata.txt`.
Run the command below:
```bash
./mk-ca-bundle.pl -v 2>_after
```
Confirm that `../src/node_root_certs.h` was updated.
Determine what changes were made by diffing the before and after files:
```console
% diff _before _after
11d10
< Parsing: Visa eCommerce Root
106d104
< Parsing: TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı H5
113,117d110
< Parsing: Certplus Root CA G1
< Parsing: Certplus Root CA G2
< Parsing: OpenTrust Root CA G1
< Parsing: OpenTrust Root CA G2
< Parsing: OpenTrust Root CA G3
134c127,136
< Done (133 CA certs processed, 20 skipped).
---
> Parsing: GlobalSign Root CA - R6
> Parsing: OISTE WISeKey Global Root GC CA
> Parsing: GTS Root R1
> Parsing: GTS Root R2
> Parsing: GTS Root R3
> Parsing: GTS Root R4
> Parsing: UCA Global G2 Root
> Parsing: UCA Extended Validation Root
> Parsing: Certigna Root CA
> Done (135 CA certs processed, 16 skipped).
```
Use the diff to update the message below, and commit `src/node_root_certs.h`:
```text
crypto: update root certificates
Update the list of root certificates in src/node_root_certs.h with
tools/mk-ca-bundle.pl.
Certificates added:
- GlobalSign Root CA - R6
- OISTE WISeKey Global Root GC CA
- GTS Root R1
- GTS Root R2
- GTS Root R3
- GTS Root R4
- UCA Global G2 Root
- UCA Extended Validation Root
- Certigna Root CA
Certificates removed:
- Visa eCommerce Root
- TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı H5
- Certplus Root CA G1
- Certplus Root CA G2
- OpenTrust Root CA G1
- OpenTrust Root CA G2
- OpenTrust Root CA G3
```
</details>
[NSS release schedule]: https://wiki.mozilla.org/NSS:Release_Versions
[tag list]: https://hg.mozilla.org/projects/nss/tags

117
doc/contributing/maintaining/maintaining-shared-library-support.md Normal file
View File

@ -0,0 +1,117 @@
# Maintaining shared library support
Node.js unofficially supports a build option where Node.js is built as
a shared library. The shared library is called libnode with additional postfixes
as appropriate for the platform (for example libnode.dll on windows).
The shared library provides a way to embed Node.js into other
applications and to have multiple applications use a single copy of
Node.js instead of having to bundle in the full Node.js footprint
into each application. For workloads that require multiple Node.js
instances this can result in a significant footprint savings.
This document provides an outline of the approach and things to look
out for when maintaining the shared library support.
Currently, shared library support has only been tested on:
* Linux
* macOS
* Windows
* AIX
## Building with shared library option
On non-Windows platforms, Node.js is built with the shared library
option by adding `--shared` to the configure step. On Windows
platforms Node.js is built with the shared library option by
adding `dll` to the vcbuild command line.
Once built there are two key components:
* executable - node
* library - libnode
The node executable is a thin wrapper around libnode which is
generated so that we can run the standard Node.js test suite
against the shared library.
The executable and library will have extensions as appropriate
for the platform on which they are built. For
example, node.exe on windows and node on other platforms for
the executable.
libnode may have additional naming components, as an example
in a build on macOS `libnode.105.dylib`. For non-windows platforms
the additional naming components include the `NODE_MODULE_VERSION` and
the appropriate postfix used for shared libraries on the platform.
In cases where an application links against the shared
library it is up to the application developer to add options
so that the shared library can be found by the application or
to set the LIBPATH (AIX), LD\_LIBRARY\_PATH (Linux/Unix), etc.
so that it is found at runtime.
For the node wrapper, on linux and macOS it is built
so that it can find the shared library in one of
the following:
* the same directory as the node executable
* ../lib with the expectation that the executable is
installed in a `bin` directory at the same level
as a `lib` directory in which the shared library is
installed. This is where the default package that
is build with the shared library option will
place the executable and library.
For the node wrapper on windows it is built expecting
that both the executable and shared library will
be in the same directory as it common practice on
that platform.
For the node wrapper on AIX, it is built with
the path to the shared library hardcoded as that
is the only option.
## Exports
On windows, functions that may be linked from native
addons or additional Node.js executables need to have
NODE\_EXTERN\_PRIVATE or NODE\_EXTERN otherwise they will
not be exported by the shared library. In the case of
functions used by additional Node.js executables
(ex: `mksnapshot`) a missing NODE\_EXTERN or
NODE\_EXTERN\_PRIVATE will cause the build to fail.
NODE\_EXTERN\_PRIVATE should be used in these cases
unless the intent is to add the function to the
public embedder API.
## Native addons
For regular Node.js builds, running native addons relies on symbols
exported by the node executable. As a result any
pre-built binaries expect symbols to be exported from the executable
instead of the shared library itself.
The node executable and shared library are built and linked
so that the required symbols are exported from the node
executable. This requires some extra work on some platforms
and the process to build the node executable is a good example
of how this can be achieved. Applications that use the shared
library and want to support native addons should employ
a similar technique.
## Testing
There is currently no testing in the regular CI run for PRs. There
are some CI jobs that can be used to test the shared library support and
some are run as part of daily testing. These include:
* [node-test-commit-linux-as-shared-lib](https://ci.nodejs.org/view/Node.js%20Daily/job/node-test-commit-linux-as-shared-lib/)
* <https://ci.nodejs.org/view/All/job/node-test-commit-osx-as-shared-lib/>
* [node-test-commit-aix-shared-lib](https://ci.nodejs.org/view/Node.js%20Daily/job/node-test-commit-aix-shared-lib/)
TODO: add a Job for windows
For code that modifies/affects shared library support these CI jobs should
run and it should be validated that there are no regressions in
the test suite.

83
doc/contributing/maintaining/maintaining-single-executable-application-support.md Normal file
View File

@ -0,0 +1,83 @@
# Maintaining Single Executable Applications support
Support for [single executable applications][] is one of the key technical
priorities identified for the success of Node.js.
## High level strategy
From the [Next-10 discussions][] there are 2 approaches the project believes are
important to support:
### Compile with Node.js into executable
This is the approach followed by [boxednode][].
No additional code within the Node.js project is needed to support the
option of compiling a bundled application along with Node.js into a single
executable application.
### Bundle into existing Node.js executable
This is the approach followed by [pkg][].
The project does not plan to provide the complete solution but instead the key
elements which are required in the Node.js executable in order to enable
bundling with the pre-built Node.js binaries. This includes:
* Looking for a segment within the executable that holds bundled code.
* Running the bundled code when such a segment is found.
It is left up to external tools/solutions to:
* Bundle code into a single script.
* Generate a command line with appropriate options.
* Add a segment to an existing Node.js executable which contains
the command line and appropriate headers.
* Re-generate or removing signatures on the resulting executable
* Provide a virtual file system, and hooking it in if needed to
support native modules or reading file contents.
However, the project also maintains a separate tool, [postject][], for injecting
arbitrary read-only resources into the binary such as those needed for bundling
the application into the runtime.
## Planning
Planning for this feature takes place in the [single-executable repository][].
## Upcoming features
Currently, only running a single embedded CommonJS file is supported but support
for the following features are in the list of work we'd like to get to:
* Running an embedded ESM file.
* Running an archive of multiple files.
* Embedding [Node.js CLI options][] into the binary.
* [XCOFF][] executable format.
* Run tests on Alpine Linux.
* Run tests on s390x Linux.
* Run tests on ppc64 Linux.
## Disabling single executable application support
To disable single executable application support, build Node.js with the
`--disable-single-executable-application` configuration option.
## Implementation
When built with single executable application support, the Node.js process uses
[`postject-api.h`][] to check if the `NODE_SEA_BLOB` section exists in the
binary. If it is found, it passes the buffer to
[`single_executable_application.js`][], which executes the contents of the
embedded script.
[Next-10 discussions]: https://github.com/nodejs/next-10/blob/main/meetings/summit-nov-2021.md#single-executable-applications
[Node.js CLI options]: https://nodejs.org/api/cli.html
[XCOFF]: https://www.ibm.com/docs/en/aix/7.2?topic=formats-xcoff-object-file-format
[`postject-api.h`]: https://github.com/nodejs/node/blob/71951a0e86da9253d7c422fa2520ee9143e557fa/test/fixtures/postject-copy/node_modules/postject/dist/postject-api.h
[`single_executable_application.js`]: https://github.com/nodejs/node/blob/main/lib/internal/main/single_executable_application.js
[boxednode]: https://github.com/mongodb-js/boxednode
[pkg]: https://github.com/vercel/pkg
[postject]: https://github.com/nodejs/postject
[single executable applications]: https://github.com/nodejs/node/blob/main/doc/contributing/technical-priorities.md#single-executable-applications
[single-executable repository]: https://github.com/nodejs/single-executable

62
doc/contributing/maintaining/maintaining-the-build-files.md Normal file
View File

@ -0,0 +1,62 @@
# Maintaining the build files
This document explains how to maintain the build files in the codebase.
## Overview
On how to build the Node.js core, see [Building Node.js](../../../BUILDING.md).
There are three main build files that may be directly run when building Node.js:
* `configure`: A Python script that detects system capabilities and runs
[GYP][]. It generates `config.gypi` which includes parameters used by GYP to
create platform-dependent build files. Its output is usually in one of these
formats: Makefile, MSbuild, ninja, or XCode project files (the main
Makefile mentioned below is maintained separately by humans). For a detailed
guide on this script, see [configure][].
* `vcbuild.bat`: A Windows Batch Script that locates build tools, provides a
subset of the targets available in the [Makefile][], and a few
targets of its own. For a detailed guide on this script, see
[vcbuild.bat](#vcbuildbat).
* `Makefile`: A Makefile that can be run with GNU Make. It provides a set of
targets that build and test the Node.js binary, produce releases and
documentation, and interact with the CI to run benchmarks or tests. For a
detailed guide on this file, see [Makefile][].
On Windows `vcbuild.bat` runs [configure][] before building the
Node.js binary, on other systems `configure` must be run manually before running
`make` on the `Makefile`.
## vcbuild.bat
To see the help text, run `.\vcbuild help`. Update this file when you need to
update the build and testing process on Windows.
## configure
The `configure` script recognizes many CLI flags for special build formulas.
Many are not represented by `vcbuild` shortcuts, and need to be passed
either by:
* Calling `python configure --XXX --YYY=PPPP` directly, followed by `vcbuild
noprojgen`
* Setting `set config_flags=--XXX --YYY=PPPP` before calling `vcbuild`
To see the help text, run `python configure --help`. Update this file when you
need to update the configuration process.
## Makefile
To see the help text, run `make help`. This file is not generated, it is
maintained by humans. This is not usually run on Windows, where
[vcbuild.bat](#vcbuildbat) is used instead.
### Options
* `-j <n>`: number of threads used to build the binary. On the non-CI
targets, the parallel tests will take up all the available cores, regardless
of this option.
[GYP]: https://gyp.gsrc.io/docs/UserDocumentation.md
[Makefile]: #makefile
[configure]: #configure

128
doc/contributing/maintaining/maintaining-types-for-nodejs.md Normal file
View File

@ -0,0 +1,128 @@
# Maintaining types for Node.js
While JavaScript is a weakly-typed language, there are some complementary tools
like [TypeScript][] and [Flow][], which allow developers to annotate the source
code of their JavaScript projects. While many people don't annotate their code,
or make use of annotations at all, there are enough who do that the project has
agreed it's important to work towards having [suitable types for end-users][].
## High level approach - maintaining types
There are a number of ways that types could be maintained for Node.js ranging
from shipping them with the Node.js runtime to having them be externally
maintained.
The different options were discussed as part of the [next-10][] effort and it
was agreed that maintaining them externally is the best approach.
Some of the advantages to this approach include:
* Node.js maintainers do not need to be familiar with any given type
system/technology.
* Types can be updated without requiring Node.js releases.
The agreement was that the ideal flow would be as follows:
* APIs are added/documented in the existing Node.js markdown files.
* Automation in the Node.js project creates a machine readable JSON
representation of the API from the documentation.
* Automation within external type projects consumes the JSON and automatically
generates a PR to add the API.
## High level approach - development workflow
The number of people using TypeScript with Node.js is significant enough
that providing a good developer experience is important. While TypeScript
is identified specifically, a secondary goal is that what we provide to improve
development experience with TypeScript would apply to other type
systems and transpiled languages as well.
We have agreed that the approach will **NOT** include bundling TypeScript
tools with Node.js but instead improve the developer experience for how
those tools are installed/configured to work with Node.js.
The high level developer experience we are working towards was captured in the
[next-10 TypeScript mini-summit](https://github.com/nodejs/next-10/pull/150)
and is as follows:
1. When Node.js is started with an entry point that is not a file type that
Node.js recognizes, for example `node script.ts`, an informative error
message is printed that directs users to a webpage where they can
learn how to configure Node.js to support that file type.
* If the file was a TypeScript file, a TypeScript specific message with a
reference to a link on Nodejs.org specific on learning how to
configure TypeScript will be provided.
* For other file types a generic message and shared webpage will be
used.
2. Node.js gains support for loading configuration from a file. Most, if not
all, of the configuration supported by `NODE_OPTIONS` would be
supported in this file (which might be the `package.json` that lives
near the entry point file). The webpage with instructions would tell
users what configuration to put in this file to get Node.js to support
their file type.
3. When Node.js is run with the correct configuration, either in a file or
`NODE_OPTIONS` or flags, the unknown file type is executed as expected.
Some additional specifics around the current approach include:
* Loaders already provide a number of the components needed to
satisfy the requirements above. They already provide the Node.js
options that are needed to achieve many of the requirements above.
* `package.json` as the location for the config is potentially a good
choice as Node.js already looks for it as part of startup.
* The implementation chosen should allow for different configuration
in/for different environments/conditions such as production
versus development, or different types of hosted environments
such as serverless vs traditional, etc.; Node.js would not make
any recommendations or have any expectations as to what the
separate configuration blocks should be named or what their
purposes should be, just that a configuration file should have
the ability to provide different configurations for user-defined
conditions.
* There is no plan to define a default tsconfig.json for all Node.js users
* We don't have consensus on providing an opinionated default but
that should be explored after the initial steps are complete.
* It will be important that as part of the messaging around this
functionality that we avoid confusion that could lead people to ship
TypeScript files (e.g. `script.ts`) instead of the processed files
(e.g. `script.js`).
## Generation/Consumption of machine readable JSON files
When you run `make doc` the canonical markdown files used to
document the Node.js APIs in the
[doc/api][]
directory are converted to both an `.html` file and a `.json` file.
As part of the regular build/release process both the `html` and
`json` files are published to [nodejs.org][].
The generator that does the conversion is in the
[tools/doc][]
directory.
## Markdown structure
The constraints required on the markdown files in the
[doc/api][] directory
in order to be able to generate the JSON files are defined in the
[documentation-style-guide][].
## Planned changes (as of Jan 1 2022)
While JSON files are already being generated and published, they are not
structured well enough for them to be easily consumed by the type projects.
Generally external teams need some custom scripts along with manual fixup
afterwards.
There is an ongoing effort to add additional markdown constraints and
then update the flow in order to be able to generate a better
JSON output.
[Flow]: https://flow.org/
[TypeScript]: https://www.typescriptlang.org/
[doc/api]: https://github.com/nodejs/node/tree/HEAD/doc/api
[documentation-style-guide]: https://github.com/nodejs/node/blob/HEAD/doc/README.md#documentation-style-guide
[next-10]: https://github.com/nodejs/next-10/blob/HEAD/meetings/summit-nov-2021.md#suitable-types-for-end-users
[nodejs.org]: https://nodejs.org/en/docs/
[suitable types for end-users]: https://github.com/nodejs/node/blob/HEAD/doc/contributing/technical-priorities.md#suitable-types-for-end-users
[tools/doc]: https://github.com/nodejs/node/tree/HEAD/tools/doc

90
doc/contributing/maintaining/maintaining-web-assembly.md Normal file
View File

@ -0,0 +1,90 @@
# Maintaining WebAssembly
Support for [WebAssembly](https://webassembly.org/)
has been identified as one of the
[top technical priorities](https://github.com/nodejs/node/blob/main/doc/contributing/technical-priorities.md#webassembly)
for the future success of Node.js.
This document provides an overview of our high-level strategy for
supporting WebAssembly and information about our current implementation
as a starting point for contributors.
## High-level approach
The key elements of our WebAssembly strategy include:
* Up-to-date core WebAssembly support
* Support for high-level APIs
* Making it easy to load WebAssembly
* Making sure the core Node.js APIs are compatible with WebAssembly
and can be called in an efficient manner from WebAssembly
### Up-to-date core WebAssembly support
Node.js gets its core WebAssembly support through V8. We don't need
to do anything specific to support this, all we have to do is keep
the version of V8 as up-to-date as possible.
### Key API support
As a runtime, Node.js must implement a number of APIs in addition
to the core WebAssembly support in order to be a good choice to run
WebAssembly. The project has currently identified these additional
APIs as important:
* WebAssembly System Interface (WASI). This provides the ability for
WebAssembly to interact with the outside world. Node.js currently
has an implementation (see below for more details).
* [WebAssembly Web API](https://www.w3.org/TR/wasm-web-api-1/). Node.js
currently has an implementation of streaming module compilation and
instantiation. As this and other specifications evolve, keeping up with them
will be important.
* [WebAssembly Component Model](https://github.com/WebAssembly/component-model/).
This API is still in the definition stage but the project should
keep track of its development as a way to simplify native code
integration.
### Making it as easy as possible to load WASM
The most important thing we can do on this front is to either find and
reference resources or provide resources on how to:
* Compile your WebAssembly code (outside of Node.js) and integrate that
into an npm workflow.
* Load and run WebAssembly code in your Node.js application.
It is also important to support and track the ongoing work in ESM to enable
loading of WebAssembly with ESM.
### Making sure the core Node.js APIs are compatible with WebAssembly
Use cases for which Node.js will be a good runtime will include code
both in JavaScript and compiled into WebAssembly. It is important
that Node.js APIs are able to be called from WebAssembly in
an efficient manner without extra buffer copies. We need to:
* Review APIs and identify those that can be called often from
WebAssembly.
* Where appropriate, make additions to identified APIs to allow
a pre-existing buffer to be passed in order to avoid copies.
## Current implementation and assets
### WebAssembly System Interface (WASI)
The Node.js WASI implementation is maintained in the
[uvwasi](https://github.com/nodejs/uvwasi) repository in the
Node.js GitHub organization. As needed, an updated copy
is vendored into the Node.js deps in
[deps/uvwasi](https://github.com/nodejs/node/tree/main/deps/uvwasi).
In addition to the code from uvwasi, Node.js includes bindings and
APIs that allow WebAssembly to be run with WASI support from Node.js.
The documentation for this API is in
[WebAssembly System Interface (WASI)](https://nodejs.org/api/wasi.html).
The implementation of the bindings and the public API is in:
* [src/node\_wasi.h](https://github.com/nodejs/node/blob/main/src/node_wasi.h)
* [src/node\_wasi.cc](https://github.com/nodejs/node/blob/main/src/node_wasi.cc)
* [lib/wasi.js](https://github.com/nodejs/node/blob/main/lib/wasi.js)