Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53

.github/workflows/build_packages.yaml

@@ -280,7 +280,7 @@ jobs:
         name: ${{ matrix.profile }}
         path: packages/${{ matrix.profile }}
     - name: install dos2unix
-      run: sudo apt-get update && sudo apt install -y dos2unix
+      run: sudo apt-get update -y && sudo apt install -y dos2unix
     - name: get packages
       run: |
         set -eu

apps/emqx/include/emqx_release.hrl

@@ -35,7 +35,7 @@
 -define(EMQX_RELEASE_CE, "5.3.0").
 
 %% Enterprise edition
--define(EMQX_RELEASE_EE, "5.3.0-rc.2").
+-define(EMQX_RELEASE_EE, "5.3.0").
 
 %% The HTTP API version
 -define(EMQX_API_VERSION, "5.0").

changes/ce/fix-1111682.en.md

@@ -1 +0,0 @@
-Fixed an issue where logging would stop if "Rotation Size" would be set to `infinity` on file log handlers.

changes/e5.3.0.en.md

@@ -0,0 +1,48 @@
+# Releases
+
+## e5.3.0
+
+### Enhancements
+
+- [#11597](https://github.com/emqx/emqx/pull/11597) Upgraded ekka to 0.15.13, which incorporates the following changes:
+  - Upgraded Mria to 0.6.2.
+  - Introduced the ability to configure the bootstrap data sync batch size, as detailed in [Mria PR](https://github.com/emqx/mria/pull/159).
+  - Enhanced the reliability of mria_membership processes, as described in [Mria PR](https://github.com/emqx/mria/pull/156).
+  - Fix log message formatting error.
+  - Added `node.default_bootstrap_batch_size` option to EMQX configuration.
+  Increasing the value of this option can greatly reduce a replicant node startup time, especially when the EMQX cluster interconnect network latency is high and the EMQX built-in database holds a large amount of data, e.g. when the number of subscriptions is high.
+- [#11620](https://github.com/emqx/emqx/pull/11620) Added a new rule-engine SQL function `bytesize` to get the size of a byte-string. e.g. `SELECT * FROM "t/#" WHERE bytesize(payload) > 10`.
+- [#11642](https://github.com/emqx/emqx/pull/11642) Updated to quicer version 0.0.200 in preparation for enabling openssl3 support for QUIC transport.
+
+- [#11610](https://github.com/emqx/emqx/pull/11610) Implemented a preliminary Role-Based Access Control for the Dashboard.
+
+  In this version, there are two predefined roles:
+  - Administrator: This role could access all resources.
+
+  - Viewer: This role can only view resources and data, corresponding to all GET requests in the REST API.
+
+- [#11631](https://github.com/emqx/emqx/pull/11631) Added Single Sign-On (SSO) feature and integrated with LDAP.
+
+- [#11656](https://github.com/emqx/emqx/pull/11656) Integrated the SAML 2.0 Support for SSO.
+
+- [#11599](https://github.com/emqx/emqx/pull/11599) Supported audit logs to record operations from CLI, REST API, and Dashboard in separate log files.
+
+### Bug Fixes
+
+- [#11682](https://github.com/emqx/emqx/pull/11682) Fixed an issue where logging would stop if "Rotation Size" would be set to `infinity` on file log handlers.
+- [#11567](https://github.com/emqx/emqx/pull/11567) Improve EMQX graceful shutdown (`emqx stop` command):
+  - Increase timeout from 1 to 2 minutes.
+  - Printed an error message if EMQX can't stop gracefully within the configured timeout.
+  - Print periodic status messages while EMQX is shutting down.
+- [#11584](https://github.com/emqx/emqx/pull/11584) Fixed telemetry reporting error on Windows when os_mon module is unavailable.
+- [#11605](https://github.com/emqx/emqx/pull/11605) Lowered CMD_overridden log severity from warning to info.
+- [#11622](https://github.com/emqx/emqx/pull/11622) Upgraded rpc library gen_rpc from 2.8.1 to 3.1.0.
+- [#11623](https://github.com/emqx/emqx/pull/11623) Upgraded library `esockd` from 5.9.6 to 5.9.7. This upgrade included:
+  * Enhancements regarding proxy protocol error and timeout. [esockd pr#178](https://github.com/emqx/esockd/pull/178)
+  * Lowered `ssl_error` exceptions to info-level logging. [esockd pr#180](https://github.com/emqx/esockd/pull/180)
+  * Malformed MQTT packet parsing exception log level is lowered from `error` to `info`.
+  * In command `emqx ctl listeners` output, the `shutdown_count` counter is incremented
+  when TLS handshake failure (`ssl_error`) or Malformed packet (`frame_error`) happens.
+- [#11661](https://github.com/emqx/emqx/pull/11661) Fixed log formatter when log.HANDLER.formatter is set to 'json'. The bug was introduced in v5.0.4 where the log line was no longer a valid JSON, but prefixed with timestamp string and level name.
+- [#11667](https://github.com/emqx/emqx/pull/11667) Disable access to the `logout` endpoint by the API key, this endpoint is for the Dashboard only.
+- [#11627](https://github.com/emqx/emqx/pull/11627) Fixed resources cleanup in HStreamdB bridge. Prior to this fix, HStreamDB bridge might report errors during bridge configuration updates, since hstreamdb client/producer were not stopped properly.

changes/v5.3.0.en.md

@@ -0,0 +1,32 @@
+# v5.3.0
+
+## Enhancements
+
+- [#11597](https://github.com/emqx/emqx/pull/11597) Upgraded ekka to 0.15.13, which incorporates the following changes:
+  - Upgraded Mria to 0.6.2.
+  - Introduced the ability to configure the bootstrap data sync batch size, as detailed in [Mria PR](https://github.com/emqx/mria/pull/159).
+  - Enhanced the reliability of mria_membership processes, as described in [Mria PR](https://github.com/emqx/mria/pull/156).
+  - Fix log message formatting error.
+  - Added `node.default_bootstrap_batch_size` option to EMQX configuration.
+  Increasing the value of this option can greatly reduce a replicant node startup time, especially when the EMQX cluster interconnect network latency is high and the EMQX built-in database holds a large amount of data, e.g. when the number of subscriptions is high.
+- [#11620](https://github.com/emqx/emqx/pull/11620) Added a new rule-engine SQL function `bytesize` to get the size of a byte-string. e.g. `SELECT * FROM "t/#" WHERE bytesize(payload) > 10`.
+- [#11642](https://github.com/emqx/emqx/pull/11642) Updated to quicer version 0.0.200 in preparation for enabling openssl3 support for QUIC transport.
+
+## Bug Fixes
+
+- [#11682](https://github.com/emqx/emqx/pull/11682) Fixed an issue where logging would stop if "Rotation Size" would be set to `infinity` on file log handlers.
+- [#11567](https://github.com/emqx/emqx/pull/11567) Improve EMQX graceful shutdown (`emqx stop` command):
+  - Increase timeout from 1 to 2 minutes.
+  - Printed an error message if EMQX can't stop gracefully within the configured timeout.
+  - Print periodic status messages while EMQX is shutting down.
+- [#11584](https://github.com/emqx/emqx/pull/11584) Fixed telemetry reporting error on Windows when os_mon module is unavailable.
+- [#11605](https://github.com/emqx/emqx/pull/11605) Lowered CMD_overridden log severity from warning to info.
+- [#11622](https://github.com/emqx/emqx/pull/11622) Upgraded rpc library gen_rpc from 2.8.1 to 3.1.0.
+- [#11623](https://github.com/emqx/emqx/pull/11623) Upgraded library `esockd` from 5.9.6 to 5.9.7. This upgrade included:
+  * Enhancements regarding proxy protocol error and timeout. [esockd pr#178](https://github.com/emqx/esockd/pull/178)
+  * Lowered `ssl_error` exceptions to info-level logging. [esockd pr#180](https://github.com/emqx/esockd/pull/180)
+  * Malformed MQTT packet parsing exception log level is lowered from `error` to `info`.
+  * In command `emqx ctl listeners` output, the `shutdown_count` counter is incremented
+  when TLS handshake failure (`ssl_error`) or Malformed packet (`frame_error`) happens.
+- [#11661](https://github.com/emqx/emqx/pull/11661) Fixed log formatter when log.HANDLER.formatter is set to 'json'. The bug was introduced in v5.0.4 where the log line was no longer a valid JSON, but prefixed with timestamp string and level name.
+- [#11667](https://github.com/emqx/emqx/pull/11667) Disable access to the `logout` endpoint by the API key, this endpoint is for the Dashboard only.

deploy/charts/emqx-enterprise/Chart.yaml

@@ -14,8 +14,8 @@ type: application
 
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
-version: 5.3.0-rc.2
+version: 5.3.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application.
-appVersion: 5.3.0-rc.2
+appVersion: 5.3.0

scripts/rel/cut.sh

@@ -233,56 +233,55 @@ if [ -d "${CHECKS_DIR}" ]; then
     done
 fi
 
-generate_changelog () {
-    local from_tag
-    from_tag="${PREV_TAG:-}"
-    if [[ -z $from_tag ]]; then
-        from_tag="$(./scripts/find-prev-rel-tag.sh "$PROFILE")"
-    fi
-    # num_en=$(git diff --name-only -a "${from_tag}...HEAD" "changes" | grep -c '.en.md')
-    # num_zh=$(git diff --name-only -a "${from_tag}...HEAD" "changes" | grep -c '.zh.md')
-    # if [ "$num_en" -ne "$num_zh" ]; then
-    #     echo "Number of English and Chinese changelog files added since ${from_tag} do not match."
-    #     exit 1
-    # fi
-    ./scripts/rel/format-changelog.sh -b "${from_tag}" -l 'en' -v "$TAG" > "changes/${TAG}.en.md"
-    # ./scripts/rel/format-changelog.sh -b "${from_tag}" -l 'zh' -v "$TAG" > "changes/${TAG}.zh.md"
-    git add changes/"${TAG}".*.md
-    if [ -n "$(git diff --staged --stat)" ]; then
-        git commit -m "docs: Generate changelog for ${TAG}"
-    else
-        logmsg "No changelog update."
-    fi
-}
-
 check_changelog() {
     local file="changes/${TAG}.en.md"
     if [ ! -f  "$file" ]; then
         logerr "Changelog file $file is missing."
+        logerr "Generate it with command: ./scripts/rel/format-changelog.sh -b ${PREV_TAG} -v ${TAG} > ${file}"
         exit 1
     fi
 }
 
-if [ "$DRYRUN" = 'yes' ]; then
-    logmsg "Release tag is ready to be created with command: git tag $TAG"
-else
+check_bpapi() {
+    local fname
     case "$TAG" in
-        *rc*)
-            true
-            ;;
-        *alpha*)
-            true
+        *.0)
+            fname="$(echo "$TAG" | sed 's/^e//; s/\.0$//')"
+            fpath="apps/emqx/test/emqx_static_checks_data/${fname}.bpapi"
+            logmsg "Checking $fpath"
+            if [ ! -f "$fpath" ]; then
+                logerr "BPAPI file missing: $fpath"
+                exit 1
+            fi
             ;;
-        *beta*)
+        *)
             true
             ;;
-        e*)
-            check_changelog
-            ;;
-        v*)
-            generate_changelog
-            ;;
     esac
+}
+
+case "$TAG" in
+    *rc*)
+        true
+        ;;
+    *alpha*)
+        true
+        ;;
+    *beta*)
+        true
+        ;;
+    e*)
+        check_bpapi
+        check_changelog
+        ;;
+    v*)
+        check_changelog
+        ;;
+esac
+
+if [ "$DRYRUN" = 'yes' ]; then
+    logmsg "Release tag is ready to be created with command: git tag $TAG"
+else
     git tag "$TAG"
     logmsg "$TAG is created OK."
     logwarn "Don't forget to push the tag!"

scripts/rel/format-changelog.sh

@@ -57,8 +57,7 @@ case "${LANGUAGE:-}" in
         true
         ;;
     *)
-        logerr "-l|--lang must be 'en' or 'zh'"
-        exit 1
+        LANGUAGE='en'
         ;;
 esac