emqx4.3/changes/e5.1.1.en.md

e5.1.1

Enhancements

• #10667 The MongoDB connector and bridge have been refactored into a separate app to improve the code structure.
• #11115 Added info logs to indicate when buffered messages are dropped due to time-to-live (TTL) expiration.
• #11133 Renamed deliver_rate to delivery_rate in the configuration of retainer, while being compatible with the previous deliver_rate.
• #11137 Refactored the Dashboard listener configuration to use a nested ssl_options field for SSL settings.
• #11138 Changed the default value of k8s api_server from http://127.0.0.1:9091 to https://kubernetes.default.svc:443.
  • emqx_ctl conf show cluster no longer displays irrelevant configuration items when discovery_strategy=static. Configuration information related to etcd/k8s/dns will not be shown.
  • Removed zones(deprecated config key) from emqx_ctl conf show_keys.
• #11165 Removed the /configs/limiter API from swagger.json. Only the API documentation was removed, and the /configs/limiter API functionalities remain unchanged.
• #11166 Added 3 random SQL functions to the rule engine:
  • random(): Generates a random number between 0 and 1 (0.0 =< X < 1.0).
  • uuid_v4(): Generates a random UUID (version 4) string.
  • uuid_v4_no_hyphen(): Generates a random UUID (version 4) string without hyphens.
• #11180 Added a new configuration API /configs (GET/PUT) that supports reloading the HOCON format configuration file.
• #11226 Unified the listener switch to enable, while being compatible with the previous enabled.
• #11249 Added /license/setting REST API endpoint to read and update licensed connections usage alarm watermark.
• #11251 Added the /cluster/topology REST API endpoint: A GET request to this endpoint returns the cluster topology, showing connections between RLOG core and replicant nodes.
• #11253 The Webhook/HTTP bridge has been refactored into its own Erlang application. This allows for more flexibility in the future and allows the bridge to be run as a standalone application.
• #11079 Added support for custom headers in messages for Kafka bridge producer mode.
• #11132 Added support for MQTT action authorization based on QoS level and Retain flag values. Now, EMQX can verify whether clients have the permission to publish/subscribe using specific QoS levels, and whether they have the permission to publish retained messages.
• #11207 Updated the driver versions of multiple data bridges to enhance security and ensure that sensitive data will not be leaked. This includes:
  • TDengine
  • MongoDB
  • MySQL
  • Clickhouse
• #11241 Schema Registry has been refactored into its own Erlang application. This allows for more flexibility in the future.
• #11020 Upgraded emqtt dependency to prevent sensitive data leakage in the debug log.
• #11135 Improved time offset parser in rule engine and return uniform error codes.
• #11236 Improved the speed of clients querying in REST API /clients endpoint with default parameters.

Bug Fixes

• #11004 Wildcards are no longer allowed for the destination topic in topic rewrite.
• #11026 Addressed an inconsistency in the usage of div and mod operations within the rule engine. Previously, the div' operation could only be used as an infix operation, and mod could only be applied through a function call. Now, both div and mod can be used via function call syntax and infix syntax.
• #11037 When starting an HTTP connector, EMQX now returns a descriptive error in case the system is unable to connect to the remote target system.
• #11039 Fixed database number validation for Redis connector. Previously, negative numbers were accepted as valid database numbers.
• #11074 Fixed a bug to adhere to Protocol spec MQTT-5.0 [MQTT-3.8.3-4].
• #11077 Fixed a crash when updating listener binding with a non-integer port.
• #11094 Fixed an issue where connection errors in Kafka Producer would not be reported when reconnecting the bridge.
• #11103 Updated erlcloud dependency.
• #11106 Added validation for the maximum number of worker_pool_size of a bridge resource. Now the maximum amount is 1024 to avoid large memory consumption from an unreasonable number of workers.
• #11118 Ensured that validation errors in REST API responses are slightly less confusing. Now, if there are out-of-range errors, they will be presented as {"value": 42, "reason": {"expected": "1..10"}, ...}, replacing the previous usage of expected_type with expected.
• #11126 Rule metrics for async mode bridges will set failure counters correctly now.
• #11134 Fixed the value of the uppercase authorization header not being obfuscated in the log.
• #11139 The Redis connector has been refactored into its own Erlang application to improve the code structure.
• #11145 Added several fixes and improvements in Ekka and Mria. Ekka:
  • Improved cluster discovery log messages to consistently describe actual events Ekka PR.
  • Removed deprecated cluster auto-clean configuration parameter (it has been moved to Mria) Ekka PR. Mria:
  • Ping now only runs on replicant nodes. Previously, mria_lb was trying to ping both stopped and running replicant nodes, which could result in timeout errors. Mria PR
  • Used null_copies storage when copying $mria_rlog_sync table. This fix has no effect on EMQX for now, as $mria_rlog_sync is only used in mria:sync_transaction/2,3,4, which is not utilized by EMQX. Mria PR
• #11148 Fixed an issue when nodes tried to synchronize configuration update operations to a node which has already left the cluster.
• #11150 Wait for Mria table when emqx_psk app is being started to ensure that PSK data is synced to replicant nodes even if they don't have init PSK file.
• #11151 The MySQL connector has been refactored into its own Erlang application to improve the code structure.
• #11158 Wait for Mria table when the mnesia backend of retainer starts to avoid a possible error of the retainer when joining a cluster.
• #11162 Fixed an issue in webhook bridge where, in async query mode, HTTP status codes like 4XX and 5XX would be treated as successes in the bridge metrics.
• #11164 Reintroduced support for nested (i.e.: ${payload.a.b.c}) placeholders for extracting data from rule action messages without the need for calling json_decode(payload) first.
• #11172 Fixed the payload field in rule engine SQL being duplicated in the below situations:
  • When using a foreach sentence without the as sub-expression and selecting all fields (using the * or omitting the do sub-expression). For example: FOREACH payload.sensors FROM "t/#"
  • When selecting the payload field and all fields. For example: SELECT payload.sensors, * FROM "t/#"
• #11174 Fixed the encoding of the server key coming from an ingress MQTT bridge. Before the fix, it was encoded as a list of integers corresponding to the ASCII characters of the server string.
• #11184 Config value for mqtt.max_packet_size now has a max value of 256MB as defined by the protocol.
• #11192 Fixed an issue with producing invalid HOCON file when an atom type was used. Also removed unnecessary " around keys and latin1 strings from HOCON file.
• #11195 Fixed an issue where the REST API could create duplicate subscriptions for specified clients of the Stomp gateway.
• #11206 Made the username and password params of CoAP client optional in connection mode.
• #11208 Fixed the issue of abnormal data statistics for LwM2M clients.
• #11211 HTTP API DELETE operations on non-existent resources now consistently returns 404.
• #11214 Fixed a bug where node configuration may fail to synchronize correctly when the node joins the cluster.
• #11229 Fixed an issue that prevented plugins from starting/stopping after changing configuration via emqx ctl conf load.
• #11237 The headers default value in /prometheus API should be a map instead of a list.
• #11250 Fixed a bug when the order of MQTT packets withing a WebSocket packet will be reversed.
• #11271 Ensured that the range of all percentage type configurations is from 0% to 100% in the REST API and configuration. For example, sysom.os.sysmem_high_watermark=101% is invalid now.
• #11272 Fixed a typo in the log, where an abnormal PUBREL packet was mistakenly referred to as pubrec.
• #11281 Restored support for the special $queue/ shared subscription topic prefix.
• #11294 Fixed emqx ctl cluster join, leave, and status commands.
• #11306 Fixed rule action metrics inconsistency where dropped requests were not accounted for.
• #11309 Improved startup order of EMQX applications. Simplified build scripts and improved code reuse.
• #11322 Added support for importing additional configurations from EMQX backup file (emqx ctl import command):
  • rule_engine (previously not imported due to the bug)
  • topic_metrics (previously not implemented)
  • slow_subs (previously not implemented).
• #10645 Changed health check for Oracle Database, PostgreSQL, MySQL and Kafka Producer data bridges to ensure target table/topic exists.
• #11107 MongoDB bridge health check now returns the failure reason.
• #11139 The Redis bridge has been refactored into its own Erlang application to improve the code structure and to make it easier to maintain.
• #11151 The MySQL bridge has been refactored into its own Erlang application to improve the code structure and to make it easier to maintain.
• #11163 Hid topology.pool_size in MondoDB bridges and fixed it to 1 to avoid confusion.
• #11175 Now when using a nonexistent hostname for connecting to MySQL, a 400 error is returned rather than 503 in the REST API.
• #11198 Fixed global rebalance status evaluation on replicant nodes. Previously, /api/v5/load_rebalance/global_status API method could return incomplete results if handled by a replicant node.
• #11223 In InfluxDB bridging, mixing decimals and integers in a field may lead to serialization failure in the Influx Line Protocol, resulting in the inability to write to the InfluxDB bridge (when the decimal point is 0, InfluxDB mistakenly interprets it as an integer). See also: InfluxDB v2.7 Line-Protocol.
• #11225 The username field in PostgreSQL/Timescale/MatrixDB bridges configuration is now a required one.
• #11242 Restarted emqx_ee_schema_registry when a node joins a cluster. As emqx_ee_schema_registry uses Mria tables, a node joining a cluster needs to restart this application in order to start relevant Mria shard processes, ensuring a correct behaviour in Core/Replicant mode.
• #11266 Fixed and improved support for TDengine insert syntax:
  1. Added support for inserting into multi-table in the template. For example: insert into table_1 values (${ts}, '${id}', '${topic}') table_2 values (${ts}, '${id}', '${topic}')
  2. Added support for mixing prefixes/suffixes and placeholders in the template. For example: insert into table_${topic} values (${ts}, '${id}', '${topic}') Note: This is a breaking change. Previously, the values of string type were quoted automatically, but now they must be quoted explicitly. For example: insert into table values (${ts}, '${a_string}')
• #11307 Fixed check for table existence to return a more friendly message in the Oracle bridge.
• #11316 Fixed Pool Size value not being considered in Oracle Bridge.
• #11326 Fixed return error checking on table validation in the Oracle bridge.