Merge pull request #12442 from ieQu1/dev/ds-license-apache

Relicense apps/emqx_durable_storage under Apache 2.0

apps/emqx_durable_storage/BSL.txt

@@ -1,94 +0,0 @@
-Business Source License 1.1
-
-Licensor:             Hangzhou EMQ Technologies Co., Ltd.
-Licensed Work:        EMQX Enterprise Edition
-                      The Licensed Work is (c) 2023
-                      Hangzhou EMQ Technologies Co., Ltd.
-Additional Use Grant: Students and educators are granted right to copy,
-                      modify, and create derivative work for research
-                      or education.
-Change Date:          2028-01-26
-Change License:       Apache License, Version 2.0
-
-For information about alternative licensing arrangements for the Software,
-please contact Licensor: https://www.emqx.com/en/contact
-
-Notice
-
-The Business Source License (this document, or the “License”) is not an Open
-Source license. However, the Licensed Work will eventually be made available
-under an Open Source License, as stated in this License.
-
-License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
-“Business Source License” is a trademark of MariaDB Corporation Ab.
-
------------------------------------------------------------------------------
-
-Business Source License 1.1
-
-Terms
-
-The Licensor hereby grants you the right to copy, modify, create derivative
-works, redistribute, and make non-production use of the Licensed Work. The
-Licensor may make an Additional Use Grant, above, permitting limited
-production use.
-
-Effective on the Change Date, or the fourth anniversary of the first publicly
-available distribution of a specific version of the Licensed Work under this
-License, whichever comes first, the Licensor hereby grants you rights under
-the terms of the Change License, and the rights granted in the paragraph
-above terminate.
-
-If your use of the Licensed Work does not comply with the requirements
-currently in effect as described in this License, you must purchase a
-commercial license from the Licensor, its affiliated entities, or authorized
-resellers, or you must refrain from using the Licensed Work.
-
-All copies of the original and modified Licensed Work, and derivative works
-of the Licensed Work, are subject to this License. This License applies
-separately for each version of the Licensed Work and the Change Date may vary
-for each version of the Licensed Work released by Licensor.
-
-You must conspicuously display this License on each original or modified copy
-of the Licensed Work. If you receive the Licensed Work in original or
-modified form from a third party, the terms and conditions set forth in this
-License apply to your use of that work.
-
-Any use of the Licensed Work in violation of this License will automatically
-terminate your rights under this License for the current and all other
-versions of the Licensed Work.
-
-This License does not grant you any right in any trademark or logo of
-Licensor or its affiliates (provided that you may use a trademark or logo of
-Licensor as expressly required by this License).
-
-TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
-AN “AS IS” BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
-EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
-TITLE.
-
-MariaDB hereby grants you permission to use this License’s text to license
-your works, and to refer to it using the trademark “Business Source License”,
-as long as you comply with the Covenants of Licensor below.
-
-Covenants of Licensor
-
-In consideration of the right to use this License’s text and the “Business
-Source License” name and trademark, Licensor covenants to MariaDB, and to all
-other recipients of the licensed work to be provided by Licensor:
-
-1. To specify as the Change License the GPL Version 2.0 or any later version,
-   or a license that is compatible with GPL Version 2.0 or a later version,
-   where “compatible” means that software provided under the Change License can
-   be included in a program with software provided under GPL Version 2.0 or a
-   later version. Licensor may specify additional Change Licenses without
-   limitation.
-
-2. To either: (a) specify an additional grant of rights to use that does not
-   impose any additional restriction on the right granted in this License, as
-   the Additional Use Grant; or (b) insert the text “None”.
-
-3. To specify a Change Date.
-
-4. Not to modify this License in any other way.

apps/emqx_durable_storage/IMPLEMENTATION.md

@@ -1,119 +0,0 @@
-# General concepts
-
-In the logic layer we don't speak about replication.
-This is because we could use an external DB with its own replication logic.
-
-On the other hand, we introduce notion of shard right here at the logic layer.
-This is because shared subscription logic needs to be aware of it to some extend, as it has to split work between subscribers somehow.
-
-
-# Modus operandi
-
-1. Create a draft implementation of a milestone
-2. Test performance
-3. Test consistency
-4. Goto 1
-
-# Tables
-
-## Message storage
-
-Data is written every time a message matching certain pattern is published.
-This pattern is not part of the logic layer spec.
-
-Write throughput: very high
-
-Data size: very high
-
-Write pattern: append only
-
-Read pattern: pseudoserial
-
-Number of records: O(total write throughput * retention time)
-
-
-# Push vs. Pull model
-
-In push model we have replay agents iterating over the dataset in the shards.
-
-In pull model the client processes work with iterators directly and fetch data from the remote message storage instances via remote procedure calls.
-
-## Push pros:
-- Lower latency: message can be dispatched to the client as soon as it's persisted
-- Less worry about buffering
-
-## Push cons:
-- Needs pushback logic
-- It's not entirely justified when working with external DB that may not provide streaming API
-
-## Pull pros:
-- No need for pushback: client advances iterators at its own tempo
-
-## Pull cons:
-- 2 messages need to be sent over network for each batch being replayed.
-- RPC is generally an antipattern
-
-# Invariants
-
-- All messages written to the shard always have larger sequence number than all the iterators for the shard (to avoid missing messages during replay)
-
-
-# Parallel tracks
-
-## 0. Configuration
-
-This includes HOCON schema and an interface module that is used by the rest of the code (`emqx_ds_conf.erl`).
-
-At the early stage we need at least to implement a feature flag that can be used by developers.
-
-We should have safety measures to prevent a client from breaking down the broker by connecting with clean session = false and subscribing to `#`.
-
-## 1. Fully implement all emqx_durable_storage APIs
-
-### Message API
-
-### Session API
-
-### Iterator API
-
-## 2. Implement a filter for messages that has to be persisted
-
-We don't want to persist ALL messages.
-Only the messages that can be replayed by some client.
-
-1. Persistent sessions should signal to the emqx_broker what topic filters should be persisted.
-   Scenario:
-   - Client connects with clean session = false.
-   - Subscribes to the topic filter a/b/#
-   - Now we need to signal to the rest of the broker that messages matching `a/b/#` must be persisted.
-
-2. Replay feature (separate, optional, under BSL license): in the configuration file we have list of topic filters that specify what topics can be replayed. (Lower prio)
-   - Customers can do this: `#`
-   - Include minimum QoS of messages in the config
-
-## 3. Replace current emqx_persistent_session with the emqx_durable_storage
-
-## 4. Garbage collection
-
-## 5. Tooling for performance and consistency testing
-
-At the first stage we can just use emqttb:
-
-https://github.com/emqx/emqttb/blob/master/src/scenarios/emqttb_scenario_persistent_session.erl
-
-
-Consistency verification at the early stages can just use this test suite:
-
-`apps/emqx/test/emqx_persistent_session_SUITE.erl`
-
-## 6. Update rocksdb version in EMQX (low prio)
-
-## 9999. Alternative schema for rocksdb message table
-
-https://github.com/emqx/eip/blob/main/active/0023-rocksdb-message-persistence.md#potential-for-future-optimizations-keyspace-based-on-the-learned-topic-patterns
-
-Problem with the current bitmask-based schema:
-
-- Good: `a/b/c/#`
-
-- Bad: `+/a/b/c/d`

apps/emqx_durable_storage/src/emqx_ds_app.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
 %% Copyright (c) 2020-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 
 -module(emqx_ds_app).

apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
-%% Copyright (c) 2022-2023 EMQ Technologies Co., Ltd. All Rights Reserved.
+%% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 -module(emqx_ds_bitmask_keymapper).
 

apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
 %% Copyright (c) 2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 
 %% @doc Supervisor that contains all the processes that belong to a

apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
 %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 -module(emqx_ds_storage_layer_sup).
 

apps/emqx_durable_storage/src/emqx_ds_sup.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
 %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 -module(emqx_ds_sup).
 

apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
 %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 -module(emqx_ds_storage_bitfield_lts_SUITE).
 

apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl

@@ -1,5 +1,17 @@
 %%--------------------------------------------------------------------
-%% Copyright (c) 2020-2023 EMQ Technologies Co., Ltd. All Rights Reserved.
+%% Copyright (c) 2020-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
 %%--------------------------------------------------------------------
 
 -module(emqx_ds_message_storage_bitmask_shim).

apps/emqx_durable_storage/test/props/payload_gen.erl

@@ -1,3 +1,19 @@
+%%--------------------------------------------------------------------
+%% Copyright (c) 2023-2024 EMQ Technologies Co., Ltd. All Rights Reserved.
+%%
+%% Licensed under the Apache License, Version 2.0 (the "License");
+%% you may not use this file except in compliance with the License.
+%% You may obtain a copy of the License at
+%%
+%%     http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing, software
+%% distributed under the License is distributed on an "AS IS" BASIS,
+%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+%% See the License for the specific language governing permissions and
+%% limitations under the License.
+%%--------------------------------------------------------------------
+
 %% @doc This module provides lazy, composable producer streams that
 %% can be considered counterparts to Archiver's consumer pipes and
 %% therefore can facilitate testing