[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cherry-pick](branch-2.1) pick hive text write from master #40537

Merged
merged 7 commits into from
Sep 27, 2024

Conversation

suxiaogang223
Copy link
Contributor
@suxiaogang223 suxiaogang223 commented Sep 9, 2024

Proposed changes

pick prs:
#38549
#40183
#40315

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@suxiaogang223
Copy link
Contributor Author

run buildall

Copy link
Contributor
@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/exec/decompressor.h Show resolved Hide resolved
be/src/util/block_compression.h Show resolved Hide resolved
@suxiaogang223
Copy link
Contributor Author

run buildall

suxiaogang223 and others added 4 commits September 25, 2024 14:28
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd
## abstract
introduces the pluginization of the Authenticator component, enabling
greater flexibility and modularity within the authentication system. By
refactoring the Authenticator into a pluggable SPI (Service Provider
Interface), we allow for custom authentication mechanisms to be
seamlessly integrated and managed.

## Key Changes
AuthenticatorFactory Interface:

This interface defines the create method, which is used to create an
Authenticator instance based on initialization properties, enabling
support for various authentication mechanisms.
The factoryIdentifier method provides an identifier for the factory,
allowing differentiation between various AuthenticatorFactory
implementations (e.g., LDAP, default).
Implemented Specific Authentication Factories:

We have implemented factories for different authentication mechanisms
(such as LDAP), providing the necessary configuration support for each.


## eg

In your Maven pom.xml, add the fe-core dependency:

```
        <dependency>
            <groupId>org.apache.doris</groupId>
            <artifactId>fe-core</artifactId>
            <version>1.2-SNAPSHOT</version>
            <scope>provided</scope>
        </dependency>

```

```
import org.apache.doris.common.ConfigBase;

public class LocalAuthenticatorFactory implements AuthenticatorFactory {


    @OverRide
    public LocalAuthenticator create(Properties initProps) {
        return new LocalAuthenticator();
    }

    @OverRide
    public String factoryIdentifier() {
        return "local";
    }
}
import java.util.HashMap;
import java.util.Map;

public class LocalAuthenticator implements Authenticator {
    private Map<String, String> userStore = new HashMap<>();

    public LocalAuthenticator() {
        // Hardcoded usernames and passwords
        userStore.put("user1", "password1");
        userStore.put("user2", "password2");
        userStore.put("admin", "adminpass");
    }

    @OverRide
    public boolean authenticate(String username, String password) {
        // Authenticate by checking the hardcoded user store
        String storedPassword = userStore.get(username);
        return storedPassword != null && storedPassword.equals(password);
    }
}




```
To integrate with the SPI (Service Provider Interface) mechanism, we
created a file at
**src/main/resources/META-INF/services/org.apache.doris.mysql.authenticate.AuthenticatorFactory**
containing **org.example.LocalAuthenticatorFactory,** which allows the
system to dynamically load and utilize the LocalAuthenticatorFactory
implementation.



Your plugin needs to include all necessary dependencies, and it is
generally recommended to package them into an Uber JAR (also known as a
Fat JAR) to ensure that the plugin can run independently without
additional dependency management. Once packaged, this JAR can be placed
directly in the **fe/custom_lib** directory.
Impl ZstdDecompressor and support read hive text table which is
compressed by zstd.
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.12% (9345/25873)
Line Coverage: 27.64% (76759/277728)
Region Coverage: 26.42% (39408/149169)
Branch Coverage: 23.20% (20058/86452)
Coverage Report: http://coverage.selectdb-in.cc/coverage/51c2a4c697b2850cc3862a062d04cbd93411d104_51c2a4c697b2850cc3862a062d04cbd93411d104/report/index.html

@suxiaogang223
Copy link
Contributor Author

run p0

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.12% (9350/25885)
Line Coverage: 27.65% (76806/277797)
Region Coverage: 26.42% (39421/149193)
Branch Coverage: 23.21% (20064/86462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ebb6f9a3eddbd1d2a50fcab31a7165fd1f34e21b_ebb6f9a3eddbd1d2a50fcab31a7165fd1f34e21b/report/index.html

@suxiaogang223
Copy link
Contributor Author

run buildall

@suxiaogang223 suxiaogang223 changed the title [cherry-pick](branch-2.1) pick some pr from master [cherry-pick](branch-2.1) pick hive text write from master Sep 27, 2024
Copy link
Contributor
@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 27, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.12% (9350/25887)
Line Coverage: 27.63% (76770/277840)
Region Coverage: 26.42% (39415/149204)
Branch Coverage: 23.20% (20056/86464)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02eaee6246e05eb4f26cb3a8d4891c22ccdb936f_02eaee6246e05eb4f26cb3a8d4891c22ccdb936f/report/index.html

@suxiaogang223
Copy link
Contributor Author

run p0

@morningman morningman merged commit 0b4552f into apache:branch-2.1 Sep 27, 2024
21 of 25 checks passed
@suxiaogang223 suxiaogang223 deleted the hive_text_write21 branch September 28, 2024 02:28
@yiguolei yiguolei mentioned this pull request Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. meta-change reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants