|
|
@@ -4,15 +4,11 @@
|
|
|
|
|
|
## Preface
|
|
|
|
|
|
-This document explains the DolphinScheduler application configurations according to DolphinScheduler-1.3.x versions.
|
|
|
+This document explains the DolphinScheduler application configurations.
|
|
|
|
|
|
## Directory Structure
|
|
|
|
|
|
-Currently, all the configuration files are under [conf ] directory.
|
|
|
-Check the following simplified DolphinScheduler installation directories to have a direct view about the position of [conf] directory and configuration files it has.
|
|
|
-This document only describes DolphinScheduler configurations and other topics are not going into.
|
|
|
-
|
|
|
-[Note: the DolphinScheduler (hereinafter called the ‘DS’) .]
|
|
|
+The directory structure of DolphinScheduler is as follows:
|
|
|
|
|
|
```
|
|
|
├── LICENSE
|
|
|
@@ -100,27 +96,13 @@ This document only describes DolphinScheduler configurations and other topics ar
|
|
|
|
|
|
## Configurations in Details
|
|
|
|
|
|
-serial number| service classification| config file|
|
|
|
-|--|--|--|
|
|
|
-1|startup or shutdown DS application|dolphinscheduler-daemon.sh
|
|
|
-2|datasource config properties|datasource.properties
|
|
|
-3|ZooKeeper config properties|zookeeper.properties
|
|
|
-4|common-service[storage] config properties|common.properties
|
|
|
-5|API-service config properties|application-api.properties
|
|
|
-6|master-service config properties|master.properties
|
|
|
-7|worker-service config properties|worker.properties
|
|
|
-8|alert-service config properties|alert.properties
|
|
|
-9|quartz config properties|quartz.properties
|
|
|
-10|DS environment variables configuration script[install/start DS]|install_config.conf
|
|
|
-11|load environment variables configs <br /> [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]|dolphinscheduler_env.sh
|
|
|
-12|services log config files|API-service log config : logback-api.xml <br /> master-service log config : logback-master.xml <br /> worker-service log config : logback-worker.xml <br /> alert-service log config : logback-alert.xml
|
|
|
-
|
|
|
-
|
|
|
-### dolphinscheduler-daemon.sh [startup or shutdown DS application]
|
|
|
-
|
|
|
-dolphinscheduler-daemon.sh is responsible for DS startup and shutdown.
|
|
|
+
|
|
|
+
|
|
|
+### dolphinscheduler-daemon.sh [startup or shutdown DolphinScheduler application]
|
|
|
+
|
|
|
+dolphinscheduler-daemon.sh is responsible for DolphinScheduler startup and shutdown.
|
|
|
Essentially, start-all.sh or stop-all.sh startup and shutdown the cluster via dolphinscheduler-daemon.sh.
|
|
|
-Currently, DS just makes a basic config, remember to config further JVM options based on your practical situation of resources.
|
|
|
+Currently, DolphinScheduler just makes a basic config, remember to config further JVM options based on your practical situation of resources.
|
|
|
|
|
|
Default simplified parameters are:
|
|
|
```bash
|
|
|
@@ -137,321 +119,206 @@ export DOLPHINSCHEDULER_OPTS="
|
|
|
"
|
|
|
```
|
|
|
|
|
|
-> "-XX:DisableExplicitGC" is not recommended due to may lead to memory link (DS dependent on Netty to communicate).
|
|
|
+> "-XX:DisableExplicitGC" is not recommended due to may lead to memory link (DolphinScheduler dependent on Netty to communicate).
|
|
|
|
|
|
-### datasource.properties [datasource config properties]
|
|
|
+### Database connection related configuration
|
|
|
+
|
|
|
+DolphinScheduler uses Spring Hikari to manage database connections, configuration file location:
|
|
|
+
|
|
|
+|Service| Configuration file |
|
|
|
+|--|--|
|
|
|
+|Master Server | `master-server/conf/application.yaml`|
|
|
|
+|Api Server| `api-server/conf/application.yaml`|
|
|
|
+|Worker Server| `worker-server/conf/application.yaml`|
|
|
|
+|Alert Server| `alert-server/conf/application.yaml`|
|
|
|
+
|
|
|
+The default configuration is as follows:
|
|
|
|
|
|
-DS uses Druid to manage database connections and default simplified configs are:
|
|
|
|Parameters | Default value| Description|
|
|
|
|--|--|--|
|
|
|
-spring.datasource.driver-class-name||datasource driver
|
|
|
-spring.datasource.url||datasource connection url
|
|
|
-spring.datasource.username||datasource username
|
|
|
-spring.datasource.password||datasource password
|
|
|
-spring.datasource.initialSize|5| initial connection pool size number
|
|
|
-spring.datasource.minIdle|5| minimum connection pool size number
|
|
|
-spring.datasource.maxActive|5| maximum connection pool size number
|
|
|
-spring.datasource.maxWait|60000| max wait milliseconds
|
|
|
-spring.datasource.timeBetweenEvictionRunsMillis|60000| idle connection check interval
|
|
|
-spring.datasource.timeBetweenConnectErrorMillis|60000| retry interval
|
|
|
-spring.datasource.minEvictableIdleTimeMillis|300000| connections over minEvictableIdleTimeMillis will be collect when idle check
|
|
|
-spring.datasource.validationQuery|SELECT 1| validate connection by running the SQL
|
|
|
-spring.datasource.validationQueryTimeout|3| validate connection timeout[seconds]
|
|
|
-spring.datasource.testWhileIdle|true| set whether the pool validates the allocated connection when a new connection request comes
|
|
|
-spring.datasource.testOnBorrow|true| validity check when the program requests a new connection
|
|
|
-spring.datasource.testOnReturn|false| validity check when the program recalls a connection
|
|
|
-spring.datasource.defaultAutoCommit|true| whether auto commit
|
|
|
-spring.datasource.keepAlive|true| runs validationQuery SQL to avoid the connection closed by pool when the connection idles over minEvictableIdleTimeMillis
|
|
|
-spring.datasource.poolPreparedStatements|true| open PSCache
|
|
|
-spring.datasource.maxPoolPreparedStatementPerConnectionSize|20| specify the size of PSCache on each connection
|
|
|
-
|
|
|
-
|
|
|
-### zookeeper.properties [zookeeper config properties]
|
|
|
+|spring.datasource.driver-class-name| org.postgresql.Driver |datasource driver|
|
|
|
+|spring.datasource.url| jdbc:postgresql://127.0.0.1:5432/dolphinscheduler |datasource connection url|
|
|
|
+|spring.datasource.username|root|datasource username|
|
|
|
+|spring.datasource.password|root|datasource password|
|
|
|
+|spring.datasource.hikari.connection-test-query|select 1|validate connection by running the SQL|
|
|
|
+|spring.datasource.hikari.minimum-idle| 5| minimum connection pool size number|
|
|
|
+|spring.datasource.hikari.auto-commit|true|whether auto commit|
|
|
|
+|spring.datasource.hikari.pool-name|DolphinScheduler|name of the connection pool|
|
|
|
+|spring.datasource.hikari.maximum-pool-size|50| maximum connection pool size number|
|
|
|
+|spring.datasource.hikari.connection-timeout|30000|connection timeout|
|
|
|
+|spring.datasource.hikari.idle-timeout|600000|Maximum idle connection survival time|
|
|
|
+|spring.datasource.hikari.leak-detection-threshold|0|Connection leak detection threshold|
|
|
|
+|spring.datasource.hikari.initialization-fail-timeout|1|Connection pool initialization failed timeout|
|
|
|
+
|
|
|
+Note that DolphinScheduler also supports database configuration through `bin/env/dolphinscheduler_env.sh`.
|
|
|
+
|
|
|
+
|
|
|
+### Zookeeper related configuration
|
|
|
+DolphinScheduler uses Zookeeper for cluster management, fault tolerance, event monitoring and other functions. Configuration file location:
|
|
|
+|Service| Configuration file |
|
|
|
+|--|--|
|
|
|
+|Master Server | `master-server/conf/application.yaml`|
|
|
|
+|Api Server| `api-server/conf/application.yaml`|
|
|
|
+|Worker Server| `worker-server/conf/application.yaml`|
|
|
|
+
|
|
|
+The default configuration is as follows:
|
|
|
|
|
|
|Parameters | Default value| Description|
|
|
|
|--|--|--|
|
|
|
-zookeeper.quorum|localhost:2181| ZooKeeper cluster connection info
|
|
|
-zookeeper.dolphinscheduler.root|/dolphinscheduler| DS is stored under ZooKeeper root directory
|
|
|
-zookeeper.session.timeout|60000| session timeout
|
|
|
-zookeeper.connection.timeout|30000| connection timeout
|
|
|
-zookeeper.retry.base.sleep|100| time to wait between subsequent retries
|
|
|
-zookeeper.retry.max.sleep|30000| maximum time to wait between subsequent retries
|
|
|
-zookeeper.retry.maxtime|10| maximum retry times
|
|
|
-
|
|
|
+|registry.zookeeper.namespace|dolphinscheduler|namespace of zookeeper|
|
|
|
+|registry.zookeeper.connect-string|localhost:2181| the connection string of zookeeper|
|
|
|
+|registry.zookeeper.retry-policy.base-sleep-time|60ms|time to wait between subsequent retries|
|
|
|
+|registry.zookeeper.retry-policy.max-sleep|300ms|maximum time to wait between subsequent retries|
|
|
|
+|registry.zookeeper.retry-policy.max-retries|5|maximum retry times|
|
|
|
+|registry.zookeeper.session-timeout|30s|session timeout|
|
|
|
+|registry.zookeeper.connection-timeout|30s|connection timeout|
|
|
|
+|registry.zookeeper.block-until-connected|600ms|waiting time to block until the connection succeeds|
|
|
|
+|registry.zookeeper.digest|~|digest of zookeeper|
|
|
|
+
|
|
|
+Note that DolphinScheduler also supports zookeeper related configuration through `bin/env/dolphinscheduler_env.sh`.
|
|
|
|
|
|
### common.properties [hadoop、s3、yarn config properties]
|
|
|
|
|
|
-Currently, common.properties mainly configures Hadoop,s3a related configurations.
|
|
|
-| Parameters | Default value | Description |
|
|
|
-|--|--|--|
|
|
|
-data.basedir.path | /tmp/dolphinscheduler | local directory used to store temp files
|
|
|
-resource.storage.type | NONE | type of resource files: HDFS, S3, NONE
|
|
|
-resource.storage.upload.base.path | /dolphinscheduler | storage path of resource files
|
|
|
-resource.aws.access.key.id | minioadmin | access key id of S3
|
|
|
-resource.aws.secret.access.key | minioadmin | secret access key of S3
|
|
|
-resource.aws.region |us-east-1 | region of S3
|
|
|
-resource.aws.s3.bucket.name | dolphinscheduler | bucket name of S3
|
|
|
-resource.aws.s3.endpoint | http://minio:9000 | endpoint of S3
|
|
|
-resource.hdfs.root.user | hdfs | configure users with corresponding permissions if storage type is HDFS
|
|
|
-resource.hdfs.fs.defaultFS | hdfs://mycluster:8020 | If resource.storage.type=S3, then the request url would be similar to 's3a://dolphinscheduler'. Otherwise if resource.storage.type=HDFS and hadoop supports HA, copy core-site.xml and hdfs-site.xml into 'conf' directory
|
|
|
-hadoop.security.authentication.startup.state | false | whether hadoop grant kerberos permission
|
|
|
-java.security.krb5.conf.path | /opt/krb5.conf | kerberos config directory
|
|
|
-login.user.keytab.username | hdfs-mycluster@ESZ.COM | kerberos username
|
|
|
-login.user.keytab.path | /opt/hdfs.headless.keytab | kerberos user keytab
|
|
|
-kerberos.expire.time | 2 | kerberos expire time,integer,the unit is hour
|
|
|
-yarn.resourcemanager.ha.rm.ids | | specify the yarn resourcemanager url. if resourcemanager supports HA, input HA IP addresses (separated by comma), or input null for standalone
|
|
|
-yarn.application.status.address | http://ds1:8088/ws/v1/cluster/apps/%s | keep default if ResourceManager supports HA or not use ResourceManager, or replace ds1 with corresponding hostname if ResourceManager in standalone mode
|
|
|
-dolphinscheduler.env.path | env/dolphinscheduler_env.sh | load environment variables configs [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
|
|
|
-development.state | false | specify whether in development state
|
|
|
-task.resource.limit.state | false | specify whether in resource limit state
|
|
|
-
|
|
|
-
|
|
|
-### application-api.properties [API-service log config]
|
|
|
+Currently, common.properties mainly configures Hadoop,s3a related configurations. Configuration file location:
|
|
|
|
|
|
-|Parameters | Default value| Description|
|
|
|
+|Service| Configuration file |
|
|
|
+|--|--|
|
|
|
+|Master Server | `master-server/conf/common.properties`|
|
|
|
+|Api Server| `api-server/conf/common.properties`|
|
|
|
+|Worker Server| `worker-server/conf/common.properties`|
|
|
|
+|Alert Server| `alert-server/conf/common.properties`|
|
|
|
+
|
|
|
+The default configuration is as follows:
|
|
|
+
|
|
|
+| Parameters | Default value | Description |
|
|
|
|--|--|--|
|
|
|
-server.port|12345|api service communication port
|
|
|
-server.servlet.session.timeout|7200|session timeout
|
|
|
-server.servlet.context-path|/dolphinscheduler | request path
|
|
|
-spring.servlet.multipart.max-file-size|1024MB| maximum file size
|
|
|
-spring.servlet.multipart.max-request-size|1024MB| maximum request size
|
|
|
-server.jetty.max-http-post-size|5000000| jetty maximum post size
|
|
|
-spring.messages.encoding|UTF-8| message encoding
|
|
|
-spring.jackson.time-zone|GMT+8| time zone
|
|
|
-spring.messages.basename|i18n/messages| i18n config
|
|
|
-security.authentication.type|PASSWORD| authentication type
|
|
|
-security.authentication.ldap.user.admin|read-only-admin|admin user account when you log-in with LDAP
|
|
|
-security.authentication.ldap.urls|ldap://ldap.forumsys.com:389/|LDAP urls
|
|
|
-security.authentication.ldap.base-dn|dc=example,dc=com|LDAP base dn
|
|
|
-security.authentication.ldap.username|cn=read-only-admin,dc=example,dc=com|LDAP username
|
|
|
-security.authentication.ldap.password|password|LDAP password
|
|
|
-security.authentication.ldap.user.identity-attribute|uid|LDAP user identity attribute
|
|
|
-security.authentication.ldap.user.email-attribute|mail|LDAP user email attribute
|
|
|
-security.authentication.ldap.user.not-exist-action|CREATE|action when LDAP user is not exist. Default CREATE: automatically create user when user not exist, DENY: deny log-in when user not exist
|
|
|
-traffic.control.global.switch|false|traffic control global switch
|
|
|
-traffic.control.max-global-qps-rate|300|global max request number per second
|
|
|
-traffic.control.tenant-switch|false|traffic control tenant switch
|
|
|
-traffic.control.default-tenant-qps-rate|10|default tenant max request number per second
|
|
|
-traffic.control.customize-tenant-qps-rate||customize tenant max request number per second
|
|
|
-
|
|
|
-### master.properties [master-service log config]
|
|
|
+|data.basedir.path | /tmp/dolphinscheduler | local directory used to store temp files|
|
|
|
+|resource.storage.type | NONE | type of resource files: HDFS, S3, NONE|
|
|
|
+|resource.upload.path | /dolphinscheduler | storage path of resource files|
|
|
|
+|aws.access.key.id | minioadmin | access key id of S3|
|
|
|
+|aws.secret.access.key | minioadmin | secret access key of S3|
|
|
|
+|aws.region | us-east-1 | region of S3|
|
|
|
+|aws.s3.endpoint | http://minio:9000 | endpoint of S3|
|
|
|
+|hdfs.root.user | hdfs | configure users with corresponding permissions if storage type is HDFS|
|
|
|
+|fs.defaultFS | hdfs://mycluster:8020 | If resource.storage.type=S3, then the request url would be similar to 's3a://dolphinscheduler'. Otherwise if resource.storage.type=HDFS and hadoop supports HA, copy core-site.xml and hdfs-site.xml into 'conf' directory|
|
|
|
+|hadoop.security.authentication.startup.state | false | whether hadoop grant kerberos permission|
|
|
|
+|java.security.krb5.conf.path | /opt/krb5.conf | kerberos config directory|
|
|
|
+|login.user.keytab.username | hdfs-mycluster@ESZ.COM | kerberos username|
|
|
|
+|login.user.keytab.path | /opt/hdfs.headless.keytab | kerberos user keytab|
|
|
|
+|kerberos.expire.time | 2 | kerberos expire time,integer,the unit is hour|
|
|
|
+|yarn.resourcemanager.ha.rm.ids | 192.168.xx.xx,192.168.xx.xx | specify the yarn resourcemanager url. if resourcemanager supports HA, input HA IP addresses (separated by comma), or input null for standalone|
|
|
|
+|yarn.application.status.address | http://ds1:8088/ws/v1/cluster/apps/%s | keep default if ResourceManager supports HA or not use ResourceManager, or replace ds1 with corresponding hostname if ResourceManager in standalone mode|
|
|
|
+|development.state | false | specify whether in development state|
|
|
|
+|resource.manager.httpaddress.port | 8088 | the port of resource manager|
|
|
|
+|yarn.job.history.status.address | http://ds1:19888/ws/v1/history/mapreduce/jobs/%s | job history status url of yarn|
|
|
|
+|datasource.encryption.enable | false | whether to enable datasource encryption|
|
|
|
+|datasource.encryption.salt | !@#$%^&* | the salt of the datasource encryption|
|
|
|
+|data-quality.jar.name | dolphinscheduler-data-quality-dev-SNAPSHOT.jar | the jar of data quality|
|
|
|
+|support.hive.oneSession | false | specify whether hive SQL is executed in the same session|
|
|
|
+|sudo.enable | true | whether to enable sudo|
|
|
|
+|alert.rpc.port | 50052 | the RPC port of Alert Server|
|
|
|
+|zeppelin.rest.url | http://localhost:8080 | the RESTful API url of zeppelin|
|
|
|
+
|
|
|
+
|
|
|
+### Api-server related configuration
|
|
|
+Location: `api-server/conf/application.yaml`
|
|
|
|
|
|
|Parameters | Default value| Description|
|
|
|
|--|--|--|
|
|
|
-master.listen.port|5678|master listen port
|
|
|
-master.exec.threads|100|master-service execute thread number, used to limit the number of process instances in parallel
|
|
|
-master.exec.task.num|20|defines the number of parallel tasks for each process instance of the master-service
|
|
|
-master.dispatch.task.num|3|defines the number of dispatch tasks for each batch of the master-service
|
|
|
-master.host.selector|LowerWeight|master host selector, to select a suitable worker to run the task, optional value: random, round-robin, lower weight
|
|
|
-master.heartbeat.interval|10|master heartbeat interval, the unit is second
|
|
|
-master.task.commit.retryTimes|5|master commit task retry times
|
|
|
-master.task.commit.interval|1000|master commit task interval, the unit is millisecond
|
|
|
-master.max.cpuload.avg|-1|master max CPU load avg, only higher than the system CPU load average, master server can schedule. default value -1: the number of CPU cores * 2
|
|
|
-master.reserved.memory|0.3|master reserved memory, only lower than system available memory, master server can schedule. default value 0.3, the unit is G
|
|
|
-
|
|
|
-
|
|
|
-### worker.properties [worker-service log config]
|
|
|
+|server.port|12345|api service communication port|
|
|
|
+|server.servlet.session.timeout|120m|session timeout|
|
|
|
+|server.servlet.context-path|/dolphinscheduler/ |request path|
|
|
|
+|spring.servlet.multipart.max-file-size|1024MB|maximum file size|
|
|
|
+|spring.servlet.multipart.max-request-size|1024MB|maximum request size|
|
|
|
+|server.jetty.max-http-post-size|5000000|jetty maximum post size|
|
|
|
+|spring.banner.charset|UTF-8|message encoding|
|
|
|
+|spring.jackson.time-zone|UTC|time zone|
|
|
|
+|spring.jackson.date-format|"yyyy-MM-dd HH:mm:ss"|time format|
|
|
|
+|spring.messages.basename|i18n/messages|i18n config|
|
|
|
+|security.authentication.type|PASSWORD|authentication type|
|
|
|
+|security.authentication.ldap.user.admin|read-only-admin|admin user account when you log-in with LDAP|
|
|
|
+|security.authentication.ldap.urls|ldap://ldap.forumsys.com:389/|LDAP urls|
|
|
|
+|security.authentication.ldap.base.dn|dc=example,dc=com|LDAP base dn|
|
|
|
+|security.authentication.ldap.username|cn=read-only-admin,dc=example,dc=com|LDAP username|
|
|
|
+|security.authentication.ldap.password|password|LDAP password|
|
|
|
+|security.authentication.ldap.user.identity.attribute|uid|LDAP user identity attribute|
|
|
|
+|security.authentication.ldap.user.email.attribute|mail|LDAP user email attribute|
|
|
|
+
|
|
|
+### Master Server related configuration
|
|
|
+Location: `master-server/conf/application.yaml`
|
|
|
|
|
|
|Parameters | Default value| Description|
|
|
|
|--|--|--|
|
|
|
-worker.listen.port|1234|worker-service listen port
|
|
|
-worker.exec.threads|100|worker-service execute thread number, used to limit the number of task instances in parallel
|
|
|
-worker.heartbeat.interval|10|worker-service heartbeat interval, the unit is second
|
|
|
-worker.max.cpuload.avg|-1|worker max CPU load avg, only higher than the system CPU load average, worker server can be dispatched tasks. default value -1: the number of CPU cores * 2
|
|
|
-worker.reserved.memory|0.3|worker reserved memory, only lower than system available memory, worker server can be dispatched tasks. default value 0.3, the unit is G
|
|
|
-worker.groups|default|worker groups separated by comma, e.g., 'worker.groups=default,test' <br> worker will join corresponding group according to this config when startup
|
|
|
-worker.tenant.auto.create|true|tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true.
|
|
|
-worker.tenant.distributed.user|false|Scenes to be used for distributed users.For example,users created by FreeIpa are stored in LDAP.This parameter only applies to Linux, When this parameter is true, worker.tenant.auto.create has no effect and will not automatically create tenants.
|
|
|
-
|
|
|
-### alert.properties [alert-service log config]
|
|
|
+|master.listen-port|5678|master listen port|
|
|
|
+|master.fetch-command-num|10|the number of commands fetched by master|
|
|
|
+|master.pre-exec-threads|10|master prepare execute thread number to limit handle commands in parallel|
|
|
|
+|master.exec-threads|100|master execute thread number to limit process instances in parallel|
|
|
|
+|master.dispatch-task-number|3|master dispatch task number per batch|
|
|
|
+|master.host-selector|lower_weight|master host selector to select a suitable worker, default value: LowerWeight. Optional values include random, round_robin, lower_weight|
|
|
|
+|master.heartbeat-interval|10|master heartbeat interval, the unit is second|
|
|
|
+|master.task-commit-retry-times|5|master commit task retry times|
|
|
|
+|master.task-commit-interval|1000|master commit task interval, the unit is millisecond|
|
|
|
+|master.state-wheel-interval|5|time to check status|
|
|
|
+|master.max-cpu-load-avg|-1|master max CPU load avg, only higher than the system CPU load average, master server can schedule. default value -1: the number of CPU cores * 2|
|
|
|
+|master.reserved-memory|0.3|master reserved memory, only lower than system available memory, master server can schedule. default value 0.3, the unit is G|
|
|
|
+|master.failover-interval|10|failover interval, the unit is minute|
|
|
|
+|master.kill-yarn-job-when-task-failover|true|whether to kill yarn job when failover taskInstance|
|
|
|
+
|
|
|
+
|
|
|
+### Worker Server related configuration
|
|
|
+Location: `worker-server/conf/application.yaml`
|
|
|
|
|
|
|Parameters | Default value| Description|
|
|
|
|--|--|--|
|
|
|
-alert.type|EMAIL|alter type|
|
|
|
-mail.protocol|SMTP|mail server protocol
|
|
|
-mail.server.host|xxx.xxx.com|mail server host
|
|
|
-mail.server.port|25|mail server port
|
|
|
-mail.sender|xxx@xxx.com|mail sender email
|
|
|
-mail.user|xxx@xxx.com|mail sender email name
|
|
|
-mail.passwd|111111|mail sender email password
|
|
|
-mail.smtp.starttls.enable|true|specify mail whether open tls
|
|
|
-mail.smtp.ssl.enable|false|specify mail whether open ssl
|
|
|
-mail.smtp.ssl.trust|xxx.xxx.com|specify mail ssl trust list
|
|
|
-xls.file.path|/tmp/xls|mail attachment temp storage directory
|
|
|
-||following configure WeCom[optional]|
|
|
|
-enterprise.wechat.enable|false|specify whether enable WeCom
|
|
|
-enterprise.wechat.corp.id|xxxxxxx|WeCom corp id
|
|
|
-enterprise.wechat.secret|xxxxxxx|WeCom secret
|
|
|
-enterprise.wechat.agent.id|xxxxxxx|WeCom agent id
|
|
|
-enterprise.wechat.users|xxxxxxx|WeCom users
|
|
|
-enterprise.wechat.token.url|https://qyapi.weixin.qq.com/cgi-bin/gettoken? <br /> corpid=$corpId&corpsecret=$secret|WeCom token url
|
|
|
-enterprise.wechat.push.url|https://qyapi.weixin.qq.com/cgi-bin/message/send? <br /> access_token=$token|WeCom push url
|
|
|
-enterprise.wechat.user.send.msg||send message format
|
|
|
-enterprise.wechat.team.send.msg||group message format
|
|
|
-plugin.dir|/Users/xx/your/path/to/plugin/dir|plugin directory
|
|
|
-
|
|
|
-
|
|
|
-### quartz.properties [quartz config properties]
|
|
|
+|worker.listen-port|1234|worker-service listen port|
|
|
|
+|worker.exec-threads|100|worker-service execute thread number, used to limit the number of task instances in parallel|
|
|
|
+|worker.heartbeat-interval|10|worker-service heartbeat interval, the unit is second|
|
|
|
+|worker.host-weight|100|worker host weight to dispatch tasks|
|
|
|
+|worker.tenant-auto-create|true|tenant corresponds to the user of the system, which is used by the worker to submit the job. If system does not have this user, it will be automatically created after the parameter worker.tenant.auto.create is true.|
|
|
|
+|worker.max-cpu-load-avg|-1|worker max CPU load avg, only higher than the system CPU load average, worker server can be dispatched tasks. default value -1: the number of CPU cores * 2|
|
|
|
+|worker.reserved-memory|0.3|worker reserved memory, only lower than system available memory, worker server can be dispatched tasks. default value 0.3, the unit is G|
|
|
|
+|worker.groups|default|worker groups separated by comma, e.g., 'worker.groups=default,test' <br> worker will join corresponding group according to this config when startup|
|
|
|
+|worker.alert-listen-host|localhost|the alert listen host of worker|
|
|
|
+|worker.alert-listen-port|50052|the alert listen port of worker|
|
|
|
+
|
|
|
+### Alert Server related configuration
|
|
|
+Location: `alert-server/conf/application.yaml`
|
|
|
|
|
|
-This part describes quartz configs and configure them based on your practical situation and resources.
|
|
|
|Parameters | Default value| Description|
|
|
|
|--|--|--|
|
|
|
-org.quartz.jobStore.driverDelegateClass | org.quartz.impl.jdbcjobstore.StdJDBCDelegate |
|
|
|
-org.quartz.jobStore.driverDelegateClass | org.quartz.impl.jdbcjobstore.PostgreSQLDelegate |
|
|
|
-org.quartz.scheduler.instanceName | DolphinScheduler |
|
|
|
-org.quartz.scheduler.instanceId | AUTO |
|
|
|
-org.quartz.scheduler.makeSchedulerThreadDaemon | true |
|
|
|
-org.quartz.jobStore.useProperties | false |
|
|
|
-org.quartz.threadPool.class | org.quartz.simpl.SimpleThreadPool |
|
|
|
-org.quartz.threadPool.makeThreadsDaemons | true |
|
|
|
-org.quartz.threadPool.threadCount | 25 |
|
|
|
-org.quartz.threadPool.threadPriority | 5 |
|
|
|
-org.quartz.jobStore.class | org.quartz.impl.jdbcjobstore.JobStoreTX |
|
|
|
-org.quartz.jobStore.tablePrefix | QRTZ_ |
|
|
|
-org.quartz.jobStore.isClustered | true |
|
|
|
-org.quartz.jobStore.misfireThreshold | 60000 |
|
|
|
-org.quartz.jobStore.clusterCheckinInterval | 5000 |
|
|
|
-org.quartz.jobStore.acquireTriggersWithinLock|true |
|
|
|
-org.quartz.jobStore.dataSource | myDs |
|
|
|
-org.quartz.dataSource.myDs.connectionProvider.class | org.apache.dolphinscheduler.service.quartz.DruidConnectionProvider |
|
|
|
-
|
|
|
-
|
|
|
-### install_config.conf [DS environment variables configuration script[install or start DS]]
|
|
|
-
|
|
|
-install_config.conf is a bit complicated and is mainly used in the following two places.
|
|
|
-* DS Cluster Auto Installation.
|
|
|
-
|
|
|
-> System will load configs in the install_config.conf and auto-configure files below, based on the file content when executing 'install.sh'.
|
|
|
-> Files such as dolphinscheduler-daemon.sh, datasource.properties, zookeeper.properties, common.properties, application-api.properties, master.properties, worker.properties, alert.properties, quartz.properties, etc.
|
|
|
-
|
|
|
-* Startup and Shutdown DS Cluster.
|
|
|
-
|
|
|
-> The system will load masters, workers, alert-server, API-servers and other parameters inside the file to startup or shutdown DS cluster.
|
|
|
-
|
|
|
-#### File Content
|
|
|
-
|
|
|
-```bash
|
|
|
-
|
|
|
-# Note: please escape the character if the file contains special characters such as `.*[]^${}\+?|()@#&`.
|
|
|
-# eg: `[` escape to `\[`
|
|
|
-
|
|
|
-# Database type (DS currently only supports PostgreSQL and MySQL)
|
|
|
-dbtype="mysql"
|
|
|
-
|
|
|
-# Database url and port
|
|
|
-dbhost="192.168.xx.xx:3306"
|
|
|
-
|
|
|
-# Database name
|
|
|
-dbname="dolphinscheduler"
|
|
|
-
|
|
|
-
|
|
|
-# Database username
|
|
|
-username="xx"
|
|
|
+|server.port|50053|the port of Alert Server|
|
|
|
+|alert.port|50052|the port of alert|
|
|
|
|
|
|
-# Database password
|
|
|
-password="xx"
|
|
|
|
|
|
-# ZooKeeper url
|
|
|
-zkQuorum="192.168.xx.xx:2181,192.168.xx.xx:2181,192.168.xx.xx:2181"
|
|
|
+### Quartz related configuration
|
|
|
|
|
|
-# DS installation path, such as '/data1_1T/dolphinscheduler'
|
|
|
-installPath="/data1_1T/dolphinscheduler"
|
|
|
-
|
|
|
-# Deployment user
|
|
|
-# Note: Deployment user needs 'sudo' privilege and has rights to operate HDFS.
|
|
|
-# Root directory must be created by the same user if using HDFS, otherwise permission related issues will be raised.
|
|
|
-deployUser="dolphinscheduler"
|
|
|
-
|
|
|
-
|
|
|
-# Followings are alert-service configs
|
|
|
-# Mail server host
|
|
|
-mailServerHost="smtp.exmail.qq.com"
|
|
|
-
|
|
|
-# Mail server port
|
|
|
-mailServerPort="25"
|
|
|
-
|
|
|
-# Mail sender
|
|
|
-mailSender="xxxxxxxxxx"
|
|
|
-
|
|
|
-# Mail user
|
|
|
-mailUser="xxxxxxxxxx"
|
|
|
-
|
|
|
-# Mail password
|
|
|
-mailPassword="xxxxxxxxxx"
|
|
|
-
|
|
|
-# Whether mail supports TLS
|
|
|
-starttlsEnable="true"
|
|
|
-
|
|
|
-# Whether mail supports SSL. Note: starttlsEnable and sslEnable cannot both set true.
|
|
|
-sslEnable="false"
|
|
|
-
|
|
|
-# Mail server host, same as mailServerHost
|
|
|
-sslTrust="smtp.exmail.qq.com"
|
|
|
-
|
|
|
-# Specify which resource upload function to use for resources storage, such as sql files. And supported options are HDFS, S3 and NONE. HDFS for upload to HDFS and NONE for not using this function.
|
|
|
-resourceStorageType="NONE"
|
|
|
-
|
|
|
-# if S3, write S3 address. HA, for example: s3a://dolphinscheduler，
|
|
|
-# Note: s3 make sure to create the root directory /dolphinscheduler
|
|
|
-defaultFS="hdfs://mycluster:8020"
|
|
|
-
|
|
|
-# If parameter 'resourceStorageType' is S3, following configs are needed:
|
|
|
-s3Endpoint="http://192.168.xx.xx:9010"
|
|
|
-s3AccessKey="xxxxxxxxxx"
|
|
|
-s3SecretKey="xxxxxxxxxx"
|
|
|
-
|
|
|
-# If ResourceManager supports HA, then input master and standby node IP or hostname, eg: '192.168.xx.xx,192.168.xx.xx'. Or else ResourceManager run in standalone mode, please set yarnHaIps="" and "" for not using yarn.
|
|
|
-yarnHaIps="192.168.xx.xx,192.168.xx.xx"
|
|
|
-
|
|
|
-
|
|
|
-# If ResourceManager runs in standalone, then set ResourceManager node ip or hostname, or else remain default.
|
|
|
-singleYarnIp="yarnIp1"
|
|
|
-
|
|
|
-# Storage path when using HDFS/S3
|
|
|
-resourceUploadPath="/dolphinscheduler"
|
|
|
-
|
|
|
-
|
|
|
-# HDFS/S3 root user
|
|
|
-hdfsRootUser="hdfs"
|
|
|
-
|
|
|
-# Followings are Kerberos configs
|
|
|
-
|
|
|
-# Specify Kerberos enable or not
|
|
|
-kerberosStartUp="false"
|
|
|
-
|
|
|
-# Kdc krb5 config file path
|
|
|
-krb5ConfPath="$installPath/conf/krb5.conf"
|
|
|
-
|
|
|
-# Keytab username
|
|
|
-keytabUserName="hdfs-mycluster@ESZ.COM"
|
|
|
-
|
|
|
-# Username keytab path
|
|
|
-keytabPath="$installPath/conf/hdfs.headless.keytab"
|
|
|
-
|
|
|
-
|
|
|
-# API-service port
|
|
|
-apiServerPort="12345"
|
|
|
-
|
|
|
-
|
|
|
-# All hosts deploy DS
|
|
|
-ips="ds1,ds2,ds3,ds4,ds5"
|
|
|
-
|
|
|
-# Ssh port, default 22
|
|
|
-sshPort="22"
|
|
|
-
|
|
|
-# Master service hosts
|
|
|
-masters="ds1,ds2"
|
|
|
-
|
|
|
-# All hosts deploy worker service
|
|
|
-# Note: Each worker needs to set a worker group name and default name is "default"
|
|
|
-workers="ds1:default,ds2:default,ds3:default,ds4:default,ds5:default"
|
|
|
+This part describes quartz configs and configure them based on your practical situation and resources.
|
|
|
|
|
|
-# Host deploy alert-service
|
|
|
-alertServer="ds3"
|
|
|
+|Service| Configuration file |
|
|
|
+|--|--|
|
|
|
+|Master Server | `master-server/conf/application.yaml`|
|
|
|
+|Api Server| `api-server/conf/application.yaml`|
|
|
|
+
|
|
|
+The default configuration is as follows:
|
|
|
+
|
|
|
+|Parameters | Default value|
|
|
|
+|--|--|
|
|
|
+|spring.quartz.properties.org.quartz.threadPool.threadPriority | 5|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.isClustered | true|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.class | org.quartz.impl.jdbcjobstore.JobStoreTX|
|
|
|
+|spring.quartz.properties.org.quartz.scheduler.instanceId | AUTO|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.tablePrefix | QRTZ_|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.acquireTriggersWithinLock|true|
|
|
|
+|spring.quartz.properties.org.quartz.scheduler.instanceName | DolphinScheduler|
|
|
|
+|spring.quartz.properties.org.quartz.threadPool.class | org.quartz.simpl.SimpleThreadPool|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.useProperties | false|
|
|
|
+|spring.quartz.properties.org.quartz.threadPool.makeThreadsDaemons | true|
|
|
|
+|spring.quartz.properties.org.quartz.threadPool.threadCount | 25|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.misfireThreshold | 60000|
|
|
|
+|spring.quartz.properties.org.quartz.scheduler.makeSchedulerThreadDaemon | true|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.driverDelegateClass | org.quartz.impl.jdbcjobstore.PostgreSQLDelegate|
|
|
|
+|spring.quartz.properties.org.quartz.jobStore.clusterCheckinInterval | 5000|
|
|
|
|
|
|
-# Host deploy API-service
|
|
|
-apiServers="ds1"
|
|
|
-```
|
|
|
|
|
|
### dolphinscheduler_env.sh [load environment variables configs]
|
|
|
|
|
|
@@ -491,11 +358,11 @@ export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
|
|
|
export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH
|
|
|
```
|
|
|
|
|
|
-### Services logback configs
|
|
|
+### Log related configuration
|
|
|
|
|
|
-Services name| logback config name |
|
|
|
---|--|
|
|
|
-API-service logback config |logback-api.xml|
|
|
|
-master-service logback config|logback-master.xml |
|
|
|
-worker-service logback config|logback-worker.xml |
|
|
|
-alert-service logback config|logback-alert.xml |
|
|
|
+|Service| Configuration file |
|
|
|
+|--|--|
|
|
|
+|Master Server | `master-server/conf/logback-spring.xml`|
|
|
|
+|Api Server| `api-server/conf/logback-spring.xml`|
|
|
|
+|Worker Server| `worker-server/conf/logback-spring.xml`|
|
|
|
+|Alert Server| `alert-server/conf/logback-spring.xml`|
|
rickchengx