No Description

Jay Chung 87a88e3662 [feat] Support set execute type to pydolphinscheduler (#12871) 2 years ago
.github 1347a8f94c [chore] Separate Python API into another repository (#12779) 2 years ago
.idea 5b7a609a0e Add icon in idea (#12829) 2 years ago
.mvn 8c19a7776e [Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557) 3 years ago
deploy 454f1303a0 Add minio as demo storage for helm charts (#12861) 2 years ago
docs 229c554912 [feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2 years ago
dolphinscheduler-alert 065d5caccc Only expose necessary actuator endpoints (#12571) 2 years ago
dolphinscheduler-aop 6c5ab3fb5c [Bug] [dolphinscheduler-aop] aspectj-maven-plugin version undefined (#12756) 2 years ago
dolphinscheduler-api 87a88e3662 [feat] Support set execute type to pydolphinscheduler (#12871) 2 years ago
dolphinscheduler-api-test 08335b1032 [Improvement][Task] Improved way to collect yarn job's appIds (#12197) 2 years ago
dolphinscheduler-bom 43d69ae157 Upgrade clickhouse jdbc driver (#12639) 2 years ago
dolphinscheduler-common 43d69ae157 Upgrade clickhouse jdbc driver (#12639) 2 years ago
dolphinscheduler-dao f249f2b257 [fix-12675]edit workflow related task, workflow's task version change (#12692) 2 years ago
dolphinscheduler-data-quality 43d69ae157 Upgrade clickhouse jdbc driver (#12639) 2 years ago
dolphinscheduler-datasource-plugin 43d69ae157 Upgrade clickhouse jdbc driver (#12639) 2 years ago
dolphinscheduler-dist 43d69ae157 Upgrade clickhouse jdbc driver (#12639) 2 years ago
dolphinscheduler-e2e 08335b1032 [Improvement][Task] Improved way to collect yarn job's appIds (#12197) 2 years ago
dolphinscheduler-master 229c554912 [feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2 years ago
dolphinscheduler-meter d46e4659d9 [Improvement][Style] Fix remaining style errors and wildcard imports (#12412) 2 years ago
dolphinscheduler-microbench 0a6e8af864 [Improvement][Style] Update spotless junit4 check scope (#12450) 2 years ago
dolphinscheduler-registry d46e4659d9 [Improvement][Style] Fix remaining style errors and wildcard imports (#12412) 2 years ago
dolphinscheduler-remote 08335b1032 [Improvement][Task] Improved way to collect yarn job's appIds (#12197) 2 years ago
dolphinscheduler-scheduler-plugin dedff70f90 [Improvement] Merge spi.utils into common.utils (#12502) 2 years ago
dolphinscheduler-service 00b18ed422 [Improve]Remove the code logger.isDebugEnabled() #5918 #10542 (#12738) 2 years ago
dolphinscheduler-spi dedff70f90 [Improvement] Merge spi.utils into common.utils (#12502) 2 years ago
dolphinscheduler-standalone-server 6d79414491 Add instructions on how to configure logging level for debugging purpose (#12699) 2 years ago
dolphinscheduler-task-plugin 229c554912 [feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2 years ago
dolphinscheduler-tools b71d4983dd [Feature-6586][Server]add some ds process definition demo when init (#11759) 2 years ago
dolphinscheduler-ui 229c554912 [feature][task] Add Kubeflow task plugin for MLOps scenario (#12843) 2 years ago
dolphinscheduler-worker a47b4d0672 [Feature][Task] Transfer files between tasks (#12552) 2 years ago
images 2a519ea900 [doc] Replace the dag screenshot of the README. (#10798) 2 years ago
licenses 4bec792160 [Fix][Common] rewrite code generate,fix bit shift (#6914) 3 years ago
script 72e50b25f1 [Improvement][script] Add installPath verification for install.sh (#12523) 2 years ago
style d3f087e29b [feat][ci] Add pre-commit for easy spotless config (#12192) 2 years ago
tools 43d69ae157 Upgrade clickhouse jdbc driver (#12639) 2 years ago
.asf.yaml 20c4fe5e67 [Improvement][Document] Update the project introduction (#12501) 2 years ago
.dlc.json 997b022b7c [doc][ci] Add http header to avoid github 403 in dlc (#12509) 2 years ago
.flake8 1347a8f94c [chore] Separate Python API into another repository (#12779) 2 years ago
.gitattributes d9d070974c Create .gitattributes 6 years ago
.gitignore 5b7a609a0e Add icon in idea (#12829) 2 years ago
.gitmodules 6a02870926 [Feature][style] Add spotless maven plugin for automatic style fix. (#11272) 2 years ago
.licenserc.yaml ad4f344283 [fix][test] Fix flaky test in CI (#12017) 2 years ago
.pre-commit-config.yaml 1347a8f94c [chore] Separate Python API into another repository (#12779) 2 years ago
CONTRIBUTING.md a44817fc46 [Doc][Style] Fix doc format once for all (#12006) 2 years ago
LICENSE 8c19a7776e [Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557) 3 years ago
NOTICE deb1624c3e Update NOTICE year (#7784) 3 years ago
README.md 6896fa9b47 [fix][doc] Dead link for python api (#12848) 2 years ago
README_zh_CN.md a44817fc46 [Doc][Style] Fix doc format once for all (#12006) 2 years ago
lombok.config d9516e2236 add lombok config (#11212) 2 years ago
mvnw 8c19a7776e [Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557) 3 years ago
mvnw.cmd 8c19a7776e [Improve][CI]Upgrade maven-wrapper to 3.1.0 (#8557) 3 years ago
pom.xml 1347a8f94c [chore] Separate Python API into another repository (#12779) 2 years ago

README.md

Dolphin Scheduler Official Website

dolphinscheduler.apache.org

License codecov Quality Gate Status Twitter Follow Slack Status

Stargazers over time

EN doc

Features

Apache DolphinScheduler is the modern data workflow orchestration platform with powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available out of the box

The key features for DolphinScheduler are as follows:

  • Easy to deploy, we provide 4 ways to deploy, such as Standalone deployment,Cluster deployment,Docker / Kubernetes deployment and Rainbond deployment
  • Easy to use, there are four ways to create workflows:

    • Visually, create tasks by dragging and dropping tasks
    • PyDolphinScheduler, Creating workflows via Python API, aka workflow-as-code
    • Yaml definition, mapping yaml into workflow(have to install PyDolphinScheduler currently)
    • Open API, Creating workflows
  • Highly Reliable, DolphinScheduler uses a decentralized multi-master and multi-worker architecture, which naturally supports horizontal scaling and high availability

  • High performance, its performance is N times faster than other orchestration platform and it can support tens of millions of tasks per day

  • Supports multi-tenancy

  • Supports various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, Hive, Spark SQL), Python, Procedure, Sub_Workflow, Http, K8s, Jupyter, MLflow, SageMaker, DVC, Pytorch, Amazon EMR, etc

  • Orchestrating workflows and dependencies, you can pause/stop/recover task any time, failed tasks can be set to automatically retry

  • Visualizing the running state of the task in real-time and seeing the task runtime log

  • What you see is what you get when you edit the task on the UI

  • Backfill can be operated on the UI directly

  • Perfect project, resource, data source-level permission control

  • Displaying workflow history in tree/Gantt chart, as well as statistical analysis on the task status & process status in each workflow

  • Supports internationalization

  • Cloud Native, DolphinScheduler supports orchestrating multi-cloud/data center workflow, and supports custom task type

  • More features waiting for partners to explore

User Interface Screenshots

dag data-source home master workflow-tree

QuickStart in Docker

Please refer the official website document: QuickStart in Docker

QuickStart in Kubernetes

Please refer to the official website document: QuickStart in Kubernetes

How to Build

./mvnw clean install -Prelease

Artifact:

dolphinscheduler-dist/target/apache-dolphinscheduler-${latest.release.version}-bin.tar.gz: Binary package of DolphinScheduler
dolphinscheduler-dist/target/apache-dolphinscheduler-${latest.release.version}-src.tar.gz: Source code package of DolphinScheduler

Get Help

  1. Submit an issue
  2. Join our slack and send your question to channel #general
  3. Send email to users@dolphinscheduler.apache.org or dev@dolphinscheduler.apache.org

Community

You are very welcome to communicate with the developers and users of Dolphin Scheduler. There are two ways to find them:

  1. Join the Slack channel Slack
  2. Follow the Twitter account of DolphinScheduler and get the latest news on time

How to Contribute

The community welcomes everyone to contribute, please refer to this page to find out more: How to contribute.

Thanks

DolphinScheduler is based on a lot of excellent open-source projects, such as Google guava, grpc, netty, quartz, and many open-source projects of Apache and so on. We would like to express our deep gratitude to all the open-source projects used in DolphinScheduler. We hope that we are not only the beneficiaries of open-source, but also give back to the community. Besides, we hope everyone who have the same enthusiasm and passion for open source could join in and contribute to the open-source community

Landscapes



  

DolphinScheduler enriches the CNCF CLOUD NATIVE Landscape.

License

Please refer to the LICENSE file