DolphinScheduler development

Software Requests

Before setting up the DolphinScheduler development environment, please make sure you have installed the software as below:

Git: DolphinScheduler version control system
JDK: DolphinScheduler backend language, have to use version after JDK 1.8
Maven: Java Package Management System
Node: DolphinScheduler frontend, have to use version after Node 12.20.2

Clone Git Repository

Download the git repository through your git management tool, here we use git-core as an example

mkdir dolphinscheduler
cd dolphinscheduler
git clone git@github.com:apache/dolphinscheduler.git

compile source code

ii. Run mvn clean install -Prelease -Dmaven.test.skip=true

Docker image build

DolphinScheduler will release new Docker images after it released, you could find them in Docker Hub.

If you want to modify DolphinScheduler source code, and build Docker images locally, you can run when finished the modification shell cd dolphinscheduler ./mvnw -B clean package \ -Dmaven.test.skip \ -Dmaven.javadoc.skip \ -Dmaven.checkstyle.skip \ -Ddocker.tag=<TAG> \ -Pdocker,release

When the command is finished you could find them by command docker imaegs.

If you want to modify DolphinScheduler source code, build and push Docker images to your registry ，you can run when finished the modification

cd dolphinscheduler
./mvnw -B clean deploy \
   -Dmaven.test.skip \
   -Dmaven.javadoc.skip \
   -Dmaven.checkstyle.skip \
   -Dmaven.deploy.skip \
   -Ddocker.tag=<TAG> \
   -Ddocker.hub=<HUB_URL> \
   -Pdocker,release

If you want to modify DolphinScheduler source code, and also want to add customize dependencies of Docker image, you can modify the definition of Dockerfile after modifying the source code. You can run the following command to find all Dockerfile files.

cd dolphinscheduler
find . -iname 'Dockerfile'

Then run the Docker build command above

You could create custom Docker images base on those images if you want to change image like add some dependencies or upgrade package.

FROM dolphinscheduler-standalone-server
RUN apt update ; \
    apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \

Note： Docker will build and push linux/amd64,linux/arm64 multi-architecture images by default

Have to use version after Docker 19.03, because after 19.03 docker contains buildx

Notice

There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode

Standalone mode: Recommended，more convenient to build development environment, it can cover most scenes.
Normal mode: Separate server master, worker, api, which can cover more test environments than standalone, and it is more like production environment in real life.

DolphinScheduler Standalone Quick Start

Note: Use standalone server only for development and debugging, because it uses H2 Database as default database and Zookeeper Testing Server which may not be stable in production.

Standalone is only supported in DolphinScheduler 1.3.9 and later versions.

Standalone server is able to connect to external databases like mysql and postgresql, see Standalone Deployment for instructions.

Git Branch Choose

Use different Git branch to develop different codes

If you want to develop based on a binary package, switch git branch to specific release branch, for example, if you want to develop base on 1.3.9, you should choose branch 1.3.9-release.
If you want to develop the latest code, choose branch branch dev.

Start backend server

Find the class org.apache.dolphinscheduler.server.StandaloneServer in Intellij IDEA and clikc run main function to startup.

Start frontend server

Install frontend dependencies and run it.

Note: You can see more detail about the frontend setting in frontend development.

cd dolphinscheduler-ui
pnpm install
pnpm run dev

The browser access address http://localhost:3000 can login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123

DolphinScheduler Normal Mode

Prepare

zookeeper

Download ZooKeeper, and extract it.

Create directory zkData and zkLog
Go to the zookeeper installation directory, copy configure file zoo_sample.cfg to conf/zoo.cfg, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper
```
# We use path /data/zookeeper/data and /data/zookeeper/datalog here as example
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/datalog
```
Run ./bin/zkServer.sh in terminal by command ./bin/zkServer.sh start.

Database

The DolphinScheduler's metadata is stored in relational database. Currently supported MySQL and Postgresql. We use MySQL as an example. Start the database and create a new database named dolphinscheduler as DolphinScheduler metabase

After creating the new database, run the sql file under dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql directly in MySQL to complete the database initialization

Start Backend Server

Following steps will guide how to start the DolphinScheduler backend service

Backend Start Prepare

Open project: Use IDE open the project, here we use Intellij IDEA as an example, after opening it will take a while for Intellij IDEA to complete the dependent download
File change
- If you use MySQL as your metadata database, you need to modify dolphinscheduler/pom.xml and change the scope of the mysql-connector-java dependency to compile. This step is not necessary to use PostgreSQL
- Modify database configuration, modify the database configuration in the dolphinscheduler-master/src/main/resources/application.yaml
- Modify database configuration, modify the database configuration in the dolphinscheduler-worker/src/main/resources/application.yaml
- Modify database configuration, modify the database configuration in the dolphinscheduler-api/src/main/resources/application.yaml

We here use MySQL with database, username, password named dolphinscheduler as an example

   spring:
     datasource:
       driver-class-name: com.mysql.jdbc.Driver
       url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
       username: dolphinscheduler
       password: dolphinscheduler

Log level: add a line <appender-ref ref="STDOUT"/> to the following configuration to enable the log to be displayed on the command line

dolphinscheduler-master/src/main/resources/logback-spring.xml dolphinscheduler-worker/src/main/resources/logback-spring.xml dolphinscheduler-api/src/main/resources/logback-spring.xml

here we add the result after modify as below:

  <root level="INFO">
  +  <appender-ref ref="STDOUT"/>
    <appender-ref ref="APILOGFILE"/>
    <appender-ref ref="SKYWALKING-LOG"/>
  </root>

Note: Only DolphinScheduler 2.0 and later versions need to inatall plugin before start server. It not need before version 2.0.

Server start

There are three services that need to be started, including MasterServer, WorkerServer, ApiApplicationServer.

MasterServer：Execute function main in the class org.apache.dolphinscheduler.server.master.MasterServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-spring.xml -Ddruid.mysql.usePingMethod=false -Dspring.profiles.active=mysql
WorkerServer：Execute function main in the class org.apache.dolphinscheduler.server.worker.WorkerServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-spring.xml -Ddruid.mysql.usePingMethod=false -Dspring.profiles.active=mysql
ApiApplicationServer：Execute function main in the class org.apache.dolphinscheduler.api.ApiApplicationServer by Intellij IDEA, with the configuration VM Options -Dlogging.config=classpath:logback-spring.xml -Dspring.profiles.active=api,mysql. After it started, you could find Open API documentation in http://localhost:12345/dolphinscheduler/doc.html

The mysql in the VM Options -Dspring.profiles.active=mysql means specified configuration file

Start Frontend Server

Install frontend dependencies and run it

cd dolphinscheduler-ui
pnpm install
pnpm run dev