DataCenterCDC-DS/dolphinscheduler-python/pydolphinscheduler/docs/source/concept.rst

.. Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.

Concepts
========

In this section, you would know the core concepts of *PyDolphinScheduler*.

Process Definition
------------------

Process definition describe the whole things except `tasks`_ and `tasks dependence`_, which including
name, schedule interval, schedule start time and end time. You would know scheduler 

Process definition could be initialized in normal assign statement or in context manger.

.. code-block:: python

   # Initialization with assign statement
   pd = ProcessDefinition(name="my first process definition")

   # Or context manger 
   with ProcessDefinition(name="my first process definition") as pd:
       pd.submit()

Process definition is the main object communicate between *PyDolphinScheduler* and DolphinScheduler daemon.
After process definition and task is be declared, you could use `submit` and `run` notify server your definition.

If you just want to submit your definition and create workflow, without run it, you should use attribute `submit`.
But if you want to run the workflow after you submit it, you could use attribute `run`.

.. code-block:: python

   # Just submit definition, without run it
   pd.submit()
   
   # Both submit and run definition
   pd.run()

Schedule
~~~~~~~~

We use parameter `schedule` determine the schedule interval of workflow, *PyDolphinScheduler* support seven
asterisks expression, and each of the meaning of position as below

.. code-block:: text

    * * * * * * *
    ┬ ┬ ┬ ┬ ┬ ┬ ┬
    │ │ │ │ │ │ │
    │ │ │ │ │ │ └─── year
    │ │ │ │ │ └───── day of week (0 - 7) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
    │ │ │ │ └─────── month (1 - 12)
    │ │ │ └───────── day of month (1 - 31)
    │ │ └─────────── hour (0 - 23)
    │ └───────────── min (0 - 59)
    └─────────────── second (0 - 59)

Here we add some example crontab:

- `0 0 0 * * ? *`: Workflow execute every day at 00:00:00.
- `10 2 * * * ? *`: Workflow execute hourly day at ten pass two.
- `10,11 20 0 1,2 * ? *`: Workflow execute first and second day of month at 00:20:10 and 00:20:11.

Tenant
~~~~~~

Tenant is the user who run task command in machine or in virtual machine. it could be assign by simple string.

.. code-block:: python

   # 
   pd = ProcessDefinition(name="process definition tenant", tenant="tenant_exists")

.. note::

   Make should tenant exists in target machine, otherwise it will raise an error when you try to run command

Tasks
-----

Task is the minimum unit running actual job, and it is nodes of DAG, aka directed acyclic graph. You could define
what you want to in the task. It have some required parameter to make uniqueness and definition.

Here we use :py:meth:`pydolphinscheduler.tasks.Shell` as example, parameter `name` and `command` is required and must be provider. Parameter
`name` set name to the task, and parameter `command` declare the command you wish to run in this task.

.. code-block:: python

   # We named this task as "shell", and just run command `echo shell task`
   shell_task = Shell(name="shell", command="echo shell task")

If you want to see all type of tasks, you could see :doc:`tasks/index`.

Tasks Dependence
~~~~~~~~~~~~~~~~

You could define many tasks in on single `Process Definition`_. If all those task is in parallel processing,
then you could leave them alone without adding any additional information. But if there have some tasks should
not be run unless pre task in workflow have be done, we should set task dependence to them. Set tasks dependence
have two mainly way and both of them is easy. You could use bitwise operator `>>` and `<<`, or task attribute 
`set_downstream` and `set_upstream` to do it.

.. code-block:: python

   # Set task1 as task2 upstream
   task1 >> task2
   # You could use attribute `set_downstream` too, is same as `task1 >> task2`
   task1.set_downstream(task2)
   
   # Set task1 as task2 downstream
   task1 << task2
   # It is same as attribute `set_upstream`
   task1.set_upstream(task2)
   
   # Beside, we could set dependence between task and sequence of tasks,
   # we set `task1` is upstream to both `task2` and `task3`. It is useful
   # for some tasks have same dependence.
   task1 >> [task2, task3]

Task With Process Definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In most of data orchestration cases, you should assigned attribute `process_definition` to task instance to
decide workflow of task. You could set `process_definition` in both normal assign or in context manger mode

.. code-block:: python

   # Normal assign, have to explicit declaration and pass `ProcessDefinition` instance to task
   pd = ProcessDefinition(name="my first process definition")
   shell_task = Shell(name="shell", command="echo shell task", process_definition=pd)

   # Context manger, `ProcessDefinition` instance pd would implicit declaration to task
   with ProcessDefinition(name="my first process definition") as pd:
       shell_task = Shell(name="shell", command="echo shell task",

With both `Process Definition`_, `Tasks`_  and `Tasks Dependence`_, we could build a workflow with multiple tasks.