Browse Source

[python] Add missing doc about config and connect remote server (#9443)

which includes `configuration`, `run example`
`how to connect remote server`

close: #9286, #9284, #8917
Jiajie Zhong 3 years ago
parent
commit
aaf2042ec4

+ 96 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst

@@ -0,0 +1,96 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Configuration
+=============
+
+Export Configuration File
+-------------------------
+
+pydolphinscheduler allows you to change the built-in configurations via CLI or editor you like. pydolphinscheduler
+integrated built-in configurations in its package, but you could also export it locally by CLI
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --init
+
+And it will create a new YAML file in the path `~/pydolphinscheduler/config.yaml` by default. If you want to export
+it to another path, you should set `PYDOLPHINSCHEDULER_HOME` before you run command :code:`pydolphinscheduler config --init`.
+
+.. code-block:: bash
+
+    $ export PYDOLPHINSCHEDULER_HOME=<CUSTOM_PATH>
+    $ pydolphinscheduler config --init
+
+After that, your configuration file will export into `<CUSTOM_PATH>/config.yaml` instead of the default path.
+
+Change Configuration
+--------------------
+
+In section `export configuration file`_ you export the configuration file locally, and as a local file, you could
+edit it with any editor you like. After you save your change in your editor, the latest configuration will work
+when you run your workflow code.
+
+You could also query or change the configuration via CLI :code:`config --get <config>` or :code:`config --get <config> <val>`.
+Both `--get` and `--set` could be call one or more times in single command, and you could only set the leaf
+node of the configuration but could get the parent configuration, there are simple examples below:
+
+.. code-block:: bash
+
+    # Get single configuration in the leaf node
+    $ pydolphinscheduler config --get java_gateway.address
+    The configuration query as below:
+
+    java_gateway.address = 127.0.0.1
+
+    # Get multiple configuration in the leaf node
+    $ pydolphinscheduler config --get java_gateway.address --get java_gateway.port
+    The configuration query as below:
+
+    java_gateway.address = 127.0.0.1
+    java_gateway.port = 25333
+
+    # Get parent configuration which contain multiple leaf nodes
+    $ pydolphinscheduler config --get java_gateway
+    The configuration query as below:
+
+    java_gateway = ordereddict([('address', '127.0.0.1'), ('port', 25333), ('auto_convert', True)])
+
+    # Set single configuration
+    $ pydolphinscheduler config --set java_gateway.address 192.168.1.1
+    Set configuration done.
+
+    # Set multiple configuration
+    $ pydolphinscheduler config --set java_gateway.address 192.168.1.1 --set java_gateway.port 25334
+    Set configuration done.
+
+    # Set configuration not in leaf node will fail
+    $ pydolphinscheduler config --set java_gateway 192.168.1.1,25334,True
+    Raise error.
+
+For more information about our CLI, you could see document :doc:`cli`.
+
+All Configurations
+------------------
+
+Here are all our configurations for pydolphinscheduler.
+
+.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml
+   :language: yaml
+   :lines: 18-
+
+

+ 26 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst

@@ -0,0 +1,26 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+How To
+======
+
+In this section 
+
+.. toctree::
+   :maxdepth: 2
+   
+   remote-submit

+ 51 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst

@@ -0,0 +1,51 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Submit Your Code from Different machine
+=======================================
+
+Generally, we use pydolphinscheduler as a client to DolphinScheduler, and consider we may change our workflow
+code frequently, the best practice is running :ref:`python gateway service <start:start python gateway service>`
+in your server machine and submit the workflow code from your development machine, like a laptop or PC. This behavior
+is supported by pydolphinscheduler out of box with one or two single command lines. 
+
+Export Configuration File
+-------------------------
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --init
+
+your could find more detail in :ref:`configuration exporting <config:export configuration file>`
+
+Run API Server in Other Host
+----------------------------
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname>
+
+your could find more detail in :ref:`configuration setting <config:change configuration>`
+
+Run API Server in Other Port
+----------------------------
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --set java_gateway.port <your-python-gateway-service-port>
+
+your could find more detail in :ref:`configuration setting <config:change configuration>`

+ 2 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst

@@ -32,7 +32,9 @@ then go and see :doc:`tutorial` for more detail.
    tutorial
    tutorial
    concept
    concept
    tasks/index
    tasks/index
+   howto/index
    cli
    cli
+   config
    api
    api
 
 
 Indices and tables
 Indices and tables

+ 32 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst

@@ -107,6 +107,37 @@ the server is health if keyword `ApiApplicationServer` in the console.
    yaml config path `python-gateway.enabled : true` in api-server's configuration path in `api-server/conf/application.yaml`.
    yaml config path `python-gateway.enabled : true` in api-server's configuration path in `api-server/conf/application.yaml`.
    The default value is true and Python gateway service start when api server is been started.
    The default value is true and Python gateway service start when api server is been started.
 
 
+Run an Example
+--------------
+
+Before run an example for pydolphinscheduler, you should get the example code from it source code. You could run
+single bash command to get it
+
+.. code-block:: bash
+
+   $ wget https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py
+
+or you could copy-paste the content from `tutorial source code`_. And then you could run the example in your
+terminal
+
+.. code-block:: bash
+
+   $ python tutorial.py
+
+If you want to submit your workflow to a remote API server, which means that your workflow script is different
+from the API server, you should first change pydolphinscheduler configuration and then submit the workflow script
+
+.. code-block:: bash
+
+   $ pydolphinscheduler config --init
+   $ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname>
+   $ python tutorial.py
+
+.. note::
+
+   You could see more information in :doc:`config` about all the configurations pydolphinscheduler supported.
+
+
 What's More
 What's More
 -----------
 -----------
 
 
@@ -117,3 +148,4 @@ maybe you could go and play with all :doc:`tasks/index` *PyDolphinScheduler* sup
 .. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download
 .. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download
 .. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org
 .. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org
 .. _`install Apache DolphinScheduler`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html
 .. _`install Apache DolphinScheduler`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html
+.. _`tutorial source code`: https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py