Browse Source

[python] Add missing doc about config and connect remote server (#9443)

which includes `configuration`, `run example`
`how to connect remote server`

close: #9286, #9284, #8917
Jiajie Zhong 3 years ago
parent
commit
aaf2042ec4

+ 96 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst

@@ -0,0 +1,96 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Configuration
+=============
+
+Export Configuration File
+-------------------------
+
+pydolphinscheduler allows you to change the built-in configurations via CLI or editor you like. pydolphinscheduler
+integrated built-in configurations in its package, but you could also export it locally by CLI
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --init
+
+And it will create a new YAML file in the path `~/pydolphinscheduler/config.yaml` by default. If you want to export
+it to another path, you should set `PYDOLPHINSCHEDULER_HOME` before you run command :code:`pydolphinscheduler config --init`.
+
+.. code-block:: bash
+
+    $ export PYDOLPHINSCHEDULER_HOME=<CUSTOM_PATH>
+    $ pydolphinscheduler config --init
+
+After that, your configuration file will export into `<CUSTOM_PATH>/config.yaml` instead of the default path.
+
+Change Configuration
+--------------------
+
+In section `export configuration file`_ you export the configuration file locally, and as a local file, you could
+edit it with any editor you like. After you save your change in your editor, the latest configuration will work
+when you run your workflow code.
+
+You could also query or change the configuration via CLI :code:`config --get <config>` or :code:`config --get <config> <val>`.
+Both `--get` and `--set` could be call one or more times in single command, and you could only set the leaf
+node of the configuration but could get the parent configuration, there are simple examples below:
+
+.. code-block:: bash
+
+    # Get single configuration in the leaf node
+    $ pydolphinscheduler config --get java_gateway.address
+    The configuration query as below:
+
+    java_gateway.address = 127.0.0.1
+
+    # Get multiple configuration in the leaf node
+    $ pydolphinscheduler config --get java_gateway.address --get java_gateway.port
+    The configuration query as below:
+
+    java_gateway.address = 127.0.0.1
+    java_gateway.port = 25333
+
+    # Get parent configuration which contain multiple leaf nodes
+    $ pydolphinscheduler config --get java_gateway
+    The configuration query as below:
+
+    java_gateway = ordereddict([('address', '127.0.0.1'), ('port', 25333), ('auto_convert', True)])
+
+    # Set single configuration
+    $ pydolphinscheduler config --set java_gateway.address 192.168.1.1
+    Set configuration done.
+
+    # Set multiple configuration
+    $ pydolphinscheduler config --set java_gateway.address 192.168.1.1 --set java_gateway.port 25334
+    Set configuration done.
+
+    # Set configuration not in leaf node will fail
+    $ pydolphinscheduler config --set java_gateway 192.168.1.1,25334,True
+    Raise error.
+
+For more information about our CLI, you could see document :doc:`cli`.
+
+All Configurations
+------------------
+
+Here are all our configurations for pydolphinscheduler.
+
+.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml
+   :language: yaml
+   :lines: 18-
+
+

+ 26 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst

@@ -0,0 +1,26 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+How To
+======
+
+In this section 
+
+.. toctree::
+   :maxdepth: 2
+   
+   remote-submit

+ 51 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst

@@ -0,0 +1,51 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Submit Your Code from Different machine
+=======================================
+
+Generally, we use pydolphinscheduler as a client to DolphinScheduler, and consider we may change our workflow
+code frequently, the best practice is running :ref:`python gateway service <start:start python gateway service>`
+in your server machine and submit the workflow code from your development machine, like a laptop or PC. This behavior
+is supported by pydolphinscheduler out of box with one or two single command lines. 
+
+Export Configuration File
+-------------------------
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --init
+
+your could find more detail in :ref:`configuration exporting <config:export configuration file>`
+
+Run API Server in Other Host
+----------------------------
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname>
+
+your could find more detail in :ref:`configuration setting <config:change configuration>`
+
+Run API Server in Other Port
+----------------------------
+
+.. code-block:: bash
+
+    $ pydolphinscheduler config --set java_gateway.port <your-python-gateway-service-port>
+
+your could find more detail in :ref:`configuration setting <config:change configuration>`

+ 2 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst

@@ -32,7 +32,9 @@ then go and see :doc:`tutorial` for more detail.
    tutorial
    concept
    tasks/index
+   howto/index
    cli
+   config
    api
 
 Indices and tables

+ 32 - 0
dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst

@@ -107,6 +107,37 @@ the server is health if keyword `ApiApplicationServer` in the console.
    yaml config path `python-gateway.enabled : true` in api-server's configuration path in `api-server/conf/application.yaml`.
    The default value is true and Python gateway service start when api server is been started.
 
+Run an Example
+--------------
+
+Before run an example for pydolphinscheduler, you should get the example code from it source code. You could run
+single bash command to get it
+
+.. code-block:: bash
+
+   $ wget https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py
+
+or you could copy-paste the content from `tutorial source code`_. And then you could run the example in your
+terminal
+
+.. code-block:: bash
+
+   $ python tutorial.py
+
+If you want to submit your workflow to a remote API server, which means that your workflow script is different
+from the API server, you should first change pydolphinscheduler configuration and then submit the workflow script
+
+.. code-block:: bash
+
+   $ pydolphinscheduler config --init
+   $ pydolphinscheduler config --set java_gateway.address <your-api-server-ip-or-hostname>
+   $ python tutorial.py
+
+.. note::
+
+   You could see more information in :doc:`config` about all the configurations pydolphinscheduler supported.
+
+
 What's More
 -----------
 
@@ -117,3 +148,4 @@ maybe you could go and play with all :doc:`tasks/index` *PyDolphinScheduler* sup
 .. _`instructions for all platforms here`: https://wiki.python.org/moin/BeginnersGuide/Download
 .. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org
 .. _`install Apache DolphinScheduler`: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html
+.. _`tutorial source code`: https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py