Airflow 2正式出来了
Airflow 2.0.0, 2020-12-17
https://github.com/apache/airflow/releases/tag/2.0.0
1.新的写dag脚本方式。(使用python注释)
@task
def extract():
return {“1001”: 301.27, “1002”: 433.21, “1003”: 502.22}
2.Fully specified REST API (AIP-32)
更全的 REST API 接口。提供外部调用。
3.Massive Scheduler performance improvements
As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.
更快的启动任务
4.Scheduler is now HA compatible (AIP-15)
调度高可用。(主备)
5.Task Groups (AIP-34)
可以在任务中建子任务(一个任务中包含多个子任务),可以在web UI上钻取操作。
6.Refreshed UI
自动刷新介面任务状态。不用手工老点介面刷新了。
7.Smart Sensors for reduced load from sensors (AIP-17)
更智能化的检查器功能。(例如检查其它dag中的任务),在大量使用检查器时,减小资源使用。
8.Simplified KubernetesExecutor
减化 KubernetesExecutor 执行器。
9.Airflow core and providers: Splitting Airflow into 60+ packages
将airflow 安装包分成多个安装包,可单独下载
10.Configuration
Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.
将配置文件分成多个独立的模块文件。