【centos】vmware安装airflow流程-易微帮

文章目录

1.下载系统https://mirrors.aliyun.com/centos/7/isos/x86_64/ 选择DVD2009

2.在VMware中，除了修改安装位置，其它选择默认安装centos7

3.用户名为root,登录

4.网络适配器为桥接模式。

5.使用命令消除显示器警告。

出现了“WARNING! The remote SSH server rejected X11 forwarding request.”，意思是“远程SSH服务器拒绝X11转发请求。

yum install xorg-x11-xauth -y

6.安装anaconda

6.1下载包：

wget https://repo.anaconda.com/archive/Anaconda3-5.3.0-Linux-x86_64.sh

如果wget 命令不存在，先安装wget

yum install wget -y

6.2授予安装包权限并安装

chmod +x Anaconda3-5.3.0-Linux-x86_64.sh
./Anaconda3-5.3.0-Linux-x86_64.sh

缺少依赖安装依赖

yum install -y bzip2

查看隐藏文件

ls -al

末尾配置环境变量并更新

export PATH=/root/anaconda3/bin:$PATH

source ~/.bashrc

查看命令执行成功没

conda -V

7.取消默认进入base命令

conda config --set auto_activate_base false

上面的失败了，在vi ~/.bashrc后输入

source activate airflow

8.换源

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/

conda config --set show_channel_urls yes

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

文件所在/root/.config/pip/pip.conf

conda config --remove channels defaults  ## 移除默认源

https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/linux-64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch

9.安装airflow

conda create -n af python==3.8.12

pip install apache-airflow

airflow db init

google-re2安装失败时

pip install google-re2==1.0

10.启动

airflow webserver -p 8080 -D

关闭防火墙

systemctl stop firewalld

参考的https://blog.csdn.net/xiaoyaGrace/article/details/105701852

访问端口

http://192.168.251.113:8080/

11.创建用户

airflow users  create --role Admin --username admin --email admin --firstname admin --lastname admin --password 7758

启动定时任务示例

airflow scheduler -D

12.启停脚本

#!/bin/bash
case $1 in
"start"){
	echo " --------启动 airflow-------"
	ssh 192.168.251.113 "conda activate af;airflow webserver -p 8080 -D;airflow scheduler -D;conda deactivate"
};;
"stop"){
	echo " --------关闭 airflow-------"
	ps -ef|egrep 'scheduler|airflow-webserver'|grep -v grep|awk '{print $2}'|xargs kill -15
};;
esac

13.（已舍弃，往下看新的）安装mysql。

下载mysql57 mysql-5.7.23-linux-glibc2.12-x86_64.tar.gz

wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.26-linux-glibc2.12-x86_64.tar.gz

解压移动到local改名

tar -xvf mysql-5.7.26-linux-glibc2.12-x86_64.tar.gz 
mv mysql-5.7.26-linux-glibc2.12-x86_64 /usr/local/
mv mysql-5.7.26-linux-glibc2.12-x86_64/ mysql

创建用户，并给数据目录赋予权限

groupadd mysql
useradd -r -g mysql mysql

cd /
mkdir -p data
cd data/
mkdir -p mysql```
chown mysql:mysql -R /data/mysql

配置参数

vim /etc/my.cnf

[mysqld]
bind-address=0.0.0.0
port=3306
user=mysql
basedir=/usr/local/mysql
datadir=/data/mysql
socket=/tmp/mysql.sock
log-error=/data/mysql/mysql.err
pid-file=/data/mysql/mysql.pid
#character config
character_set_server=utf8mb4
symbolic-links=0

初始化mysql

cd /usr/local/mysql/bin/
 ./mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql/ --datadir=/data/mysql/ --user=mysql --initialize

启动mysql，并更改root 密码

service mysql start

找不到mysql命令

 cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql
 ln -s /usr/local/mysql/bin/mysql /usr/bin

检测启动

ps -ef|grep mysql

更改密码

cd /usr/local/mysql/bin
 mysql -u root -p

SET PASSWORD = PASSWORD('1234');

ALTER USER 'root'@'localhost' PASSWORD EXPIRE NEVER;

flush privileges;

如果密码错误，出现ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: YES)。请检查密码

关闭mysql

quit

service mysql stop

14.更改mysql为数据库


pip install mysqlclient==2.0.2

如果出错，手动下载，更改出错的代码，手动安装https://blog.csdn.net/qingdaoyin/article/details/111227604


pip install  pymysql mysql

设置时间戳可为null。并且位置不能配置错，在[mysqld]、[mysqld_safe]之间，最好在[mysqld_safe]前一格

vi /etc/my.cnf
explicit_defaults_for_timestamp=1

连接navicat出错1130，修改host访问权限为所有ip。https://blog.csdn.net/dabao87/article/details/80571877

连接navicat出错2002,关闭防火墙service firewalld stop。https://blog.csdn.net/xiaoyaGrace/article/details/105701852

修改airflow连接数据库的配置。

#修改配置文件。原文链接：https://blog.csdn.net/java31/article/details/121868003

vi /home/san/airflow/airflow.cfg
executor = LocalExecutor
sql_alchemy_conn = mysql://airflow:xxxxxxxx@localhost:3306/airflow?charset=utf8
sql_engine_encoding = utf-8

创建数据库airflow_db

CREATE DATABASE IF NOT EXISTS airflow_db DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

初始化数据库

airflow db init

出错，提示ModuleNotFoundError: No module named ‘MySQLdb’。表示没有客户端
pip 安装失败，因为Exception: Can not find valid pkg-config name.
使用conda安装，可以自动解决依赖问题。

conda install conda install mysqlclient

重新初始化后airflow db init。提示2002找不到文件/tmp/mysql.sock。2002, “Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)”
全局查找文件：

find / -iname  mysql.sock

创建软连接。

ln -s /var/lib/mysql/mysql.sock /tmp/

重新创建用户

airflow users  create --role Admin --username admin --email admin --firstname admin --lastname admin --password 7758

13.安装mysql。

参考https://www.bilibili.com/video/BV1mm411y7RS/
启动安装脚本(需要先授权)。查看脚本可知，帮我们改host为%可以连接了mysql,创建了用户root和密码为root.

chmod -R 777 MySQL

cd MySQL
./install.sh

14.更改mysql为airflow的数据库

【mysql】设置时间戳可为null。并且位置不能配置错，在[mysqld]、[mysqld_safe]之间，最好在[mysqld_safe]前一格

vi /etc/my.cnf
explicit_defaults_for_timestamp=1

永久关闭防火墙,使用navicat创建数据库airflow_db给airflow使用。

vi /root/.bashrc
service firewalld stop

CREATE DATABASE IF NOT EXISTS airflow_db DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

【airflow】配置更改数据库，更改多线程无序执行。#修改配置文件。原文链接：https://blog.csdn.net/java31/article/details/121868003

vi /home/san/airflow/airflow.cfg
executor = LocalExecutor
sql_alchemy_conn = mysql://root:root@localhost:3306/airflow_db?charset=utf8
sql_engine_encoding = utf-8