码迷,mamicode.com
首页 > 其他好文 > 详细

pgspider clickhouse fdw 试用

时间:2020-02-22 14:05:18      阅读:148      评论:0      收藏:0      [点我收藏+]

标签:启动服务   replace   engine   har   http   昨天   span   ppi   click   

昨天有构建过一个clickhouse 的fdw docker 镜像,今天测试下

环境使用docker-compose 运行,clickhouse 同样也是

环境准备

  • docker-compose 环境
version: "3"
services:
    server:
     image: yandex/clickhouse-server
     ports:
     - "8123:8123"
     - "9000:9000"
     - "9009:9009"
     ulimits:
      nproc: 65535
      nofile:
       soft: 262144
       hard: 262144
    client:
      image: yandex/clickhouse-client
      command: [‘--host‘, ‘server‘]
    pgspider:
      image: dalongrong/pgspider:clickhouse
      ports: 
      - "5432:5432"
      environment: 
      - "POSTGRES_PASSWORD=dalong"
  • 启动服务
docker-compose up -d
  • 初始化clickhouse 数据
创建表:
CREATE TABLE wikistat
(
    project String,
    subproject String,
    hits UInt64,
    size UInt64
) ENGINE = Log;
导入数据:
serverhost=192.168.0.103 // replace with your hostid
docker run -i yandex/clickhouse-client --format_csv_delimiter="|" --host ${serverhost} --query="INSERT INTO default.wikistat FORMAT CSV" < ./data/info.csv 

数据内容

 aa|File:Sleeping_lion.jpg|1|8030
 aa|Main_Page|1|78261
 aa|Special:Statistics|1|20493
 aa|Special:WhatLinksHere/File:Crystal_Clear_app_email.png|1|5412
 aa|Special:WhatLinksHere/File:Wikipedia-logo-fr.png|1|5370
 aa|Steward_requests/Bot_status|1|4733
 aa|Translation_teams/ru|1|4718
 aa|User:%E5%8F%B8%E5%BE%92%E4%BC%AF%E9%A2%9C|2|20096
 aa|User:149.62.201.0/24|1|4802
 aa|User:191.101.30.0/24|1|4806
 aa|User:5.172.192.0/21|1|4800
 aa|User:62.76.7.25|1|4788
  • 效果

技术图片

 

 

使用clickhouse fdw

使用已经构建好,同时push dockerhub 的

  • 启用扩展
CREATE EXTENSION clickhousedb_fdw;
  • 创建外部server
CREATE SERVER myserver FOREIGN DATA WRAPPER clickhousedb_fdw OPTIONS(dbname ‘default‘, driver ‘/opt/clickhousedb/libclickhouseodbc.so‘, host ‘server‘);
  • 创建用户映射
CREATE USER MAPPING FOR postgres SERVER myserver;
  • 创建表映射
CREATE FOREIGN TABLE wikistat (
  project text,
  subproject text,
  hits int,
  size int
) SERVER myserver OPTIONS (table_name ‘wikistat‘);
  • 查询
select * from wikistat;
  • 效果

技术图片

 

 

说明

clickhouse 的fdw 是一个不错的工具,但是对于clickhouse 的操作,当前还不是很强大

参考资料

https://github.com/Percona-Lab/clickhousedb_fdw 
https://hub.docker.com/repository/docker/dalongrong/pgspider 
https://github.com/rongfengliang/pgspider-docker 
https://github.com/rongfengliang/clickhouse-fdw-docker-learning

pgspider clickhouse fdw 试用

标签:启动服务   replace   engine   har   http   昨天   span   ppi   click   

原文地址:https://www.cnblogs.com/rongfengliang/p/12344929.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!