2015年1月17日 星期六

[Apache Spark][教學] Spark x Docker x ipython Notebook !(二)-Python安裝設定篇


        文章開頭也是先謝謝前輩https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4.在上一回,[Spark][教學] Spark x Docker x ipython Notebook !(一)-Docker + Spark安裝篇,我們從大大那邊抓下來Spark的環境,但是還缺少python的環境,所以這篇文章就是要介紹怎樣在前一個環境中建立python環境.Docker在建立環境時有兩種方式,一種是進入container內,將程式安裝起來後,將建好的container儲存起來使用,另外一種就是用Dockerfile的方式紀錄安裝的過程.因為我Dockerfile還不熟,所以先介紹第一種方式.
        因為我們上回所使用的映像檔是建立在centOS上(透過$ cat/etc/*-release查詢,或透過作者的dockerfile查詢),所以要在centOS中建立Python.


  • 首先我們先更新一下套件,在centOS中,套件管理工具是yum


yum -y update

  • 接著安裝開發套件
    yum groupinstall -y development
  • 安裝其他小工具
    yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel wget gcc-c++
  • 做好準備工作之後就可以用wget來下載python
    wget http://www.python.org/ftp/python/2.7.9/Python-2.7.9.tar.xz
  • 下載完應該會放在根目錄下面,請解壓縮
    tar -xvf Python-2.7.9.tar.xz
  • 接下來就是一些很瑣碎的設定工作
    # Enter the file directory:
    cd Python-2.7.6
    
    # Start the configuration (setting the installation directory)
    # By default files are installed in /usr/local.
    # You can modify the --prefix to modify it (e.g. for $HOME).
    ./configure --prefix=/usr/local  
  • 因為我們是下載python的source soce,所以要build他,build的時候也會一直噴狀態出來,就讓他慢慢噴
    make && make altinstall 

  • 噴完之後要確定有沒有成功:
python2.7 --version
  • 但是這樣只有python,沒有ipython.在安裝ipython之前,要先安裝python的套件管理工具pip!
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
sudo /usr/local/bin/python2.7 ez_setup.py
sudo /usr/local/bin/easy_install-2.7 pip
  • 接著就可以用pip安裝ipython了
pip install ipython[all]
  • 確認有沒有成功
ipython notebook

  • python 和 ipython的設定告一段落,接下來重點來了,ipython notebook是一個web base的介面,container中沒有web瀏覽器,要怎麼開啟notebook?答案是,透過網路連進來,請看下回分曉.
  • 補充一下,如果要讓這個過程更方便,可以直接把指令加到Dockerfile裡面,讓docker在建立image檔案的時候自動幫你完成這些設定喔~~
  • 下篇連結:[Spark][教學] Spark x Docker x ipython Notebook !(三)-Docker網路設定篇


2 則留言:

  1. 謝謝您的教學,不過我在run pip install ipython[all] 的時候發生了一些問題,可否幫忙釐清一下

    ============ Building the readline extension module ============

    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying readline.py -> build/lib.linux-x86_64-2.7
    running egg_info
    writing gnureadline.egg-info/PKG-INFO
    writing top-level names to gnureadline.egg-info/top_level.txt
    writing dependency_links to gnureadline.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'gnureadline.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'gnureadline.egg-info/SOURCES.txt'
    running build_ext
    building 'gnureadline' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/Modules
    creating build/temp.linux-x86_64-2.7/Modules/2.x
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_RL_CALLBACK -DHAVE_RL_CATCH_SIGNAL -DHAVE_RL_COMPLETION_APPEND_CHARACTER -DHAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK -DHAVE_RL_COMPLETION_MATCHES -DHAVE_RL_COMPLETION_SUPPRESS_APPEND -DHAVE_RL_PRE_INPUT_HOOK -I. -I/usr/local/include/python2.7 -c Modules/2.x/readline.c -o build/temp.linux-x86_64-2.7/Modules/2.x/readline.o
    In file included from Modules/2.x/readline.c:31:
    ./readline/readline.h:385: warning: function declaration isn't a prototype
    gcc -pthread -shared build/temp.linux-x86_64-2.7/Modules/2.x/readline.o readline/libreadline.a readline/libhistory.a -lncurses -o build/lib.linux-x86_64-2.7/gnureadline.so
    /usr/bin/ld: cannot find -lncurses
    collect2: ld returned 1 exit status
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
    Command "/usr/local/bin/python2.7 -c "import setuptools, tokenize;__file__='/tmp/pip-build-52Bzcn/gnureadline/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-5sZPUa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-52Bzcn/gnureadline

    回覆刪除
    回覆
    1. /usr/bin/ld: cannot find -lncurses
      這行顯示你有些gcc的套件沒有安裝,這個要看您使用的os版本決定要安裝哪個套件
      可以用關鍵字+OS版本上網搜尋解法

      刪除