Posted on | In

Python Decorator

Why Decorator

試想一個場景，如何在不變動下述程式片段的情況下，在進入Function時，印出Function名稱?

1
2
def foo():
    print('Hello foo!')

Read more »

Posted on | In

Create and Launch Android Virtrual Device via Command

Basic Infomation

  • OS: Ubuntu 16.4 LTS
  • Android System Image: android_24_armeabi-v7a

Introduction

正常步驟下，會使用Android Studio中的AVD(Android Virtrual Device) Manager管理與啓動Android emulator，
但有時需透過Terminal的方式達到相同的目的，例如自動化沙箱

Read more »

Posted on | In

Setup Raspberry Pi Using a Mac

Basic Infomation

  • OS: macOS High Sierra
  • Raspberry Pi device: Raspberry Pi model B, 1GB RAM, 802.11b/g/n, Wireless LAN, Bluetooth 4.1
  • Raspberry OS: Raspbian stretch lite 2018-04-18
  • SD card: SanDisk Ultra 32GB
Read more »

Posted on | In

Spark on YARN Deployment Modes

YARN Client mode

  • ① YARN Client向ResourceManager提交申請
  • ② ResourceManager接收到請求後，在集群中選擇一個NodeManager分配Container，並在Container中啟動ApplicationMaster process
    • Dirver process在YARN client上運行，並初始化Spark Context
  • ③ Spark Context初始完後，與ApplicationMaster進行溝通，透過ApplicationMaster向ResourceManager申請Container，ResourceManager收到請求後，在集群中選擇一個NodeManager分配Container
  • ④ ApplicationMaster通知NodeManager在Container中啟動Spark executor
  • ⑤ Spark executor向Driver註冊，並在之後將自身狀態回報給Driver
  • ⑥ Spark Context將Task分配給Spark executor
  • ⑦ 所有Task執行結束，YARN Client向ResourceManager提交註銷ApplicationMaster
    Read more »

Posted on | In

Launching Spark on YARN

Launching YARN

1
2
3
4
5
[hadoop@testmain ~]$ cd $HADOOP_HOME
[hadoop@testmain hadoop]$ ./sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-testmain.out
localhost: starting nodemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-testmain.out
Read more »