Hero Image
Golang os/exec usage (notes)

Golang os/exec usage (notes) Add extra environment variables before running a command cmd := exec.Command("programToExecute") additionalEnv := "FOO=bar" newEnv := append(os.Environ(), additionalEnv) cmd.Env = newEnv out, err := cmd.CombinedOutput() if err != nil { log.Fatalf("cmd.Run() failed with %s\n", err) } fmt.Printf("%s", out) Use os/exec with pipes or bash expressions on Linux rcmd := `iw dev | awk '$1=="Interface"{print $2}'` cmd := exec.Command("bash", "-c", rcmd) out, err := cmd.CombinedOutput() if err != nil { log.Println(err.Error()) } log.Println(string(out)) Use os/exec with batch expressions on Windows cmd := exec.Command("cmd", "/c", "ffmpeg -i myfile.mp4 myfile.mp3 && del myfile.mp4") out, err := cmd.CombinedOutput() if err != nil { log.Println(err.Error()) } log.Println(string(out)) Fix local language issues in Windows CMD with os/exec cmd := exec.Command("cmd", "/c", "chcp 65001 && netsh WLAN show drivers") out, err := cmd.CombinedOutput() if err != nil { log.Println(err.Error()) } log.Println(string(out)) When using this method, remember to remove the first line of output (the chcp response) before further processing.

Friday, November 12, 2021 Read
Hero Image
Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management Planning considerations During search, each shard uses one thread. Too many shards can slow things down. A node has multiple thread pools by function. The search (count/search/suggest) pool size is int((# of allocated processors * 3) / 2) + 1 and the default queue_size is 1000. The write (index/delete/update/bulk) pool size is # of allocated processors and the default queue_size is also 1000. (Official docs - Thread pools)

Wednesday, November 10, 2021 Read
Hero Image
Cloudflare Traffic sequence

Traffic sequence Traffic to your application runs through the following sequence on Cloudflare’s edge: DDoS URL Rewrites Page Rules IP Access Rules Bots Firewall Rules Rate Limiting Managed Rules Header Modification Access Workers

Wednesday, November 3, 2021 Read
Hero Image
Regex for Markdown Syntax

Regex for Markdown Syntax basic-syntax # Headings headerOne = `(#{1}\s)(.*)` headeTwo = `(#{2}\s)(.*)` headerThree = `(#{3}\s)(.*)` headerFour = `(#{4}\s)(.*)` headerFive = `(#{5}\s)(.*)` headerSix = `(#{6}\s)(.*)` # Bold and Italic Text boldItalicText = `(\*|\_)+(\S+)(\*|\_)+` # Links linkText = `(\[.*\])(\((http)(?:s)?(\:\/\/).*\))` # Images imageFile = `(\!)(\[(?:.*)?\])(\(.*(\.(jpg|png|gif|tiff|bmp))(?:(\s\"|\')(\w|\W|\d)+(\"|\'))?\)` # Unordered List listText = `(^(\W{1})(\s)(.*)(?:$)?)+` # Numbered Text numberedListText = `(^(\d+\.)(\s)(.*)(?:$)?)+` # Block Quotes blockQuote = `((^(\>{1})(\s)(.*)(?:$)?)+` # Inline Code inlineCode = "(\\`{1})(.*)(\\`{1})" # Code Block codeBlock = "(\\`{3}\\n+)(.*)(\\n+\\`{3})" # Horizontal Line horizontalLine = `(\=|\-|\*){3}` # Email Text emailText = `(\<{1})(\S+@\S+)(\>{1})` # TABLES tableText = `(((\|)([a-zA-Z\d+\s#!@'"():;\\\/.\[\]\^<={$}>?(?!-))]+))+(\|))(?:\n)?((\|)(-+))+(\|)(\n)((\|)(\W+|\w+|\S+))+(\|$)`

Friday, October 29, 2021 Read
Hero Image
Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection. # This will install GUI and make it as a default startup option and then restart the machine. $ sudo yum install xrdp tigervnc-server ~# sudo su ~# passwd ~# systemctl enable --now xrdp ~# netstat -antup | grep xrdp tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN 10202/xrdp tcp 0 0 127.0.0.1:3350 0.0.0.0:* LISTEN 10201/xrdp-sesman

Wednesday, October 20, 2021 Read
Hero Image
[Notes] Build an Ubuntu remote desktop environment on GCP

[Notes] Build an Ubuntu remote desktop environment on GCP Install # dependency sudo apt-get install ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal # VNC Server sudo apt-get install vnc4server # After install, run vncserver and set the password when prompted vncserver Modify ~/.vnc/xstartup #!/bin/sh # Uncomment the following two lines for normal desktop: # unset SESSION_MANAGER # exec /etc/X11/xinit/xinitrc [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources xsetroot -solid grey vncconfig -iconic & x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" & x-window-manager & gnome-panel & gnome-settings-daemon & metacity & nautilus & Execute # Kill the current vncserver and restart vncserver -kill :1 # vncserver defaults to port 5900; :1 is 5901, and so on vncserver :1 Start vncserver on reboot @reboot /usr/bin/vncserver :1

Wednesday, October 20, 2021 Read
Hero Image
Understand the crypto trio: Encode, Encrypt, and Hash

Understand the crypto trio: Encode, Encrypt, and Hash Encoding It does not change the data and has no encryption effect. It simply represents the data in another form. A classic example is Morse code. JavaScript has two useful functions: encodeURI and decodeURI, which encode special characters in URLs (spaces, punctuation, etc.) into URL-safe formats. Base64 encodes binary data into ASCII characters. Huffman Coding is a lossless compression encoding algorithm. In short, it compresses by abbreviating frequently used symbols. Encryption Encryption and decryption require a key. In the simplest Caesar cipher, each letter is shifted by an offset, and that offset is the key for encryption/decryption.

Monday, October 18, 2021 Read
Hero Image
Fix Nginx 500 errors (too many open files, connection)

Fix Nginx 500 errors (too many open files, connection) Nginx 500 errors only show up in logs. Two common cases: socket() failed (24: Too many open files) while connecting to upstream $ sudo su - www-data $ ulimit -n # check current limit (ulimit -a shows all params) 1024 # vim /etc/security/limits.conf # set nofile (max number of open files) # add/modify the following two lines * soft nofile 655360 * hard nofile 655360 ulimit -n # log out and log back in to see the new value 655360 # If ulimit -n is not 655360, run ulimit -n 655360 to force set it # Then verify with ulimit -n or ulimit -Sn (soft) and ulimit -Hn (hard) (or ulimit -a). # Calculate and set from system level lsof | wc -l # count open files sudo vim /etc/sysctl.conf fs.file-max = 3268890 sudo sysctl -p 512 worker_connections are not enough while connecting to upstream # /etc/nginx/nginx.conf worker_connections 10240; # Refer to Nginx CoreModule # worker_processes 2; # worker_rlimit_nofile 10240; # events { # # worker_connections 10240; # } # Increasing Nginx connections can slow down overall speed because php-cgi is not enough. # Adjust as follows. # php-cgi was started with phpfcgid_children="10" and phpfcgid_requests="500" # ab was run on another server, connect via a switch using GBit ethernet # http://till.klampaeckel.de/blog/archives/30-PHP-performance-III-Running-nginx.html # vim /etc/nginx/nginx.conf worker_connections 10240; worker_rlimit_nofile # vim /etc/init.d/php-fcgi PHP_FCGI_CHILDREN=15 PHP_FCGI_MAX_REQUESTS=1000 change to PHP_FCGI_CHILDREN=512 # or 150 and increase gradually, watch MySQL connections PHP_FCGI_MAX_REQUESTS=10240 # The article's phpfcgid_stop() function is good and can be used if needed. # phpfcgid_stop() { # echo "Stopping $name." # pids=`pgrep php-cgi` # pkill php-cgi # wait_for_pids $pids # }

Saturday, October 9, 2021 Read
Hero Image
Super easy! Build a beautiful and handy zsh terminal

Super easy! Build a beautiful and handy zsh terminal Make your terminal stylish and fast with zsh + zim + powerlevel10k Build a 10X Engineer shell environment

Thursday, October 7, 2021 Read
Hero Image
Sign git commits with GPG

Sign git commits with GPG

Tuesday, October 5, 2021 Read
Hero Image
Gitlab-CI Introduction

Gitlab CI Concept Gitlab DevOps GitOps Workflow code push -> pipeline -> stage -> job Design plan -> code -> build -> test -> release -> deploy -> operate -> monitor -> plan Runner Executors Shell VirtualBox Docker Docker Machine Kubernetes Else… References Gitlab CI/CD Gitlab Runner .gitlab-ci.yaml Runner Register gitlab-runner register After register concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "public-shell" url = "https://gitlab.go2cloudten.com/" token = "-mdH9OAOzG5yPsf_AVnW" executor = "shell" [[runners]] name = "public-docker" url = "https://gitlab.go2cloudten.com/" token = "AcEGPPKTS1uuQ_A_qpWy" executor = "docker" [runners.docker] dns = ["192.168.185.5", "192.168.185.6"] tls_verify = false image = "registry.go2cloudten.com/it/office_sop/node:12.13.0" privileged = true disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false shm_size = 0 pull_policy = "if-not-present" volumes = ["/cache"] Repository .gitlab-ci.yaml stages: - domain check-icp: stage: domain image: registry.go2cloudten.com/it/office_sop/icp tags: - docker script: - domains=$(awk -F '|' '{if($6 ~ "Y" && ($7 ~ "West" || $7 ~ "Yuqu")) print $3}' domains-info.md | sed 's/ //g' | sort | uniq) - if [[ "${domains}" == "" ]]; then telegram.sh 'There is no domain in list' ; else telegram.sh 'Start checking ICP.' ; fi - for i in ${domains}; do result=$(checkicp ${i}); if [[ "${result}" == "未备案" ]];then telegram.sh "${i} 未备案"; sleep 1 ;fi;done - telegram.sh 'ICP check completed.' only: - schedules

Friday, September 24, 2021 Read
Hero Image
Common shell scripting tips

Common shell scripting tips Shell scripting built-in commands Shell scripting special symbols

Wednesday, September 22, 2021 Read