Python

Some Useful Modules

numpy - Scientific calculation

Numpy remove scientific notation¹:

#!/usr/env/bin python
import numpy as np
np.set_printoptions(suppress=True, formatter={
    'float_kind': '{:0.1f}'.format
})

re - Regexp

sqlalchemy - ORM

sqlalchemy	mysql
class/model	table
Column	field
instance	record

seleium - Headless browser

Selenium use specific browser:

from selenium import webdriver

driver_path = "C:/Users/username/PycharmProjects/chromedriver.exe"
brave_path = "C:/Program Files (x86)/BraveSoftware/Brave-Browser/Application/brave.exe"

option = webdriver.ChromeOptions()
option.binary_location = brave_path
# option.add_argument("--incognito") OPTIONAL
# option.add_argument("--headless") OPTIONAL

# Create new Instance of Chrome
browser = webdriver.Chrome(executable_path=driver_path, chrome_options=option)

browser.get("https://www.baidu.com")

pandas

excel process

pandas	excel
DataFrame	work sheet
Series	column
Index	line heading
row	row/line
NaN	empty cell

数据透视表（Pivot Table）是一种交互式的表，可以进行某些计算，如求和与计数等。所进行的计算与数据跟数据透视表中的排列有关。之所以称为数据透视表，是因为可以动态地改变它们的版面布置，以便按照不同方式分析数据，也可以重新安排行号、列标和页字段。每一次改变版面布置时，数据透视表会立即按照新的布置重新计算数据。另外，如果原始数据发生更改，则可以更新数据透视表。

定位数据

loc[] 通过索引名/标签定位单行、单列、多行、多列，前闭后闭

iloc[] 通过索引位置定位单行、单列、多行、多列，前闭后开

at[], iat[] 定位单个的值

多级索引

pd.MultiIndex.from_product([[list1], [list2]])

tensorflow

tensorflow 与 cuda, cudnn 对应关系：https://tensorflow.google.cn/install/source?hl=tr#gpu

tensorflow-gpu 2.4.0 with conda on Windows 10

requirements.txt:

python==3.8
tensorflow-gpu==2.4.0
cudatoolkit==11.0.3
cudnn==8.0.5.39 #(from conda-forge)
# conda search cudnn -c conda-forge
# conda install cudnn==8.0.5.39 -c conda-forge

scipy==1.4.1
numpy==1.19.2
matplotlib==3.2.1
opencv-python==4.2.0.34
tqdm==4.46.1
Pillow==8.2.0
h5py==2.10.0

tensorflow on macOS

pip install tensorflow-macos tensorflow-metal

Dict

Merge dictionary in Python

a, b = {'a': 1}, {'b': 2}

update() Example:

from copy import deepcopy
ab = deepcopy(a)
ab.update(b)

use ** to destruct Example:
```
ab = {**a, **b}
```

itertools Example:

import itertools
ab = dict(itertools.chain(a.items(), b.items()))

ChainMap This method will not update the value of same key. Example:
```
from collections import ChainMap
ab = dict(ChainMap(a, b))
```

dict.items() Example:

ab = dict(a.items() | b.items())
# or
ab = dict(list(a.items() + list(b.items())))

Pythonnic Example:

ab = {k:v for d in [a, b] for k, v in d.items()}

Python 3.9 Example:
```
ab = a | b
```

Information Security

Avoid Directory Traversal Vulnerability² security

#!/usr/env/bin python
import os

app_root = '/data/app'
filename = '../../etc/passwd'

# abspath = os.path.join(app_root, filename)
abspath = os.path.realpath(os.path.join(app_root, filename))

if os.path.commonprefix([app_root, abspath]) != app_root:
    print('403 forbidden')
else:
    print(abspath)

Avoid SSRF³

import ipaddress
import dns.resolver
# Configure the whitelist to check
DOMAINS_WHITELIST = ["baidu.com", "labslinux"]
DNS_RESOLVER = dns.resolver.Resolver()
DNS_RESOLVER.nameservers = ["1.1.1.1"]
def verify_dns_records(domain, records, type):
    error_detected = False
    if records is not None:
        for record in records:
            value = record.to_text().strip()
            try:
                ip = ipaddress.ip_address(value)

                if not ip.is_global:
                    print("[!] DNS record type '%s' for domain name '%s' resolve to
                    a non public IP address '%s'!" % (type, domain, value))
                    error_detected = True
            except ValueError:
                error_detected = True
                print("[!] '%s' is not valid IP address!" % value)
    return error_detected
def check():
    error_detected = False
    for domain in DOMAINS_WHITELIST:
        # Get the IPs of the current domain
        # See https://en.wikipedia.org/wiki/List_of_DNS_record_types
        try:
            # A = IPv4 address record
            ip_v4_records = DNS_RESOLVER.query(domain, "A")
        except Exception as e:
            ip_v4_records = None
            print("[i] Cannot get A record for domain '%s': %s\n" % (domain,e))
        try:
           ip_v6_records = DNS_RESOLVER.query(domain, "AAAA")
        except Exception as e:
            ip_v6_records = None
            print("[i] Cannot get AAAA record for domain '%s': %s\n" % (domain,e))
        # Verify the IPs obtained
        if verify_dns_records(domain, ip_v4_records, "A")
        or verify_dns_records(domain, ip_v6_records, "AAAA"):
            error_detected = True
    return error_detected
if __name__== "__main__":
    if check():
        exit(1)
    else:
        exit(0)

playground

print('Hello world')

Footnotes:

stackoverflow 的回答，官方文档有每个配置项的说明。

streamlit 的补丁

CSDN 博主 oxff644 的文章《SSRF漏洞原理及应用》