Duang～ 网络上有道常见题目：从输入 URL 到网页显示，其间发生了什么？
正好最近在整理笔记。准备从【安全】技术角度梳理 URL 相关知识
仅仅记录自己所知，如果有错误，请指出
更细节的东西请各位大佬自行处理
域名这种东西放在平时都是挑自己喜欢的去买就好
如果放在安全领域，需要考虑的就是域名生成算法
DGA（domain generate algorithm）是一种伪随机域名生成算法，可批量生成大量的伪随机域名，作为C&C（command and control server）域名。
域名生成算法经常被用作恶意软件连接 C2 中控。恶意软件定期使用 DGA 算法生成为随机域名，有效绕过黑名单检测，尝试连接，寻找 C2 中控。
通常域名生成算法分为四类：
比如几次恶意软件感染事件均使用了DGA域名：
传统的检测方法：黑名单库
现有的检测方法：长短期记忆网络 LSTM
参考资料：
想要访问特定资源，首先要判断相应资源所处目录（路径）、所处服务器（域名 + 端口）、通信方式（协议）
使用 urllib.parse 进行解析最为方便
URL 一般结构
scheme://netloc/path;parameters?query#fragment
本文主要考虑
域名是为了让人们更方便的记住所需要的资源位置，而便于计算机记录的是 IP 地址。这时就需要对 DNS 服务器进行访问，通过 ARP 协议获取对应 IP
参考链接：
564K9s2c8@1M7s2y4Q4x3@1q4Q4x3V1k6Q4x3V1k6T1L8r3!0Y4i4K6u0W2j5%4y4V1L8W2)9J5k6h3&6W2N6q4)9J5c8Y4N6W2K9i4S2A6L8W2)9#2k6U0b7@1y4U0l9@1y4e0b7I4i4K6u0r3j5i4u0@1K9h3y4D9k6g2)9J5c8X3c8W2N6r3q4A6L8s2y4Q4x3V1j5I4x3e0f1&6x3o6V1I4x3K6m8Q4x3@1k6K6M7r3#2Q4x3@1b7I4x3o6l9I4i4K6u0W2x3U0l9I4y4q4)9J5k6e0x3H3x3o6q4Q4x3X3f1#2y4e0l9I4
TCP 会使用三次握手，这里我们只需知道三次握手在流量包中的显示特征即可
主要关注 Socket 客户端的 Java 实现
参考链接：
d38K9s2c8@1M7s2y4Q4x3@1q4Q4x3V1k6Q4x3V1k6V1k6i4k6W2L8r3!0H3k6i4u0Q4x3X3g2S2L8X3c8J5L8$3W2V1i4K6u0W2j5$3!0E0i4K6u0r3M7X3g2X3k6i4u0W2L8X3y4W2i4K6u0r3K9X3q4$3j5g2)9J5c8X3&6W2N6q4)9J5c8W2y4G2j5$3E0W2N6l9`.`.
使用 HTTP 协议的通信量占大头
其中 HTTPS 中 TLS 会使用四次握手，这里我们只需知道四次握手在流量包中的显示特征即可
客户端传输内容主要使用系统内置 API 和 HTTP 框架
HTTPClient
Apache-HttpClient
OkHttp
Retrofit
参考链接：
fbaK9s2c8@1M7s2y4Q4x3@1q4Q4x3V1k6Q4x3V1k6%4N6%4N6Q4x3X3g2@1N6$3W2D9K9h3!0Q4x3X3g2U0L8$3#2Q4x3V1k6T1L8r3!0Y4i4K6u0r3y4g2)9J5k6s2N6S2P5i4y4Q4x3X3c8@1L8#2)9J5k6r3#2S2K9$3g2Q4x3X3c8Z5N6s2c8H3i4K6u0V1M7X3g2I4N6h3g2K6N6s2y4Q4x3X3c8A6L8W2)9J5k6r3A6S2N6X3p5`.
举个 okkttp 验证服务器证书的调用栈
参考链接：
696K9s2c8@1M7s2y4Q4x3@1q4Q4x3V1k6Q4x3V1k6V1k6i4k6W2L8r3!0H3k6i4u0Q4x3X3g2S2L8X3c8J5L8$3W2V1i4K6u0W2j5$3!0E0i4K6u0r3N6s2u0S2K9h3&6A6L8X3N6Q4x3V1k6S2M7Y4c8A6j5$3I4W2M7#2)9J5c8Y4y4W2j5%4g2J5K9i4c8&6i4K6u0V1M7%4y4D9i4K6y4r3K9r3I4Q4x3@1c8*7K9q4)9J5k6r3y4F1
43fK9s2c8@1M7s2y4Q4x3@1q4Q4x3V1k6Q4x3V1k6%4N6%4N6Q4x3X3g2Z5N6h3q4%4k6h3W2U0L8r3!0#2k6q4)9J5k6h3y4G2L8g2)9J5c8X3q4J5N6r3W2U0L8r3g2K6i4K6u0r3x3K6g2X3k6e0f1#2j5$3f1K6y4K6l9I4x3$3b7%4x3o6j5I4z5o6S2X3j5h3c8T1z5o6j5@1y4r3g2X3y4K6y4Q4x3X3g2Z5N6r3#2D9
下载流程一般使用 Requests 库
进行一般过滤的单线程 demo
使用 wget 实现多层级网页内容爬取
仅爬取一层的 demo
使用 Selenium WebDriver 实现网页截图
基于 Chrome 的 demo
参考资源：
在移动应用分析过程中，往往都是脱壳抓包分析一条龙，抓包是必不可少的内容
可用工具：
一般网页流量均可以使用这些工具，主要讨论移动端应用，原理基本可以理解为中间人攻击
以下统称为代理
根据通信方式
HTTPS 认证
仅服务器端证书校验
将代理的根证书置于手机中，便可以正常抓包
注：根据系统版本的不同可能需要将证书置于用户层或系统层
客户端服务器双向认证
获取 App 客户端证书，导入代理中，便可正常使用
其他用处
我们平时用到域名和端口的机会还是很多的，不提一般资源的获取，平时远程访问也会用到
CTF
像在 CTF 中，有时会用到自构造 HTTP 首部，所以搞清楚常用首部内容还是可以加速解题的
举个小：
重点关注内容：Cookie & Session
常用媒体类型
内容编码类型
由于对 URL 解析之后参数格式防御不充分，则可能会使得各种姿势的 SQL 注入等攻击手段奏效
其他场景需求
单独使用 URL 分析作防御手段的场景
抱住大佬
不同的环境对应不同的需求，数据传输方式要求工具应更新换代
目前针对移动应用抓包的最好用的办法就是肉丝大佬开发的
r0capture
优点众多，我觉得比较重要的
总结
简直是妙蛙种子吃妙脆角，妙到家了
目前在学习证书格式相关内容，学完再写点小笔记
书籍：
HTTP 笔记
HTTP & HTTPS
HTTP 各版本差异
Content-Type 列表：
传输数据方式
socket/WebSocket/WebService/http/https
import java.net.Socket;
// 获取 IP 端口号
mSocket = new Socket(mIpAddress,mClientPort);
// 获取输入输出流
mOutStream = mSocket.getOutputStream();
mInStream = mSocket.getInputStream();
// 发送获取相应信息
mOutStream.write(msg.getBytes());
mOutStream.flush();
byte[] buffer = new byte[1024];
int count = mInStream.read(buffer);
// 关闭通信连接
mOutStream.close();
mInStream.close();
mSocket.close();
import java.net.Socket;
// 获取 IP 端口号
mSocket = new Socket(mIpAddress,mClientPort);
// 获取输入输出流
mOutStream = mSocket.getOutputStream();
mInStream = mSocket.getInputStream();
// 发送获取相应信息
mOutStream.write(msg.getBytes());
mOutStream.flush();
byte[] buffer = new byte[1024];
int count = mInStream.read(buffer);
// 关闭通信连接
mOutStream.close();
mInStream.close();
mSocket.close();
// Create a neat value object to hold the URL
URL url = new URL("https://bbs.pediy.com");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("accept", "application/json");
InputStream responseStream = connection.getInputStream();
ObjectMapper mapper = new ObjectMapper();
APOD apod = mapper.readValue(responseStream, APOD.class);
System.out.println(apod.title);
// Create a neat value object to hold the URL
URL url = new URL("https://bbs.pediy.com");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("accept", "application/json");
InputStream responseStream = connection.getInputStream();
ObjectMapper mapper = new ObjectMapper();
APOD apod = mapper.readValue(responseStream, APOD.class);
System.out.println(apod.title);
var client = HttpClient.newHttpClient();
var request = HttpRequest.newBuilder(
URI.create("https://bbs.pediy.com"))
.header("accept", "application/json")
.build();
var response = client.send(request, new JsonBodyHandler<>(APOD.class));
System.out.println(response.body().get().title);
var client = HttpClient.newHttpClient();
var request = HttpRequest.newBuilder(
URI.create("https://bbs.pediy.com"))
.header("accept", "application/json")
.build();
var response = client.send(request, new JsonBodyHandler<>(APOD.class));
System.out.println(response.body().get().title);
ObjectMapper mapper = new ObjectMapper();
try (CloseableHttpClient client = HttpClients.createDefault()) {
HttpGet request = new HttpGet("https://bbs.pediy.com");
APOD response = client.execute(request, httpResponse ->
mapper.readValue(httpResponse.getEntity().getContent(), APOD.class));
System.out.println(response.title);
}
ObjectMapper mapper = new ObjectMapper();
try (CloseableHttpClient client = HttpClients.createDefault()) {
HttpGet request = new HttpGet("https://bbs.pediy.com");
APOD response = client.execute(request, httpResponse ->
mapper.readValue(httpResponse.getEntity().getContent(), APOD.class));
System.out.println(response.title);
}
ObjectMapper mapper = new ObjectMapper();
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("https://bbs.pediy.com")
.build();
Response response = client.newCall(request).execute();
APOD apod = mapper.readValue(response.body().byteStream(), APOD.class);
System.out.println(apod.title);
ObjectMapper mapper = new ObjectMapper();
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("https://bbs.pediy.com")
.build();
Response response = client.newCall(request).execute();
APOD apod = mapper.readValue(response.body().byteStream(), APOD.class);
System.out.println(apod.title);
Retrofit retrofit = new Retrofit.Builder()
.baseUrl("https://bbs.pediy.com")
.addConverterFactory(JacksonConverterFactory.create())
.build();
APODClient apodClient = retrofit.create(APODClient.class);
CompletableFuture<APOD> response = apodClient.getApod("DEMO_KEY");
APOD apod = response.get();
System.out.println(apod.title);
Retrofit retrofit = new Retrofit.Builder()
.baseUrl("https://bbs.pediy.com")
.addConverterFactory(JacksonConverterFactory.create())
.build();
APODClient apodClient = retrofit.create(APODClient.class);
CompletableFuture<APOD> response = apodClient.getApod("DEMO_KEY");
APOD apod = response.get();
System.out.println(apod.title);
com.android.org.conscrypt.TrustManagerImpl.checkServerTrusted()
android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted()
android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted()
android.security.net.config.RootTrustManager.checkServerTrusted()
com.android.org.conscrypt.Platform.checkServerTrusted()
com.android.org.conscrypt.OpenSSLSocketImpl.verifyCertificateChain()
com.android.org.conscrypt.NativeCrypto.SSL_do_handshake()
com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake()
com.android.okhttp.Connection.connectTls()
com.android.org.conscrypt.TrustManagerImpl.checkServerTrusted()
android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted()
android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted()
android.security.net.config.RootTrustManager.checkServerTrusted()
com.android.org.conscrypt.Platform.checkServerTrusted()
com.android.org.conscrypt.OpenSSLSocketImpl.verifyCertificateChain()
com.android.org.conscrypt.NativeCrypto.SSL_do_handshake()
com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake()
com.android.okhttp.Connection.connectTls()
def downloadTask(url):
log.info("{}/processing: {}".format(url))
try:
r = requests.get(url=url, verify=False, stream=True)
except:
log.error("[0] requests.get error: {}".format(url))
return
try:
if int(r.headers['Content-length']) > 1024 * 1024 * 2:
log.info("[1] exceeding quota: {}".format(url))
return
except:
pass
if r.status_code != 200:
log.error("[2] status error: {}".format(url))
return
return
def downloadTask(url):
log.info("{}/processing: {}".format(url))
try:
r = requests.get(url=url, verify=False, stream=True)
except:
log.error("[0] requests.get error: {}".format(url))
return
[培训]科锐逆向工程师培训第53期2025年7月8日开班！
