Block YouTube Ads on AppleTV by Decrypting and Stripping Ads from Profobuf

Sections

Part 1 – Setup pfSense on Bare-Metal

1. Why block Ads and Behaviour Tracking?
2. Required Router Hardware
3. Unboxing the Hardware
4. Install pfSense on Bare Metal
5. First pfSense Boot
6. Enable the AES-NI Cryptographic Instruction
7. Enable RAM Disk
8. Dashboard Widgets
9. Adblocking with pfBlockerNG
10. Isolate LANs for Security
11. Class B IPv4 172.31.1.0/24 Network for Untrusted Devices
12. Add Firewall Rules

Part 2 – Isolate Network LANs

1. Setup the Untrusted Wi-Fi AP
2. Automatic pfSense Configuration Backups
3. Unable to Reach 172.31.1.x from 192.168.10.x
4. Replace Stock Firmware on the AC1200 Wi-Fi Access Point
5. Archer C5 v2 into the Refuse Bin, R7000 as the New Wi-Fi AP
6. Set up the Trusted Wireless Network
7. Network Devices Interconnectivity Check
8. Windows File Sharing Gotchas
9. Public Service Announcement: Edge Browser

Part 3 – Setup DNS Adblocking

1. Block Clickbait, Incessant Ads, and Dangerous Sites
2. Intercept all DNS Requests, Even to Hardcoded DNS Servers

Part 4 – Trick the YouTube Ad Algorithm

1. How to Restrict Apple TV YouTube Ads?
2. Trick the YouTube Ad Algorithm Instead
3. Research into YouTube Advertizing Spend
4. New Goal: Convince YouTube I’m 70 and in Italy
5. Selectively Route Apple TV Over the VPN
6. Selectively Route Apple TV YouTube Traffic Over the VPN
7. Gotcha: DNS Race Condition
8. Gotcha: Authentication Trouble, Forbidden 403 Error
9. Gotcha: YouTube is Now Showing UK Ads, Not Italian Ads
10. Find a VPN Exit Node with no ASN Leak
11. Hijack Google Video DNS Queries
12. New Goal: Programmatically add IPs to the Firewall Policy Rule
13. Research Python Methods to Hijack DNS Queries
    i. Rsync Disk Backup
    ii. Install pfSense REST API
    iii. Explore the Unbound Python Module
14. Smoke Test: A Python DNS Hijacking Script

Part 5 – Decrypt HTTPS Traffic

1. New Goal: Research and install a Squid-like proxy
   i. Fun fact: Jailbreaking iPhones in Japan
2. Install a Fake-but-Trusted CA Cert on Apple TV and iPhone?
3. Experiment with Squid and SquidGuard
4. Self-Host the CA Certificate
5. Abandoning Squid: Too Slow, Too Heavy
   i. Rsync Diff of Changes
6. Install MITMProxy in a FreeBSD Jail
7. Exploring MITMProxy
8. Patch MITMProxy Source Code for Server SNI Interrogation

Part 6 – Intercept Apple TV and iOS YouTube Ads

1. Smoke Test: Intercept YouTube Ads with MITMProxy
2. Examine uBlock Origin Regex Patterns for Inspiration
3. Surgically Alter the JSON Response to Remove Ads
4. The iOS YouTube App Uses Protobuf, not JSON
5. Timing Analysis to Detect Ad Videos?
6. Decode the YouTube Protobuf Responses
7. Ad URL Polymorphism
8. Smoke Test: Intercept and Decode Protobuf in Python
   i. Pure Python Benchmarks
   ii. Pure C++ Benchmarks
9. Fuzzing the YouTube Video Ad Responses
10. Enter Burp Suite Tools for Penetration Testing
11. Exfil the Proto Schemas from the App, Cleanly?

Part 7 – Reverse-Engineer Protobuf Messages

1. Hardcore Deep-Dive into Protobuf and Wire Format
2. Exploit a Protobuf Flaw to Easily Remove All Ads by Changing One Byte
3. Smoke Test: Remove Ads from Protobuf in O(n)-Time
4. Analysis of this Successful Adblocking Technique
   i. Summary
   ii. Timing Analysis
   iii. Knock-On Benefits
   iv. Future-Proof
   v. Should Google be Worried?
5. The MITMProxy YouTube Adblocking Script

Part 8 – Summary

1. YouTube Premium
   i. Experiment in Ad Viewing
   ii. $0.15 as a Ballpark CPV
   iii. CPV from US Advertising Spend Divided by Total Views
   iv. Is YouTube Premium Worth It?
2. DMCA, Sony, Viacom
3. Summary of Accomplishments

Why block Malicious Ads and Behaviour Tracking?

You are a valuable commodity that is bought and sold without your knowledge or consent. You will be tricked with clickbait, distracted with large ads, and enticed to leave the site you are on at every opportunity. Plus, everything you do online is being monitored so your habits and searches can be remarketed and sold over and over again for years.

Privacy – Knowing what you like to watch and read, what phone you have, what you watch on Netflix, what you shop for, what you ask Alexa about, yout taste in music, etc. is unbelievably valuable to advertisers. Spying on people is such a big problem that Europe passed the GDPR law so every site you visit asks if you are okay with cookies (and we blindly click “ok” to hide the banner). We must wrestle back privacy ourselves.

Bandwidth – If privacy doesn’t concern you, how about this: it is well-known that between 25% and 40% of network traffic is ads, tracking, JavaScript to load trackers (fingerprint.js, googletagmanager.js), websocket traffic to collect how you scroll and what you type (Hotjar), and the like. Do you have a 100 Mbps internet connection? Consider it 60 Mbps!

Clickbait – Then there is clickbait. “You won’t believe what Tom Cruise did. He…” and you may want to click. Then you are in the spider’s web. How about fake news? Or articles that don’t say “sponsored” in size-8 font, but now say “underscored” to be clever. What is even real anymore? As soon as you click on clickbait, you may end up on a page with a dozen more ads that aren’t approved by Google but lead to a dark world of maliciousness. Clickbait is so incredibly profitable to scammers.

Cryptojacking – Some websites will load crypto-mining JavaScript (e.g. CoinHive.js) so while you read, they overheat and abuse your computer to try to make a few pennies. Some sites will load JavaScript that tries to steal from your crypto wallet or trick you into transferring cryptocurrency.

Top ↩

Required Router Hardware

A virtual machine, Docker image, or Raspberry Pi are not performant enough to protect a whole SMB network; We need dedicated hardware with a cryptographic instruction set so that its only function is to route, decrypt, and monitor packets in and out. Here is what I used.

• A mini PC with the AES-NI instruction set (e.g. J4125)
• Several gigabytes of DDR4 RAM (e.g. 32 GiB)
• A decent mSATA SSD drive (e.g. 128 GiB)
• A USB drive to transfer pfSense

Top ↩

Unboxing the Hardware

I’ve ordered a mini J4125 PC from AliExpress, ordered 32 GB of DDR4 RAM and a 128 GB mSATA from Amazon, and will assemble them for the first time now.

A beautiful box, isn’t it? It only has 3 LAN ports, but it can be extended with network switches.

Top ↩

Install pfSense on Bare Metal

I’ve never used pfSense before, so we will explore this together. The compressed image is about 360 MB and can be flashed to a USB drive with an AppImage binary of Etcher (very cool). Decisions, decisions: VGA install or serial? Let’s serial into the new router. Why not?

Well, that looks painful. It would also be a whole production to serial into the box in case of an emergency because the serial port is inside, and there isn’t even an RS232 or JTAG connector – just some narrow header pins. Yikes. Let’s go with VGA and plug a keyboard into the USB port – get ready to navigate with arrows and tabs.

I’ll follow this guide on YouTube. I’ll pass on encrypting the disk since I would like to avoid entering a passphrase each time the mini PC reboots. A stripe disk is fine since there is only one disk. I have no idea what to expect yet, so I will pass on dropping to a shell for a more advanced configuration.

Top ↩

First pfSense Boot

I ejected the USB containing the boot image (important) and rebooted the little box. It played a melody on the internal speaker (there is an internal buzzer and thankfully it isn’t very loud).

Do I need to have a LAN connection already, or can I just start the thing? I’ll just start pfSense and let it complain to me if it wants… and according to the YouTube tutorial, I should guess which port is LAN 1. I’ll do that now.

I figured out that I should set the LAN 1 to a static IP address that is not in my existing router’s DHCP range, so I went with 192.168.1.3. Now I can access an admin web portal (admin/pfsense). Hooray.

Yikes, the mini PC beeped at me and informed me that ‘admin’ has logged in. That startled me a bit, but hey that is pretty neat.

Top ↩

Enable the AES-NI Cryptographic Instruction

I played around with the wizard, used defaults, and got to the web configurator. The first thing that caught my eye was AES-NI CPU Crypto: Yes (inactive). I went out of my way to get a mini PC with AES-NI. What gives?

Ah, this needs to be enabled in System > Advanced > Miscellaneous. Why not auto-detect this and use the best option? I’m glad I spotted that, or else this mini PC might as well be a Celeron J1900 of yesteryear.

Top ↩

Enable RAM Disk

Having 32 GiB of RAM, let’s take advantage of that and use a generous amount of RAM for /var and /tmp, and since hopefully this 128 GiB SSD has wear levelling, let’s take a RAM Disk backup every hour.

Reboot! AES-NI is now active.

Top ↩

Dashboard Widgets

This dashboard is pretty slick. I’m just discovering that there are widgets that can be added to the Dashboard, including S.M.A.R.T to alert us if the SSD is going bad. Nice.

Hang on, when I added the Services Status widget, something called PC/SC Smart Card Daemon shows up. What is that? Research shows it’s a daemon for hardware smart keys that we can probably do without(?). It can be disabled in the /etc/rc.bootup file like so:

Wait. After some time went by, I noticed the router slowed down, fatally.

Top ↩

Adblocking with pfBlockerNG

This unboxing and setup has been fun, but I’d like to block all the bad traffic on my network. I’ve been using a workhorse of a DNS-level adblocker called Pi-Hole on a… yes, Pi, but it would be nice if I can reclaim that wee bit of hardware for something else and use a comparable add-on module in pfSense. Let’s explore that now.

pfBlockerNG is a very powerful package for pfSense® which provides advertisement and malicious content blocking along with geo-blocking capabilities.

Question: Do I install the first pfBlockerNG or the pfBlockerNG-devel which feels like a developer version? I’m a software developer, so this is for me, but am I a pfSense developer? No. Maybe it will show me advanced logs or I can mess about with LUA? Let’s Google this.

From here, random people are saying to install the development version. Another blogger advocates using the dev version as well. Meh, I guess we can install jq, rsync, and Python 3.8. This doesn’t feel like a development version since it has exciting dependencies.

That was painless and only added an extra 20 MiB. It seems a lot of the dependencies are part of pfSense already. The knight at the end of Raiders would say that I have chosen wisely (hey, why did Indy age like a normal person up to Indy 4 if he drank the immortality water that the thousand-year-old knight also drank?).

Wizard time.

There are a lot of options in step three. This is not like Pi-hole at all. I’m going to come back to this and set up my network instead so I accomplish retiring my Nighthawk R700 or giving it new life as a Wi-Fi AP.

Top ↩

Isolate LANs for Security

An opportunity has presented itself: I can create real networks on each of the three router Gigabit ports (not VLANs), and should I do so? Yes, yes I should. I would like a dedicated hardware network for all my home-phoning spy devices (Alexas and Apple TV) so they don’t flood my network with metrics info and “sure I’m muted and not listening to you” audio payloads back to their HQs.

I can see it now: A Wi-Fi AP on a hardware LAN that is isolated from everything else and dedicated to these gadgets, and runs through the adblocker and traps hard-coded DNS queries to 1.1.1.1 and 9.9.9.9 and others (I’ll have to explore this) so YouTube on my TV doesn’t sneakily bypass Pi-Hole any DNS-level blocker. It’s so Utopian an outcome I may not be able to sleep.

I’ve decided that my bottom-shelf TP-Link wireless router that is so old that AC1200 might as well be “A.D. 1200” is going to be my Wi-Fi AP for those IoT spy devices.

In sum, there will be a dedicated hardware LAN

• with a wireless AP (AC1200) for Amazon/Apple gadgets and the TV.
• with a wired switch for all the beefy computers and clusters in my lab.
• with another wireless AP (R7000) just for iPhones and watches.

As an aside, since doing an Offensive Security hacking course in my spare time, and I rare-earth-magnet-strongly suggest isolating Wi-Fi devices from any critical LAN segments connected to devices that touch daily banking or stock trading (or crypto wallets).

Top ↩

Class B IPv4 172.31.1.0/24 Network for Untrusted Devices

The class B IPv4 range 172.16/16 is a valid range of private IP addresses. I’m not uncomfortable with Alexa and Apple TV being even on the same class network as my main LAN segment, so I will banish them to the class B private network at the hardware level, and my more trusted LANs will be on the traditional class C network (192.168/16). This helps mitigate any misconfigured iptables rules by naturally having no routes between the two networks.

Be sure to enable the DHCP resolver on the physical NIC that will connect smart devices (which mainly just tell me the weather and creepily listen to me sleep).

From this point, DHCP works on this new network, but by default, it assigns IP addresses but does no routing. All traffic is blocked by default.

Top ↩

Add Firewall Rules

We need to manually add rules so traffic on the physical NICs goe somewhere.

There is a logging message. Let me reproduce it below.

Hint: the firewall has limited local log space. Don’t turn on logging for everything.

I read that to mean, “Congratulations on not cheaping out on your SSD. Now go forth and log everything, my son.”

I’m not a new-age, fancy-jazz, coloured-light- or smart-plug-controlling guy who forgot how to turn on a light without his phone, so I do not need to have smart devices on the same network as my phone (why create dozens of wireless attack vectors into your home?). I’m classicly trained to actuate an electromechanical current interrupter on the wall and light let there be.

Top ↩

Setup the Untrusted Wi-Fi AP

How do I reach the admin UI of AC1200 Wi-Fi AP now? I factory reset it, plugged the WAN NIC into the ETH3 NIC of the pfSense router, but both devices are just blinking at me.

I suppose I can just Wi-Fi into the factory-reset AC1200. Yikes, 2016 was a bad year for responsive web UIs I take it. This is horrible; I’ll pull out a netbook for this. One sec.

It seems the Archer C5 has no AP Mode. This is my problem, not yours, but I’m still going to vent.

Oh, and the “refresh” icon on the top of the DCHP Leases page in pfSense is not “refresh”, but “reload service”. Whoops.

Well, I bricked the AC1200 router. I will have to run an Ethernet cable manually… but, wait, my thin notebook has no Ethernet ports and needs a USB-NIC adapter. Happy Friday.

There were shenanigans, but I set the LAN IP of the AC1200 to 172.31.1.100, the ETH3 NIC IP of the pfSense router to 172.31.1.1/24, and set the pfSense DHCP service on ETH3 to assign addresses 172.31.1.101~150. What failed was setting the AC1200 to 172.31.1.2 as it was unreachable (reason unknown). Oh yes, I had to turn off firewally things and NAT Boost, and basically drop the horsepower of this TP-Link router down to that of a potato battery. The above settings allow me to access the AC1200 remotely now.

The other video ran its course, so I started following this YouTube video (set the speed to 1.5x).

One more thing: I installed the nmap package for pfSense and scanned the AC1200 router, and found some sneaky ports open.

Port 20005/tcp is a print server port that I’ve now closed. However, the Archer C5 AC1200 is vulnerable to all kinds of Kali mischief so it was wise to put it on its own network. I’m not sure how to close port 22 and the SSHd service on it the AC1200 because the stock firmware is ancient and crippled, so I’ll just have to block port 22 on the whole LAN segment.

I’ve also taken care of disallowing private networks to ingress on the WAN (see the next section to set up DMZ).

Top ↩

Unable to Reach 172.31.1.x from 192.168.10.x

Ping and Traceroute are aiding me in my efforts to connect to the AC1200 Wi-Fi AP from my Trusted LAN. I went ahead and added the subnet to the Symantec Firewall rules just in case (Symantec has its place now and then, but yes, definitely have available PC CPU horsepower to spare).

Now, it seems ICMP packets are no longer blocked between networks, but I still cannot ping the AP web management UI even though I can see the pings in the traffic logs.

I’ve even added an “any to any” firewall rule on the Untrusted network. No change.

Let’s try a stealth scan instead: sudo nmap -sS -v 172.31.1.*.

Nope, pfSense doesn’t like that at all. And, the whole network stopped working. Nice security! Also, dang.

The good news is that I’ve isolated the packet malaise to the TP-Link AC1200 box itself. I suspect that I need to add net.ipv4.ip_forward=1 to forward packets with no addresses in them, but I’d need root access to the AC1200. Let’s burn it to the ground and rebuild from its sprinkler-soaked ashes.

Top ↩

Replace Stock Firmware on the AC1200 Wi-Fi Access Point

Of course, I cannot actually stop Untrusted LAN devices from reaching the AC1200 as they all exist downstream from the pfSense box.

DD-WRT open-source router firmware, meet my ancient Archer C5 and do your thing.

The Archer C5 did not accept the DD-WRT firmware. Hmm… how about OpenWRT?

The Archer C5 did not accept the OpenWRT firmware either. What the actual facepalm (WTAF)?

Wait. My hardware is revision 2 using the Broadcom chipsets which are notoriously difficult networking chips.

Alright, so OpenWRT, DD-WRT, and Tomato projects have no firmware for this AC1200 with unpopular Broadcom chipsets. Into the refuse bin it goes.

Top ↩

Archer C5 v2 into the Refuse Bin, R7000 as the New Wi-Fi AP

I’ve dismantled the AC1200 so I do not forget why I threw it out. It’s too bad because it’s so pretty on the inside, and they always say, “It is what is inside that counts… except if you are a router with Broadcom chips.”

The R7000 is factory reset, and here is the first problem:

The R7000 is in AP mode, but I can still access the pfSense web management page from the Untrusted network. Let’s lock down the web UI in pfSense under Firewall Rules.

Top ↩

Set up the Trusted Wireless Network

The Untrusted network is now looking good. It’s time to make the other R7000 Nighthawk I have into a Wi-Fi AP as well so my phone and watch have a safe place to connect to, as well as a laptop when I want to RDP into my wired machines from the kitchen. I was saving that for a honeypot AP, but I can come back to that later.

Let’s see if I can Wi-Fi into the Wireless LAN’s R7000…

Since only my trusted devices should be on the Wireless LAN, I’ll turn off 2.4 GHz wi-fi because anything recent and wireless should support 5 GHz. That means those pesky AliExpress Pineapple wi-fi password stealers on the cheap side only use 2.4 GHz, so a neighbour is going to have to put in some effort to snoop on my network. Plus, 5 GHz is blocked more easily by walls and concrete, so I prefer it for averting medium-range snooping. But, I so am going to set up a honeypot and to brake check my faith in humanity.

It is normally straightforward to put a Wi-Fi router into AP mode by disabling WAN and DHCP.

Top ↩

Network Devices Interconnectivity Check

Do all my dozens of computers, laptops, Pis, clusters, NAS drives, and the like still connect as before? Most important is my web-scraping bot in a hardened, RAIDed, dedicated machine with its own UPS. But alas, I cannot SSH into it even though the SSH handshake packets make it to the hefty box.

Could this be our old frienemy IPv4 forwarding being disabled? Possibly. I’m able to SSH into the machine from my iPhone (seriously) when on the same network.

Nope. Adding net.ipv4.ip_forward = 1 in the right place with a restart did not yield joy.

According to dmesg -w (to tail dmesg logs), UFW (Uncomplicated Firewall) is not blocking ICMP requests or TCP requests on port 22. When I do something nutty like try to SSH on, say, port 23, then I can see the UFW block logs in dmesg. Confirmed: Packets can reach that machine.

Running tcpdump src 192.168.10.100 where the IP is from the Trusted network on the target machine shows it is responding to pings. I’m even getting replies to SSH handshake requests. So now we know that return packets are being dropped. Interesting! Aside: tcpdump is awesome.

Let’s follow the trail. Digging a little deeper I see replies to ICMP and SSH handshakes are being sent to some IP over HTTPS that I do not recognize. Bizzare. When I run the usual ipinfo tools I see that replies are going over a VPN that I completely forgot about. Ha. Replies to a different subnet are egressing over the VPN, but cannot return properly. Neat.

Now that I remember what I did in 2019, I re-added NAT alias rules, and it’s showtime again.

Top ↩

Windows File Sharing Gotchas

Your path may be smoother, but I’ve always seem to make the Trench Run instead of remote-piloting a handful of lead-filled X-Wings at light speed right through the Death Star’s reactor to make it go boom: the easy way.

I’ve added some rules to allow Static DHCP devices to talk to each other – Windows devices – but by default, the Private Network in the Windows Defender uses the local subnet as the rule scope. That means different subnets are isolated. We can’t just relax the pfSense DHCP subnet mask to say 192.168.20.0/16 because it conflicts with another subnet. Instead, just to get file sharing working, I relax the scope in Advanced Settings like below. Be sure to modify In and Out for SMB and ICMP.

Again, please add whatever subnets you desire instead of any.

Top ↩

Public Service Announcement: Edge Browser

Why does the Microsoft Edge browser start automatically and run in the background, and why can’t I kill it when I ctrl+alt+del? If you’ve asked yourself this, you’re not alone. It turns out Edge starts up when you log in and it keeps running in the background. Here is the fix:

I suggest downloading Winaero Tweaker and applying registry tweaks to cut down on the Redmond Spy Machine.

Top ↩

Block Clickbait, Endless Ads, and Dangerous Sites

Thanks to web-browser and DNS-level adblockers (i.e. Pi-hole), it’s commonplace to block bad sites, crypto-miners, fingerprinters, trackers, remarketers, banners, pop-ups, fake tech-support scam alerts, and all manner of unscrupulousness designed to take advantage of you. Let’s take pfBlockerNG on pfSense for spin.

The pie chart looks great. I followed this pfBlockerNG tutorial.

This is mportant: If you have multiple network interfaces (the mini PC has four), then you need to enable the Permit Firewall Rules for multiple interfaces and select them.

Would you like to have discretion over blocklists? Let’s add a DNS blocklist related to gambling and reload pfBlockerNG to see if a poker site is blocked on the Trusted LAN.

If you would prefer the connection to just close instead of rendering a PHP page, create a new PHP script with the following code and select it in the pfBlockerNG settings page:

Top ↩

Intercept All DNS Requests, Even to Hardcoded DNS Servers

Let’s make sure all clients behind the pfSense router use the local Unbound DNS server so pfBlockerNG can act on them. We do not want apps and home assistants to bypass our DNS server, so we have to add some NAT rules.

First, we have to block DNS over TLS (for now) and only allow local DNS requests (note the rule order):

Here is a NAT rule for one interface. I started by making a rule for each interface except WAN (obviously) like this below.

To make life simpler, I made a firewall alias of all non-WAN interfaces called Non_WAN. Covering IPv4 and IPV6 to redirect local DNS queries on port 53 to localhost are the following redirect rules:

Let’s also log trapped DNS requests. Head to the Services > DNS Resolver page, click “Display Custom Options”, and add the lines:

Well, hello there, Microsoft Windows. What are you up to trying to reach Google Tag Manager? Naughty OS. That request is now black-holed to a non-existent IP at 10.10.10.1.

Let’s turn our attention to the TV and see how it fares under DNS interception.

Top ↩

How to Restrict Apple TV and iPhone YouTube Ads?

YouTube is tricky because ads are also videos that come from the same domain, so domain-name blockers like pfBlockerNG cannot act on them. The best pfBlockerNG and Pi-hole can do is block googleadservices.com only after you watch an ad video and click on the ad.

Many people opt to use a web browser like Firefox or Chrome with uBlock Origin that acts on JavaScript as a workaround. It might be enough to watch YouTube on a web browser and stream that to a smart TV. However, we cannot restrict ads on the iPhone (without jailbreaking and compromising it).

What are our options? How can we safely restrict YouTube ads on all network devices?

Top ↩

Trick the YouTube Ad Algorithm Instead

What do ads in other parts of the world look like? Are those living in Antarctica or Low Earth Orbit getting a lot of ads too?

What would happen if we leverage the capabilities of this pfSense router to route YouTube Location Tracking information through a VPN that terminates in some remote part of the world with fewer YouTube viewers per capita? In other words, let’s make ourselves undesirable to advertisers and see if we get fewer ads.

Top ↩

Research into YouTube Advertizing Spend

Let’s do some YouTube demographics research to find a part of the world avoided by advertisers.

Let’s also check some YouTube statistics about viewers by country for insights. Thinking about following some Reddit advice and VPN’ing into India? Think again.

That was 2019. This is 2020:

I’m not a digital advertiser, but I can see that people in the UK and Canada watch a large number of videos per sitting. If I were an advertiser though, I’d pump those two countries with video ad after video ad because, statistically, those residents will take the eyeball kicking. All things being equal, I definitely need a VPN to terminate outside of Canada, the UK, and the United States (English-speaking countries) to enjoy YouTube more.

Does age play a factor? Who don’t advertisers want? I want to be that guy on paper.

Top ↩

How then to convince YouTube that I am a retired Sicilian living on a small chain island? I embellished that last part. Seventy and in Italy is sufficient.

Let’s do this. In the YouTube account…

It is doubtful that this is all it takes for our goal. Let’s find a VPN exit point in Italy.

Nice. NordVPN has about 60 servers in Italy (that’s an affiliate link by the way).

Top ↩

Selectively Route Apple TV Over the VPN

Let’s go through some tutorials to set up OpenVPN in pfSense. Just kidding! We’re going to use WireGuard – we have the Intel AES-NI crypto instruction set because we didn’t go cheap and get a yesteryear J1900 mini PC that sellers are trying to offload.

I’ll now install the FreeBSD WireGuard package.

Next, add a tunnel and enable it. According to this thread and this thread on Reddit, we need to get some information for WireGuard and NordLynx from a sacrificial Linux VM to transpose the settings (i.e. private key) to the pfSense router. No problem.

Run sudo wg showconf nordlynx to see your private key needed by the pfSense tunnel config.

Here are various screenshots that show the steps in more detail.

That should be enough to allow Diagnostics to curl Italy.

Now that the easy part is out of the way, let’s set some Policy rules to send the Apple TV traffic over the VPN to Italy as a baseline test.

From Netgate, on the order of Firewall/NAT processing:

Traffic from LAN to WAN is processed as described in the following more detailed example.

• Port forwards or 1:1 NAT on the LAN interface (e.g. proxy or DNS redirects)
• Firewall rules for the LAN interface:
  • Floating rules inbound on LAN
  • Rules for interface groups including the LAN interface
  • LAN tab rules
• 1:1 NAT or Outbound NAT rules on WAN
• Floating rules that match outbound on WAN

I’ll make an alias, for now, to hold some clients that have static DHCP entries and hostnames I gave them in pfSense.

Floating rules in have high precedence, so I’ll add some rules below the automatic pfBlockerNG rules that were created, and I’ll add a nice little blue separator while I’m here.

And here is that rule as a very long screenshot:

Apply. Wait. Let’s try it out using one of my notebooks connected to the Untrusted network.

Google is in Italian. Very cool. Now for the Apple TV.

Winner winner, chicken diner. All my YouTube is in Italian. I get some ads, not as many, but because Italians speak slowly and with a kind of sexy accent I do not mind the ads for Nutella at all.

With this technique, I no longer feel manipulated by non-English ads. I have personalized ads off, but given my new status as a retired gentleman I should turn that back on to scare away advertising dollars, er, euros. I wonder if Netflix and Amazon Prime behave any differently…

Dang. Netflix is having problems. Amazon Prime is even worse. It looks like some CSS or font files are blocked as well, and the thumbnails aren’t loading. It’s time to move to Phase Two: Tunnel only YouTube traffic over the VPN.

Top ↩

Selectively Route Apple TV YouTube Traffic Over the VPN

Let’s start by adding Firewall Policy rules to send the most common YouTube domains over the VPN.

As I’m about to add the rules, my hands hover over the keyboard not knowing what domains to tunnel. They need to be FQDN (fully-qualified domain names, no wildcards). Let’s open up a Chromium-based browser and see what traffic it generates in DevTools.

Here are some candidate FQDNs to add:

But wait, I hear you ask, why accounts.google.com and gstatic.com? This is a preventative measure just in case one of those domains is geo-jacked (Geo-IP LowJacking). I wouldn’t put it past Google engineers to geo-jack the fonts domains like fonts.googleapis.com, but I’ll take a chance they don’t in the interest of scaling to billions of page views efficiently.

Here are my new rules where I chain two of them using a tag so I can limit YouTube tunnelling to only the same untrusted machines (including Apple TV).

And with that, YouTube thinks I’m in Milan, Netflix and Prime Video think I am still in Canada, and the ads… oh the ads… they are few and far between, and when they do come on, they are just a treat to listen to in that slow, lack-of-harsh-aspirants-or-yelling of a beautiful language Italian is.

Top ↩

Time goes by…

Gotcha: DNS Race Condition

A day has gone by and I’ve noticed that I only get Nutella and Ferrero Roche ads in the middle of videos, not at the start. Odd. I did some research and this is what I found:

This means that the hostnames are resolved to IP addresses once and those IPs are used in my VPN tunnelling policy rules.

A hostname entry in a host or network type alias is periodically resolved and updated by the firewall every few minutes. The default interval is 300 seconds (5 minutes), and can be changed by adjusting the value of Aliases Hostnames Resolve Interval on System > Advanced, Firewall & NAT tab. – pfSense

Ah-ha, so I suspect there is a DNS race condition. Let me explain:

This happens if, say, the Alias Daemon updates the IPs of the FQDNs. Then, I turn on the Apple TV for the first time all day. Since the usual TTL (time-to-live) of DNS queries is 1440 seconds (30 minutes), all the YouTube DNS entries will be cache misses and will need to be updated. At this point, the IPs from the second DNS queries may be from a pool and are not guaranteed to be the same that the Alias Daemon has. When the Alias Daemon checks again in five minutes, it may resolve the FQDNs to yet different IPs!

Let’s solve this by overwriting whatever TTL (time-to-live) YouTube has in its DNS entries:

And with that, no more DNS lookup race condition.

Top ↩

Gotcha: Authentication Trouble, Forbidden 403 Error

Sometimes videos will not play. For security, YouTube embeds your IP in the googlevideo.com request. I’ve known about this since my post about Download YouTube 4K Videos with PHP back in 2016. The new problem is that various JavaScript and “are you human?” assets are tunnelled over VPN, but those darn domains like r5---sn-hpa7kn76.googlevideo.com are not tunnelled and thus come from the wrong IP. Queue the 403 Forbidden error.

Let’s fail fast with a quick experiment: I’ve gotten the IP of the above second-level domain name (SLD), added it manually to the list of domains/IPs to VPN tunnel, applied the change, and refreshed YouTube:

Excellent. Now, we just need a way to tunnel that wildcard *.googlevideo.com domain. Unfortunately, the NAT and Firewall rules work with IPs, not wildcard domain names. Can we predict or enumerate these domains?

Here is a Wireshark capture of DNS requests to *.googlevideo.com to show that the SLDs (second-level domains) are not eyeballably predictable:

Let’s drop into a web browser with adblocking disabled and walk the HAR waterfall of my interaction with YouTube that led to ads showing up.

What are GET requests like

GET https://r7---sn-uxa0n-t8ge.googlevideo.com/generate_204

doing, exactly? I’ll give this problem some thought offline.

Top ↩

Gotcha: YouTube is Now Showing UK Ads, Not Italian Ads

Before I could even solve the previous gotcha, British ads started showing up with the same frequency as if we did nothing. Ads from the UK are even more incessant than those from Canada, trailing behind the USA and India according to my earlier stats. It would be a complete failure if we get UK ads. Why does this happen suddenly? I’ve opened a fresh browser in a VM and tunnelled all traffic through Italy. The only leak I can find is when I query ipinfo.io on my Italian tunnel and see a UK address in the ASN. Could this small leak be our undoing?

Even with my browser’s language set to en_US and location data off, this is the only leak I can spot. Then, in addition to a VPN exiting in Italy, it has to be one that doesn’t leak ASN (Autonomous System Numbers – used for automated routing) that gives up a different country. Dang, Google, you’re good. I’m going to have to bring my A+ game to this one.

Top ↩

Find a VPN Exit Node with no ASN Leak

By visiting https://nordvpn.com/servers/tools/, I can see the VPN endpoint nodes in Italy. There are many Wireguard endpoints with NordVPN. Just to move things forward for this exercise, I’ll add an OpenVPN tunnel in pfSense and connect to several VPN nodes and examine the ASNs. It’s better than nothing, and more importantly, I’d like to eliminate the ASN as the leak of GeoIP information. Here is the guide I used.

Through trial and error, I found a VPN node that is registered to an ISP in Italy as found in the Abuse and ASN info.

Beautiful. Bellissimo.

Top ↩

Hijack Google Video DNS Queries

To make any of this work, I need a technique to route the wildcard *.googlevideos.com domain through the VPN.

On the other hand, if I “hold up” the DNS query for the *.googlevideo.com domains, add the IPs to some alias list, then allow the DNS response to finish the round trip, we may be in business!

Where to even start? Here are some Python example scripts just to get some inspiration. A quick, mental reverse-engineering of a handful of scripts reveals that there are some event hooks available. Nice.

Among friends, let’s say that I can build up the pool of Google video IPs in real-time. How then to add these IPs programmatically to the firewall alias list for YouTube without restarting the firewall? One person actually hacked the PHP scripts in pfSense. Tempting, but I’ll do more research. Another person created a REST API for pfSense. Jackpot!

Top ↩

Research Python Methods to Hijack DNS Requests

Why this approach? It’s future-proof, modular, elegant, maintainable, automated, and it lends itself to a future decision tree that could truly restrict YouTube ads outright.

First, I will enable SSHd in pfSense and take a peek around.

Rsync Disk Backup

Let’s take this opportunity to make a disk backup. du -h or “duh” shows that only 800 MiB is in use on the SSD. Let’s rsync the whole box from our local machine in about four minutes.

Install pfSense REST API

Now that we have a pfSense backup (I’m told just backing up config.xml works too), let’s install the REST API.

This part had me confused. You see, I was looking at the bottom of the screen wondering how the heck I could copy a truncated hash as a token. After a few tries, I noticed the green message at the top that I had been trained to ignore. It has the token.

Next, with the API credentials set up, let’s try out the API:

Explore the Unbound Python Module

Running find / -name "py*" shows that the current version of Python is 3.8.

As for the Unbound DNS Resolver, I had some luck tinkering in nano and writing simple Python 3.8 code to log DNS query messages. We now have both parts needed to dynamically update the firewall aliases and tunnel all YouTube traffic once and for all.

If you are looking for Python module docs for Unbound, here they are:

Run these commands to quickly get the documentation.

Now we have Python docs and access to all the capabilities. Excellent.

Next, take a backup of your OS or VM and install libtools and swig wherever, ./configure --with-pythonmodule, make, fix some errors in the Unbound code, make again, then you’ll have the generated python module (unboundmodule.py) in order to remove all the missing-method red error lines in PyCharm.

Top ↩

Smoke Test: A Python DNS-Hijacking Script

Here is a smoke test of the ability to hijack *.google.com DNS requests with reply IPs that the script has caught in just a few minutes (the timestamps are just to maintain a crude LRU cache):

Duplicate IP addresses are possible, and that is fine. I let the smoke test run overnight. Here is the PoC (proof of concept) script I ran as the Unbound Python module script.

When I woke up, the Unbound DNS resolver service segfaulted. Here are the logs:

Top ↩

Actually, it is not illegal to jailbreak most Apple TV boxes, so we could break in, add a root certificate valid for the pfSense box, MITM traffic from the Apple TV, and then Microsoft Bob is your uncle. That works because the pfSense box as the gateway can decrypt Apple TV traffic, inspect the request headers for the offending ad hostname, block the request, and re-encrypt other valid requests to Mountainview, California.

But, then my iPhone would still show ads because it is harder to jailbreak, plus banking apps may detect this and not work anymore. Jailbreaking is too extreme, anyway.

Let’s see what it takes to spy on the HTTPS traffic from the Apple TV and iPhone to see if we can block ad URLs that way.

Top ↩

Install a Fake-but-Trusted CA Cert on Apple TV and iPhone?

Not wanting to jailbreak and add self-signed certs to Apple TV and iPhone, how hard would it be instead to add fake-but-trusted Certificate Authority (CA) certificates to each device?

The ‘A’ in CA means there is no one higher to vet such a certificate. The ‘A’ is so powerful, that back in 2001 only a Windows patch was able to revoke some dangerous Verisign certificates. As a thought experiment, new CAs must come into existence from time to time. Let’s Encrypt is relatively new, for example. There should then be an in-warranty way to get a fake, trusted CA cert into an Apple TV and iPhone. If that is possible, then an entire world of MITM spycraft is available to decrypt TLS packets in the clear and use good ‘ol URL blocking on requests like

Let’s see how easy this would be.

In fact, there are many, many CAs. Here is a quick find / -name "*.pem" in pfSense:

Top ↩

Experiment with Squid and SquidGuard

I’m aware of mitmproxy, but it needs to be side-channel installed onto the pfSense router. Let’s see if the squid3 proxy that is available as a pfSense package can do what we need. First, I will take a bare-metal backup again so I can roll back in case mitmproxy is better.

I’ve installed those packages, and naturally, there are more buttons and options than in a space shuttle. I’ll find a guide.

I’ve followed the steps in the guide, however, since I have a large SSD and generous RAM, I’ve made a dedicated folder /squid_cache (and chown squid:proxy) with 8 GiB of cache and a juicy allowance on the per-item cache size which should also help with Docker and NPM speed-up. Two birds, one stone. With Transparent HTTPS support, this should be pretty rad.

We can actually generate a CA cert in pfSense itself.

Now, how to get it into the Apple TV and iPhone? It should be hosted somewhere, right? How about on the router?

Top ↩

Self-Host the MITM CA Certificate

Self-hosting with a single command is ridiculously easy. From the SSH shell into pfSense, I can create a web folder and server like so:

When I visit //pfsense:8000 I should get a blank page with “Hello”. From here, clients behind the pfSense router can temporarily access static documents.

To make like easier, here is a PHP script to cause the MITM cert to download.

As another smoke test, I’ll add the MITM CA to Chrome (manually) and enable the SSL Filtering. The defaults are fine in Squid. Here is the log file when I visit https://ericdraken.com:

Excellent.

However, on every other browser and machine there are HTTPS errors like so:

Top ↩

Abandoning Squid: Too Slow, Too Heavy

After a day of painfully setting up Squid and SquidGuard and adding blacklists and even manual regex for things like .+?/pagead/.+, I’m having nothing but issues with Squid. Here are the top pain points:

• It’s slow. It’s really slow.
• The ACL (Access Control List) settings are cumbersome.
• There is an issue with https://http/* (ref).
• The SquidGuard URL filter takes eons to update a list.
• The Squid UI is unbelievably lacking.

Squid makes me sad. I don’t get sad, but Squid makes me sad with its promise and ultimate letdown. I’ve now obliterated Squid and restored the router from the rsync backup I made earlier. Here is a handy little script to show a diff of what has been added by Squid and related packages.

Rsync Diff of Changes

The output is something like this under the --dry-run option:

Top ↩

Install MITMProxy in a FreeBSD Jail

Even though written in Python, I’ll give mitmproxy a try next; at the very least it can be purpose-built to block YouTube ads with its rich API and Python-hook extensibility. It was a coin toss between mitmproxy and SSLSplit – a Metasploit hack tool – to achieve on-the-fly TLS interception, but the former can be scripted with Python and has a satisfying UI. Let’s go.

You’ll notice that there are only three binaries about 24 MiB each. As I understand it, they have a self-contained Python 3 environment and frozen dependencies. I’d like to jail these binaries because, well, because. First, let’s see if there is a vulnerability report for mitmproxy at vuxml.freebsd.org. Nothing. How about at Exploit-DB? Nothing again. Good.

First, what version of FreeBSD is this pfSense install?

Now, according to this guide, I’ll need to set up jails myself as they are disabled in a default pfSense installation. Not knowing FreeBSD at all before today, I had to hack around to find a URL to download the ezjail package manually. After another bare-metal backup, here are the steps I took:

We need to do some hacking to get jail working on pfSense’s take on FreeBSD because jail is missing completely. What I’ve done is copy the jail binaries from a jail (via ezjail) back to the root system.

Let’s set up a jail for mitmproxy.

This is very important: We must enable raw sockets in this jail to allow transparent proxy mode to work. If not, MITMProxy will report errors like “Transparent mode failure: FileNotFoundError(2, ‘No such file or directory’)” or “Cannot open connection, no hostname given.” This is because raw sockets are inaccessible and server information is unavailable. We can easily edit the ezjail config file per jail like so:

This is also very important: MITMProxy calls sudo -n /sbin/pfctl -s state but there is no sudo in jail. Run pkg install sudo inside the jail.

Now we can copy over the mitmproxy binaries and take them for a spin.

Things are getting tricky with this next part. Running any of the binaries above results in:

So, there is no /lib64 folder nor any similar dynamic linker that I could find. I tried this, however:

Apparently, there is a pkg install compat6x that can solve this for us (unavailable on pfSense), however, this is getting ridiculous! Let’s try a new tactic. Since we are in a jail, we are not bound to the crippled (read: secured) pfSense environment. Maybe we can install the mitmproxy package normally in a jail?

pkg install mitmproxy

And, Bingo was his name-o. After this, simply running mitmproxy in the jailed console opens the MITMProxy UI. Nice. Note, this version may be one or two minor versions behind the master branch. Let’s clean up with rm -rf ~/mitm* /lib64 and do another bare-metal backup.

Top ↩

Exploring MITMProxy

This is getting exciting. First, in pfSense, add a virtual IP for 127.0.1.1 attached to localhost. Then, add a NAT rule to temporarily forward port [Private IPs]:8080 to 127.0.1.1:8080 to access the proxy from the LANs.

If not in the jail console, I’ll run

and add the proxy setting 192.168.20.1:8080 to my sacrificial notebook (that is auto-wiped daily). When the browser opens, we can already see colourful log entries in the MITMProxy UI.

The next step is to get the auto-generated CA PEM file used by MITMProxy (~/.mitmproxy/mitmproxy-ca-cert.pem). Since any CA cert here is snake oil, I’ll use the provided one. TLS traffic from my devices is safe as long as I use my own proxies.

Let’s put our experience from our previous attempt at self-hosting a CA into action. However, there is no PHP in the jail, so we can use a Python 3 web server instead.

After installing the CA in the Trusted Root Store on my clean notebook (and rebooting), I am treated to this display:

Let’s see if we can get this cert on my iPhone.

This is incredibly exciting. Can we LoJack the Apple TV box next?

Excellent.

But wait, the router is slowing down. mitmproxy is burning up the CPU… on idle.

Of course: Python is a single-threaded paradigm with the GIL (Global Interpreter Lock) ensuring threads do not actually run concurrently – unless they are blocking on I/O, which is the case here(?). Except, most of the CPU work is to generate TLS certs on the fly for each request. Yikes. Running mitmdump forgoes the UI and extreme logging. The extreme logging of all the headers and full responses heavily slows down mitmproxy, but mitmdump by default only logs entries like classic Apache logs – much kinder on the CPU.

For my fun, I’ll go with this CLI command:

While on YouTube, we can see the page ads clear as day with their unencrypted headers; can a simple regex now block them? They are exposed, and afraid, and their days have run out.

We can even see details about each request. For example, all the SAN info is laid out for this wide-reaching certificate. There are curiously a lot of *-cn.com domains covered by this cert.

Shortly, I’ll write a Python script to block YouTube /pagead/ URLs.

Top ↩

Patch MITMProxy Source Code for Server SNI Interrogation

This step may be optional for most, but as a reminder to myself, to make --allowed-hosts work better in Transparent Proxy Mode, the SNI of the server request needs to be checked against the list of regular expressions or else only the server’s IP is used for matching in many cases. Here is a quick patch I made that can be applied directly in the jail shell (or just type a few lines manually) for mitmproxy version 7.0.4:

With the above patch, I can now reliably intercept a few hosts and let all others pass through.

Top ↩

Smoke Test: Intercept YouTube Ads with MITMProxy

After reading the docs and navigating the mitmproxy source code in the PyCharm IDE, I’ve written a little script to block ads and tracking URLs coming from YouTube from my clean notebook. I won’t reproduce the code just yet because it didn’t succeed in blocking ads as hoped, so instead, I’ll spend the time investigating why.

Here are the smoke test filters I used where for a given top-level domain, URLs with the following partial strings are blocked:

My initial results on blocking are positive. Everything I wanted to be blocked is faithfully blocked. Note, the (failed) entries are due to my script, and the 502 failures are due to pfBlockerNG black-holing the request.

Even in the DevTools network panel, the requests are truly blocked.

Then how come I am still seeing ads? I’ve disabled HTTP/2 so that subsequent requests on the same channel don’t slide by. Mind you, sometimes the ads skip on their own, or fail to play, but they still show up. Interesting. Could YouTube be using WebSockets? I need some inspiration, so I’ll look at uBlock Origin’s regex filters for some ideas.

Top ↩

Examine uBlock Origin Regex Patterns for Inspiration

Here are some of the regex/filters that uBlock Origin uses on YouTube.

At first blush, it seems that a community of like-minded individuals is playing whack-a-mole with YouTube’s HTML and JavaScript. This has got me thinking: How does a video know to play an ad with JavaScript?

How does YouTube know if the ad converts? They must target ads for individuals, so a given video must receive some unique information about an ad, such as the click link and alt text. WebSockets would be a pain to maintain, especially with all the mobile clients. They must be using stateless JSON to relay that pertinent information in an innocuous URL request that has no telltale signs of ad-ness. Let’s hunt for this info in the JSON replies captured by mitmproxy.

Snap, Crackle, and Pop. We have a new plan: surgically alter the JSON response body to eliminate or Byzantine-up the ad information.

Top ↩

Surgically Alter the JSON Response to Remove Ads

After a bit more playful exploration, a trove of blocklorne URLs is right there in the JSON payload. In fact, most of what I am trying to block shows up right here:

However, YouTube has bobby-trapped their UI and there is more than one way their obfuscated JavaScript code can pull down the ad details.

Let’s blow it all away right now.

After a lot of fun taking apart the YouTube UI and HTTP workflow, taking into account cookies and naughty service workers, I am successfully able to strip away all the pre-roll, post-roll, mid-video, and, well, all the video ads. Here is a screenshot from mitmdump showing how select REST queries are intercepted, decrypted, modified, put back into the response, and the headers updated (content length, etc.).

With this new ability, we could even inject JavaScript into the main YouTube web page and subvert their JavaScript in a sort of ECMAScript arms race, possibly even leveraging some of the filters from uBlock Origin. However, we can hang our hats on this accomplishment for today.

Top ↩

The iOS YouTube App Uses Protobuf, not JSON

I can see very similar data in the Protocol Buffer (Protobuf) version of the same API calls as the web version to that of the YouTube iOS app. That complicates things, somewhat: We cannot lean on JSONPath to hunt down advertisement sections of JSON because with Protobuf the keys are just numbers that can even change.

I can see strings like “Telus” and “Samsung TV” and “Boxing Week” and “Buy now”. Remember when YouTube was a fun place? A fable about a Golden Goose comes to mind, Alphabet.

What is a Protocol Buffer? Here is an infographic from Data Science Blog.

As a consequence of being able to see unencrypted traffic from my iPhone, I’m taken aback by the sheer amount of tracking information laid bare; It’s like I have electrodes on my head and chest while I’m running on a treadmill and a bunch of scientists in white lab coats with clipboards are standing shoulder-to-shoulder recording everything about my internals.

The next question is: Does the iOS app protocol behave like the web app?

Top ↩

Timing Analysis to Detect Ad Videos?

The iOS network traffic is not like the web traffic; Google has teams and teams of engineers dedicated to making sure blocking their ads isn’t computationally feasible. Daunted but undeterred, I was staring at network requests to let my mind zone out and wander when I noticed a pattern I had not noticed before.

For the web version of YouTube, I can eyeball which URLs are ads and which are the videos I want to watch. Take a look:

How am I able to eyeball which video URLs are ads in this chaos?

Take a look at the query parameter range. For the web version, a chunk of the video I want is fetched from the 0th byte, then immediately another video is fetched with a range starting again at the 0th byte. Both happen near-simultaneously – faster than a human can click on a new video. It turns out this, as well as examining the clen parameter for the length of the full video (short videos are likely ads), can reasonably allow us to detect and doctor ad videos.

However, the iOS YouTube protocol does not use the range query parameter or even the Range header; video chunks use a counter like &nr=2 and &nr=3 etc. We must reverse engineer the Protobuf responses.

Top ↩

Decode the YouTube Protobuf Responses

Here are some decoded Protobuf log files I created then opened in the PyCharm IDE.

After logging decoded Protobuf messages to disk for offline analysis, I did notice something that piqued my interest.

I wonder what would happen if I were to, say, toggle those? This is tantalizing, but it is cheating, and hence no fun. Back to heuristics.

Let’s start by blocking the ads as intended.

Top ↩

Ad URL Polymorphism

The Protobuf responses are a hot mess of bytes, but there are human-readable URLs that can be grepped.

You’d think a simple LRU cache that blocks soon-encountered ad URLs could be the way to go, but, alas, the ad URLs do not quite match the URLs sent over the wire. Also, who is to say that YouTube won’t randomize the position of query-string parameters one day? We need an O(1) lookup of flagged ad URLs that are polymorphic (and group homomorphic) to live ad URLs.

It might be tempting to split a query string into a sorted dictionary and reassemble it, but we have no way of knowing what the query string boundary is. Plus, a live ad URL could add a key and disrupt the sorting.

Addionally, I’ve encountered URLs like this that purposely try to obfuscate the query params:

https://r4—sn-vgqsrns6.googlevideo.com/videoplayback
/expire/1640607416
/ei/WFrJYdWnFfyTsfIP4s2BsAk
/ip/121.35.98.26
/id/o-AE7swWOPOwXu3GyRght
/source/youtube
/requiressl/yes
/mh/wU/
mm/31,26/…

Notice how /ip/121.35.98.26/ is just &ip=121.35.98.26?

I propose heuristically scanning for query and path parameters of ad URLs with high entropy and using those as keys (fingerprints). For example, in

https://rr6—sn-uxa0n-t8gz.googlevideo.com/initplayback?source=youtube
&orc=1&oeis=1&c=IOS&oss=1&oda=1&oad=5500&ovd=5500&oaad=11000&oavd=11000
&ocs=700&oputc=1&oses=1&ofpcc=1&osbr=1&osnz=1&msp=1&odeak=1&odepv=1
&osfc=1&id=58cc678216d6aaca&ip=121.35.98.26&initcwndbps=2125000
&mt=1640373902

One could note the following candidates in descending order of length:

• rr6—sn-uxa0n-t8gz
• 58cc678216d6aaca
• 121.35.98.26
• 1640373902
• 2125000

Any or all of them could be lookup keys each pointing to the same dictionary of deconstructed query parameters. A lookup of a live URL would involve the same process of finding the highest entropy parameters and checking the URL dictionary for a match. The cache data structure can even be multi-level with the root keys being just the length of the high-entropy strings.

Top ↩

Smoke Test: Intercept and Decode Protobuf in Python

Decoding ~500 kiB of Protobuf in pure Python, especially the decoding step of converting it to over 1 MiB of human-readable text to parse the ad URLs, takes more time than the connection timeout most of the time. I’ll run some benchmarks using pure Python vs. the native C++ library.

Pure Python Benchmarks

Pure C++ Benchmarks

If you caught that, it takes about 23s in Python, and 100ms in C++! In this Never Ending Story, we have to find a way to parse the raw Protobuf payloads in Python using the C++ library libprotobuf.so. In the interest of time, I’ll use subprocess.Popen and communicate with the C++ protoc binary directly (since raw decoding is not supported in Python anyway).