The Invisible Things Lab's Blog: Playing With Qubes Networking For Fun and Profit
The Invisible Things Lab's Blog: Playing With Qubes Networking For Fun and Profit
The Invisible Things Lab's Blog: Playing With Qubes Networking For Fun and Profit
Joanna Rutkowska
Today, I would like to showcase some of the cool things that one can do
with the Qubes networking infrastructure, specifically with all the new
features that have been brought by the just released Qubes Beta 2. This
will cover the use of multiple Net VMs for creating isolated networks, the
use of a Proxy VM for creating a transparent Tor Proxy VM, as well as
Founder of Invisible Things Lab, Qubes OS project
demonstration of how to use a Standalone VM with manually assigned
lead.
devices, to create a “WiFi pen-testing” VM, which surely represents the “for
View my complete profile
fun” aspect of this post.
From the networking point of view there are three types of VMs in Qubes: twitter: @rootkovska
Qubes Project
Invisible Things Lab
Net VMs, that have networking devices assigned to them, such as e.g.
a WiFi or Ethernet card. Each Net VM contains a Xen network backend
that is used to provide networking to all VMs that are connected to this
Net VM. Blog Archive
Regular VMs (AppVMs) that use the networking provided by Net VMs
► 2015 (1)
(so they have Xen network frontends that provide virtual interfaces that ► 2014 (6)
are backed by the backend in the corresponding Net VM. ► 2013 (8)
Proxy VMs that combine both of the above: to Net VMs they look like ► 2012 (8)
regular AppVMs, because they are consumers of the networking they ▼ 2011 (17)
provide, but to other AppVMs they act as if they were Net VMs ► December (2)
themselves, allowing other VMs to connect to them. Of course the ▼ September (3)
Proxy VMs do not have directly assigned networking devices – they Playing with Qubes Networking for Fun and
Profit
use the networking provided by the Net VM that they connect to. One
Qubes Beta 2 Released!
can chain many Proxy VMs, as we will see below.
Anti Evil Maid
► August (1)
► June (3)
The virtual interfaces in client VMs are called ethX , and are provided by the
► May (3)
xen_netfront kernel module, and the corresponding interfaces in the
► April (3)
Net/Proxy VM are called vifX.Y and are created by the xen_netback
► March (2)
module. ► 2010 (15)
► 2009 (21)
Each Net and Proxy VM implements NAT, specifically masquerading, for all ► 2008 (15)
the connected VMs. Additionally to this SNAT, each Net or Proxy VM ► 2007 (15)
provides also DNAT redirection for DNS resolutions, so that each VM ► 2006 (8)
behind a Proxy or Net VM thinks that it uses a DNS in the Net/Proxy VM,
but in fact all the DNS request are DNAT-ed by all the Proxy and Net VMs
down the original DNS that is provided to the final Net VM. This smart trick Labels
allows us to avoid running a DNS caching server in Proxy/Net VMs.
qubes general trusted computing
attack os security philosophical fighting
Also, any VM-to-VM traffic, among the VMs connected to the same
for a better world exploit company news
Net/Proxy VM is blocked by default. trusted execution technology xen hacking tpm
1 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
The current infrastructure doesn't support IPv6 routing, but we will likely
add this support in the upcoming Beta 3.
When you proceed with the default installation of Qubes Beta 2, then your
initial networking topology looks like on the diagram below:
So, by default there is one Net VM, called 'netvm', that is automatically
assigned all the networking devices in the system. There is also one Proxy
VM, called 'firewallvm' that is directly connected to the default Net VM, and
which provides networking to all other VMs in the system. This Proxy VM is
used for firewall rules enforcement. Each such service VM consumes
200MB of RAM by default.
Network-isolated VMs
For example I have a 'vault' VM that I use for keeping my master PGP
keys, and other secrets, and this machine is not connected to any network.
2 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
ethernet cable on the user's desk, and the unsafe and evil Internet,
reachable via WiFi card).
It is easy to use more than one Net VMs in Qubes, and assign different
networking devices to different Net VMs, and also decide which VMs are
connected to which Net VMs. The diagram below presents an exemplary
such setup:
A simple setup with two isolated networks, and one fully isolated domain ('vault').
Currently qvm-create when used with the --net option automatically assigns
all networking devices to the just created VM, so in the example above you
would want to remove extra devices from each Net VM using qvm-pci -d ,
And now, for any other VM, just set the appropriate Net VM (either
firewallvm1 or firewallvm2, or 'none), to get it assigned to either of the
isolated networks, e.g.:
3 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
...
Let's now play a bit with Proxy VMs and see how we can use it to create a
simple Tor proxy VM. Such a VM would provide anonymized networking to
all its clients, so would allow to easily create VMs for anonymous Internet
access. The simple setup we would like to prepare is depicted on the figure
below:
Our Tor proxy would forward only the Tor traffic, so we don't have to fear
about some Tor-not-aware applications, or even intentionally malicious
ones to compromise the privacy of our connection. This is because such
applications have no way to generate traffic to the outside world without
going through our Tor proxy (unless they could exploit a hypothetical
vulnerability in the Tor process running in the Tor VM). Also, the
applications running in any VM behind the Tor proxy are not able to
determine any globally identifiable IDs, such as the user's external IP
address, the real MAC address used by real NICs, etc.
Interestingly just after writing the above paragraph, I discovered that one of
our xenstore keys had wrong permissions and, as a result, any VM could
read it and get to know the actual external IP (the key is used by a Net VM
to communicate the external IP configuration to the connected Proxy VMs,
so they could know when to update the firewall configuration). The fix for
this problem is here, and the update (qubes-core-dom0-1.6.32) is now
available for Dom0 (just do qvm-dom0-update to get it installed).
4 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
So, this represents a rather strong setup for use with Tor. Let's now have a
look at how to practically create such a configuration, step by step.
First, let's create the VM that will become our Tor proxy:
This will create a Proxy VM named 'torvm', based on the default template.
We will need to now start the template VM and install the Tor client there:
Now, in the just started template VM, lets install the Tor client and
(optionally) the Vidalia graphical frontend:
And then power off the template VM. Now, every VM based on this
template, started after the template shutdown, will also see the Tor binary in
its filesystem.
Let's now configure our torvm to properly start Tor proxying at boot:
Now, we will create the following script for starting up the Tor transparent
proxy and setting up traffic redirection using iptables:
#!/bin/sh
killall tor
QUBES_IP=$(xenstore-read qubes_ip)
TOR_TRANS_PORT=9040
if [ X$QUBES_IP == X ]; then
echo "Error getting QUBES IP!"
echo "Not starting Tor, but setting the traffic redirection
anyway to prevent leaks."
QUBES_IP="127.0.0.1"
else
/usr/bin/tor \
--SocksPort 0 \
--TransListenAddress $QUBES_IP --TransPort $TOR_TRANS_PORT \
--DNSListenAddress $QUBES_IP --DNSPort 53 \
--RunAsDaemon 1 --ControlPort 9051 \
|| echo "Error starting Tor!"
fi
5 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
Except for the “QUBES_IP=$(xenstore-read qubes_ip)” line that reads the torvm's
IP address, there is nothing Qubes-specific in the above listing. It's just a
standard way of setting up transparent Tor proxy.
It is important that this file be located in the /rw directory, as this directory is
backed by the VM's private storage and will survive VM reboots. The VM's
root file-system is read-only and all the changes to it are lost on VM
shutdown (VM gets an illusion of the root fs being writeable thanks to
Copy-On-Write mechanism, but the actual COW backing device is cleared
upon each VM shutdown).
We should also modify the /rw/config/rc.local script, to ensure that our Tor
proxy is automatically started -- just paste the following into this script:
#!/bin/sh
#!/bin/sh
/rw/config/start_tor_proxy.sh
Make sure that all the scripts are executable (chmod +x). And that's all.
Now, shutdown the torvm:
From now on, every time you start the torvm (or when Qubes starts it in
response to start of some other VM that uses torvm as its Net VM), the Tor
transparent proxy should be automatically started.
Let's test this by creating a VM that would be using the just created Tor
proxy:
6 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
Now, every time you start the anon-web VM (e.g. by clicking on the Web
browser icon in the anon-web's start menu), Qubes will also ensure that
torvm is up and running, and this in turn would configure all the Tor
proxying for this VM.
Fo additional control one might want to use Vidalia, the graphical front end
for Tor (this should be installed within the template VM that has been used
for torvm). We could easily start Vidalia by just typing:
We should however make sure to disable "Start the Tor software when
vidalia starts" option in Settings/General in Vidalia. Otherwise, Vidalia might
kill your original Tor (that has transparent proxy open) and start own without
transparent proxy enabled.
The web browser runs in the 'anon-web' VM that uses 'torvm' for networking access, and
thus all the traffic generated by 'anon-web' is routed through the Tor network, or discarded
if it's a different traffic than TCP or DNS.
Of course one case easily create more VMs that would be using torvm as
their Net VM, as so would have anonymized network access. The beauty of
this solution is that in case one of my anonymized VM gets compromised,
others do not. Plus, the already mentioned benefit, that no matter whether
apps in those VMs are buggy, or even intentionally malicious, they would
not be able to leak out the user's external IP address.
Finally let's have some fun and create a WiFi pen-testing VM. The desired
config is depicted below:
7 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
Because we would like to use all sorts of l33t h4x0r t00lz pen-testing
security software in this VM, it would make sense to create it as a
Standalone VM, which means that it would get its own copy of the whole
file-system (as opposed to just the home directory, /rw and /usr/local , as it
is the case with regular Qubes VMs). This would ease the installation of all
the extra software we would need there, and also ensure that even if the
install/build scripts were malicious, the damages would be contained only
to this very VM and nothing else. Also, for some reason the standard Linux
WiFi stack and drivers still don't support injection on (all?) most of the WiFi
cards out of the box, so we would need to patch the actual kernel drivers --
yet another reason to use a Standalone VM in this case.
So, let's create the VM first, and assign a WiFi card to it:
startup
[dom0]$ qvm-prefs -s wififun kernel none # use own copy of kernel and
modules
You can easily find the BDF address of any device using the lspci
Because it's a Standalone VM, Qubes will make a copy of the whole root
filesystem, and thus it would eat about 5GB of your disk (normal VMs
would take only as much space as their private fs takes up).
... and then install the prerequisite software there, starting with downloading
the reasonably new compat-wireless sources, together with the required
injection patches, and then building and installing the new kernel modules.
All actions below are now executed within the VM. This stuff here is really
nothing Qubes- or Xen-specific -- one would do more or less the same on
any Linux in order to get injection working (so, treat this as a free bonus
WiFi hacking tutorial on Linux).
8 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
2.6/compat-wireless-2011-07-14.tar.bz2
one-maxim.patch
fix-tx-ctl-no-ack-retry-count.patch
/mac80211.compat08082009.wl_frag+ack_v1.patch
[wififun]$ cd compat-wireless-2011-07-14
count.patch
../mac80211.compat08082009.wl_frag+ack_v1.patch
[wififun]$ make
Now, lets reboot the VM to ensure that all the patched drivers will get
properly loaded on each VM boot:
Let's first see if the WiFi driver got properly loaded and if the interface has
been created (look for wlanX interface):
[wififun]$ ifconfig -a
If yes, then proceed with the steps below (if not, then have a look into
dmesg and see what was the problem):
Note that you don't need to add any explicit masquerading rules, as they
are applied by default on Qubes VMs (you can take a look at the nat table
in the VM if you want to see by yourself).
interface=at0
dhcp-range=192.168.0.50,192.168.0.150,12h
9 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
and then start the dnsmasq daemon -- we will use it for providing DHCP to
our fake AP (the at0 interface will be created by airbase-ng and emulates
the “uplink” of a traditional AP):
and on another console (before any client connects, but after airbase-ng
got started), configure the at0 interface (make sure it matches what you
wrote into dnsmasq.conf):
... and now, just wait for a client to connect to your AP. What you do next is
only limited by your imagination... But hey, this article is about Qubes
networking and not about 0wning client systems ;)
Here's an innocent example using Moxie's sslstrip (amazing this attack still
works so well at the end of 2011...):
My 'wififun' VM in action using a simple sslstrip attack, that surprisingly still works pretty
nice...
When using custom driver domains, there are currently some catches you
should be aware:
Catch #1: When you start a driver domain late after system boot, so after
some days of uptime and extensive use of VMs, Xen might not be able to
allocate enough continues (in terms of MFNs) memory for a driver domain.
And PV driver domains, unlike normal domains or HVM driver domains, do
require MFN-continuous memory for their DMA buffers (HVM domains do
10 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
not need that, because IOMMU can create an illusion of this; even though
IOMMU is also used for PV driver domains, for protection, it doesn't actively
translate bus addresses into GMFNs).
This is usually not a big problem in practice, because in most cases all the
driver domains are started early at system boot, when there is still plenty of
non-fragmented memory available. However it might become a problem
when one wishes to start e.g. the WiFi pen-testing at some later time. The
work around is to close as many VMs as possible before starting such
driver domain, and then also reducing, for a moment, the amount of
memory assigned to Dom0:
and then starting the driver domain should be fine. Now we can start all
other domains, and that should no longer be problematic for the already
running driver domain.
Catch #2: Some network cards, notably Express Cards, might not work
well with the 3.0.4 pvops kernel that we use in all VMs by default. In that
case you might want to try to use the 2.6.38.3 xenlinux kernel in your WiFi
fun VM -- to do that, follow these steps:
vm-2.6.38.3-10.xenlinux.qubes
/wififun/kernels/
And rebuild the compat-wireless, unload, install modules, and then load
drivers again.
Summary
As you can see, Qubes Beta 2 now offers a very advanced networking
infrastructure that allows more advanced users to create very sophisticated
configurations, allowing for pretty good isolation between various domains
and networks. Qubes leaves it up to the user (or admin) to figure out what
would be the best configuration -- most users would be happy with the
default simple setup with just one Net VM and one Firewall VM, while
others would go for much more advanced setups.
11 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
A bit more advanced networking setup. The usbvm has a 3G modem assigned, and it is
possible to dynamically switch between the Net VMs without restarting any other VMs.
10 comments:
Idler said...
More of a Tor question but can 1 physical PC run multiple concurrent secure
channels?
e.g. So that I can have one VM + tor channel for hacking the Pentagon and
a different pair for posting to my anonymous blog.
Idler said...
Also, does Qubes work in a multi user fashion i.e. on my home PC for
myself + kids?
Thx
October 03, 2011 1:19 AM
Erik said...
I would not be surprised at all if the Qubes model becomes the defacto
standard in all operative systems in the future.
October 03, 2011 10:41 AM
@idler: 1) if you trust Tor, then one Tor VM should be enough (you can just
use multiple VMs connected to the Tor VM).
12 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
Anonymous said...
johnxyz said...
Hi
I would like to ask if there will be possible to run normal VMs, beside
AppVMs in Qubes and install standard operating systems in those VMs. For
example If I will need to do some work on Debian, so I will dont need reboot
computer (and running other OS on bare metal is not a good thing, beacuse
it will not be as secure, as having only Qubes installed). Also it will allow to
use simulatenously apps in Qubes AppVMs, and other operating system.
@johnxyz: support for "normal" VMs is planned after Qubes 1.0, sometime
in the first half of 2012.
December 14, 2011 9:18 PM
Anonymous said...
Is that an error:
Levi said...
Wow, just came across Qubes today. I'm going to give it a go tonight.
Thanks for sharing your awesome work.
March 10, 2012 6:11 PM
Anonymous said...
I can't seem to write any of the scripts to it without sudo, but then even
when I set them to executable they do not start with the vm.
March 25, 2012 7:36 PM
Post a Comment
13 of 14 02/13/2016 04:50 PM
The Invisible Things Lab's blog: Playing with Qubes Networkin... http://theinvisiblethings.blogspot.nl/2011/09/playing-with-qube...
14 of 14 02/13/2016 04:50 PM