jorgenmodin.net - Blog
Eff.org har en ranking av vilka meddelande-applikationer som har bäst säkerhet. För att få full pott så måste det vara open source. WhatsApp har nu 6 av 7 poäng, med väldigt genomtänkt end-to-end kryptering enligt EFF, men inte open source så det går ju inte att kolla https://www.eff.org/deeplinks/2016/04/whatsapp-rolls-out-end-end-encryption-its-1bn-users
En annan fråga är ju om det är bra eller dåligt med kryptering. Hur som helst, full pott går till:
- ChatSecure + Orbot, https://chatsecure.org/
- Jitsi + Ostel, jitsi.org | Jitsi
- Off-The-Record Messaging for Windows (Pidgin) , Pidgin, the universal chat client
- Signal / RedPhone, Open Whisper Systems, Free, Worldwide, Encrypted Phone Calls for iPhone
- Silent Phone Silent Circle
- Silent Text, Silent Circle
- Telegram (secret chats) Telegram Messenger
- TextSecure.Open Whisper Systems >> Home
It seems that the last step, folding back a snapshot in kvm, does not work in Ubuntu 15.10 and indeed has a lot of problems on other systems too. Folding back:
virsh blockcommit guestName vda --active --pivot --wait
error: internal error: unable to execute QEMU command 'block-commit': Device 'drive-virtio-disk0' is busy: block device is in use by block job: commit
It works if you stop the guest, but that kind of defeats the purpose since it is not backing up a running machine anymore. One may try to use RedHat instead and see if patches there have been quicker, since the Qemu/Kvm people seem to be aware of the problems.
Summary 2016-04-08: KVM can be used and can do this (but last step does still not work, see above), but the configuration and packages in Ubuntu 15.10 are buggy, so you have to replace stuff, and disable stuff:
Backing up a running virtual machine on Ubuntu 15.10 turned out to be not so simple. A lot of the information that Google return is outdated, giving recommendations pertaining to 8 year old Ubuntu systems or claiming that KVM cannot run GUI systems.
I started out with Virtualbox, and Virtualbox has something called snapshots. As far as I understand, when you take a snapshot, the main vdi file becomes read-only and all changes go into a new snapshot file. Several snapshots can be made and they can reference each other and branch, and be folded back into the main vdi file and into each other. I followed this guide:
# != 1 ]
"Usage: $0 VBoxName"
"Renaming old snapshot..."
edit previous --name deleteme
"Renaming current snapshot..."
edit current --name previous
"Taking new snapshot..."
"Deleting old snapshot..."
But I ended up with errors and inconsistent snapshots that were corrupt according to Virtualbox. Which is probably my fault somehow. Digging deeper I found a blog post referencing the same script, OSC — Backing up Virtual Machines in VirtualBox and that guy writes that he has gone back to shutting down the virtual machines and then back them up:
Sadly, for me personally, live VirtualBox snapshots haven’t been a terribly robust backup strategy. I’ve unfortunately seen several snapshots fail. Or, worse, had VirtualBox crash while a snapshot was taking place.
So next solution, why not put the virtual machine on a snapshotting file system, that is guaranteed to copy the vdi file in one fell swoop? This may not be enough, in the sense that there might be state stored in other files or in RAM. But it is worth a shot. One option could be btrfs, which I have been experimenting with and which has a very smooth snapshotting mechanism: You can snapshot any file or director on a btrfs volume. But timely enough, this was posted on Reddit's /r/linuxadmin:
which thread says things like:
Yes. I do not recommend this. I spent more time maintaining the one client I had running VMs on a BTRFS store than I did the fifty or so others I had running VMs on ZFS stores, for roughly a year. The replication is unreliable, the performance is incredibly hit-or-miss
The poster lauds ZFS though. ZFS is supported well in Ubuntu 15.10, but the ZFS snapshotting is not as simple as btrfs. Just using LVM should also be an option.
So, what about changing the virtual machine from Virtualbox to something else? Three options are:
VmWare costs money for the snapshotting version. I did however vaguely remember that Linode switched from one VM technology to another recently. A Google search turned up that they switched from Xen to KVM.
Ok KVM it is. What is available snapshot wise there then?
Well firstly there is a way of freezing the state of a VM, backup and then unfreeze it again Backup of running KVM qcow2 VPS - Server Fault.. Secondly it seems to have the same ability to do snapshots with mutable snapshot files as VirtualBox, but there is more documentation giving examples of what I think I want.
This is the best I have found so far [libvirt-users] Backup a VM (using live external snapshot and blockcommit:
# Create snapshot virsh snapshot-create-as --domain $VM_NAME snap --diskspec vda,file=$VM_DIR/"$VM_NAME"-snap.qcow2 --disk-only --atomic --no-metadata --quiesce # Copy frozen backing file cp $VM_DIR/"$VM_NAME".qcow2 $SNAP_FILEPATH # Blockcommit snapshot back into backing file virsh blockcommit $VM_NAME vda --active --pivot # Remove snapshot file rm $VM_DIR/"$VM_NAME"-snap.qcow2 # Variables should be self-explanatory: # - VM_DIR is the directory where the VM are stored # - VM_NAME is the name of the VM, and its qcow2 file is called # VM_NAME.qcow2 # - SNAP_FILEPATH is the full path (including name) where the backup # should be created
This seems to be essentially the same KVM – Live backups with qcow2 | Gonzalo Marcote | Open source, open mind.
Some more info here too: Live-disk-backup-with-active-blockcommit - Libvirt Wiki
The above script will only work if the following criteria are fulfilled:
- Apparmor does not interfere. One heavy handed option is to disable Apparmor altogether (I did this, see AppArmor for how-to)
- Qemu guest agent is installed in the guest
- There is a communications channel between the host and the qemu guest agent, see: QEMU Guest Agent | Gonzalo Marcote | Open source, open mind
- As of 2016-04-08 on Ubuntu 15.10, you need to manually download and update new versions of libvirt-bin and libvirt0, see discussion here: Bug #1517539 “Libvirt KVM can not create snapshot (with qemu-gue...” : Bugs : libvirt package : Ubuntu
Download links for amd64 are (to the best of my understanding and what I used):
Install with sudo dpkg -i libvirt*
(assuming you only have those two files in the directory)
Unfortunately, without tuning or disabling AppArmor there is currently no way to flatten the snapshot back into the image, as of the last comment currently at Bug #1517539 “Libvirt KVM can not create snapshot (with qemu-gue...” : Bugs : libvirt package : Ubuntu:
"Strictly speaking, the virsh command that prompted this ticket has been fixed. So, I can now successfully create a live snapshot using the proposed packages. Unfortunately, I have no way to flatten the snapshot back into the base image now."
However after having disabled AppArmor, it works, it seems.
Moving from VirtualBox to KVM
Was disarmingly simple. As long as you follow the up-to-date guides and use the virt-manager GUI, you can actually just use the vdi file directly in KVM. However then you will not get the snapshotting abilities. For that you need the qcow2 disk format. Which you can convert to directly from vdi like this:
...seems very difficult. I tried the script listed here:
but I end up with corrupt stuff, including the entire Virtual machine. VBoxManage says it cannot delete the deleteme copy and then it all goes downhill from there.
I am probably doing something wrong, but I am now looking at KVM or Xen to see if they can do snapshots. This is a lot trickier than I thought. Maybe I will need to settle for stopping the Virtualbox vm, back it up and then start it again, or somehow back up its entire state from within itself. But that does not feel right.
I'm also looking into using a snapshotting file system such as btrfs for snapshotting VM state, but btrfs has a lot of problems in this context according to this thread:
Tested (with .mv4 files). Works. 90 rotates the video 90 degrees clockwise.
ffmpeg -i input.mp4 -c copy -metadata:s:v:0 rotate=90 output.mp4
Uttrycket religiös extremist är lite konstigt. En religiös extremist är ju inte mer religiös än en vanligt religiös person. Bara knäppare.
I det antika Rom (innan kristendomen) hade man ett ord för sådana människor som trodde att de kunde manipulera gudarna för egna syften, eller trodde sig vara en själv. Man kallade dem vidskepliga. Man kallade dem inte mer religiösa.
En religion, speciellt de som är baserade på religiösa texter, måste tolkas utifrån den tid texterna skrevs i, och också tolkas för den tid vi lever i nu. Att inte göra det är inte mer religiöst, det är bara knäppt.
This seems to be possible often with I'M Intelligent Memory | Beyond Limits, a company that stacks memory units beyond the specifications. A compatibility list is here: compatibilitylist.pdf, and you can buy them here in Europe: Memphis Electronic AG @ Suchergebnis auf Amazon.de (a bit pricey I'd say, may be cheaper to buy a motherboard with more memory slots).
There are a number of open source solutions for VPN, such as OpenVPN, SoftEther and strongSwan. They all take a bit of learning to setup, no actually for StrongSwan and SoftEther there is a massive amount of learning and OpenVPN is not trailing that far behind. If you're like me.
And then there is sshuttle, a python program that uses SSH to make a tunnel to the server. A server that does not need to have sshuttle installed: The client sshuttle will connect and run the needed stuff on the server side in a similar way to e.g. Ansible. I just tested it and it seems to work fine!
Forward all traffic except DNS:
sshuttle -r username@sshserver 0.0.0.0/0
Also forward DNS queries:
sshuttle --dns -r username@sshserver 0/0
I installed it (you can use apt-get, yum or pip) and then just ran it from terminal. Done. It works!
I can read through sshuttle's code, it's 3283 lines of python code (I haven't yet I might add).
According to the SoftEther site, OpenVPN has 91'000 lines of C code and SoftEther has 378'000 lines of C and C++ code.
Now granted, they do much more. I haven't tested it much yet but sshuttle looks promising. I wonder a bit about throughput though, gotta check that. And there is no Android client.
There is also tinc by the way which seems quite interesting in other ways. Untested by me.
Wikipedia has a good page:
Observations: 10Gbits/s to 200Gbits/s seems to be the limit for transfer speed, depending on whether you are using copper, optics or RAM access over a short distance. Video cards go higher, not sure if that could be useful somehow outside of the realm of graphics.
Update 2016-04-05: There is a standard called Thunderbolt that is able to do 10Gbits/s, 20Gbits/s and in Thunderbolt 3 40Gbits/s. Thunderbolt seems to be mostly used to drive digital displaysand is mainly serial.
The fastest available SSDs seem to often max out at 5Gbits/s Best SSD 2016 - 119 Charts - UserBenchmark
Updated and clarified 2016-03-14
Zram and Zswhap are RAM compressors where your get more space in RAM for using a bit more CPU. This can make your computer handle bigger tasks without slowing down to a crawl or behave erratically.
Zram is supposed to make low memory devices use their memory more efficiently. It is apparently used a lot by e.g. TV manufacturers for their embedded Linuxes. Zram seems to work in such a way as to take RAM away from the computer and set it aside as a swapped RAM disk that uses lzo compression. So it's basically a RAM memory compressor.
Zram does not need a swap partition on a drive and in fact it may behave non-optimally if there is one: First Zram would then fill up, and any pages after that would be swapped to disk, which if we assume a LIFO usage pattern would put all the pages most in demand on the slow disk.
Zswap on the other hand is meant to improve swapping on low-memory systems with rotary hard disks or similar slow swap partitions. It also compresses memory pages and stores them in RAM, but it communicates with swap partitions present on the system and tries to cache in RAM in compressed format the pages most likely to be swapped in again any time soon.
Zswap probably should do good on any-sized system strapped for RAM with a slow swap disk. Now trying it on an 8GB system.
This is what it looks like after having run for a while on an 8GB system that hasn't been restarted since the install of zRam. It should have kicked in working, and it seems like it has shrunk the RAM use quite drastically and the swap just showing old pages since before zRam kicked in. Or I am misunderstanding something about its use. I think zRam is its own swap, but it does not show up when doing "swapon -s"
However zswap has a chequered history of being available or not depending on kernel build and is regarded as a bit unexplored:
You could also go and borrow RAM from another machine with https://oss.oracle.com/projects/tmem/dist/files/RAMster/ramster-howto.txt whiich seems to be a part of the Zcache stuff. However you are unlikely to get better bandwidth than from a local SSD methinks.