21 August, 2020

Elastic (part 1)

Over the last couple days I've started using Elasticsearch as part of an internal application.

From 0 to novice, there's a number of stumbling blocks that aren't really clear in the documentation. I hope this helps others frantically googling for these same issues 😊

Tutorials prior to version 7 don't work (without modification)

If you come across a tutorial or code snippet that uses _all, _default_, or _doc, it's not going to work. Those stopped working after Elasticsearch 6, with the removal of mapping types.

'Limit of total fields [1000] in index [index-name] has been exceeded'

By default, Elasticsearch indexes all fields in your document. Therefore, if you have data that has dynamic field names, you may not want to have all of them indexed.

To stop Elasticsearch from indexing all fields (i.e., if you just want to retreive documents by their ID), before you upload data, you can set the mappings to only index certain fields (properties):

    "mappings": {
        "enabled": false

If you're using the Python API to load the data, you can use native dicts - you don't need to use JSON strings:

index_properties = {
    'mappings': {
        'enabled': False,

es.indices.create("cool-index", body=index_properties)

Also, it doesn't look like you can currently exclude all fields except certain fields, so something like this doesn't work (please, tweet me @jlaundry if this does start working):

    "mappings": {
        "enabled": false,
        "properties": {
            "field_i_want_to_index": {
                "enabled": true

"Can't merge a non object mapping [client.ip] with an object mapping [client.ip]" and "Can't merge a non object mapping [server.ip] with an object mapping [server.ip]"

This happens because the dotted notation used in the ECS syntax needs to be fully expanded into objects.

For example, when creating your index and setting types, don't do this:

    "mappings": {
        "properties": {
            "client.address": {
                "type": "keyword"
            "client.ip": {
                "type": "ip"
            "client.port": {
                "type": "long"

Do this instead:

    "mappings": {
        "properties": {
            "client": {
                "properties": {
                    "address": {
                        "type": "keyword"
                    "ip": {
                        "type": "ip"
                    "port": {
                        "type": "long"
4 March, 2020

Using Blob storage GZip files in Python Azure Functions

I recently had a project where I needed to process a lot of GZipped files, and decided to give Azure Functions a whirl.

I had the data flowing into Azure blob storage, but when I tried to process the GZip files using the below, I kept getting an error: Exception: OSError: Not a gzipped file (b'\x1f\xef').

def main(msg: func.QueueMessage, inputblob: func.InputStream) -> None:
    filename = msg.get_body().decode('utf-8')
    logging.info(f"Working on queue item: {filename}")

    with gzip.open(inputblob, mode='rt', encoding="utf-8") as gzf:
        # Do stuff

By logging out with print(inputblob.read()), I saw something completely unexpected: the first few bytes of the input stream were \x1f\xef\xbf\xbd\x08\x00\x00\x00\x00\x00\x00\x00\xef\xbf\xbd, which had lots of UTF-8 BOM sequences (\xef\xbf\xbd).

Long story short, turns out Azure was trying to serve the GZip binary as a UTF-8 string, and there's a hidden option you can set in your function.json called dataType, which stops this sillyness.

    "type": "blob",
    "direction": "in",
    "name": "inputblob",
    "dataType": "binary",

    "path": "files/{queueTrigger}",
    "connection": "MyStorage"

Once I set the dataType to binary, gzip.open() worked fine.

26 February, 2019

Downloading .eml message content from the Microsoft Graph API

In order to report spam to Cert NZ and our upstream filter provider, I need to get copies of the mail messages with their original headers.

I used to do this with PowerShell, but the Exchange with Office 365 glue seems to be tearing apart with each change Microsoft make to their authentication systems. So here's a python script to pull it out via the Graph API instead

The API endpoint that allows us to do this is currently in Beta, and thanks to the Microsoft team for implementing this!

To get started:

  1. Go to https://apps.dev.microsoft.com/#/appList (with a Global Admin account) and register a new Converged application. Converged apps are so hot right now.
  2. Grant it the Mail.ReadWrite.All and User.Read.All Application Permissions, and don't forget to grant consent for your whole org.
  3. Create a new secret. Keep it safe.
  4. Replace the <tenant> and <app ID> with your tenant name and the Application Id from the registration
  5. Run it: python download_eml.py "[email protected]" "totally not spam"
8 August, 2017

How to get a report of Computers needing Approved Updates from WSUS using PowerShell

For some reason, Microsoft didn't include a predefined report in WSUS to show me the list of computers that are needing Approved updates... which is kinda useful for reporting/tracking purposes.

So here's a quick PowerShell to do it.

$report = @{}

$wsus = [Microsoft.UpdateServices.Administration.AdminProxy]::getUpdateServer("wsussvr", $False, 8530)

$computerScope = new-object Microsoft.UpdateServices.Administration.ComputerTargetScope
$wsus.GetComputerTargetGroups() |
    where {$_.Name -match "Workstations"} |
    ForEach-Object {
        $gid = $computerScope.ComputerTargetGroups.Add($_)

$updateScope = new-object Microsoft.UpdateServices.Administration.UpdateScope;
$updateScope.UpdateApprovalActions = [Microsoft.UpdateServices.Administration.UpdateApprovalActions]::Install
$updateScope.IncludedInstallationStates = @('Downloaded', 'Failed', 'InstalledPendingReboot', 'NotInstalled')
$updateScope.TextNotIncludes = 'Feature update to Windows 10 Pro'
$updateScope.TextNotIncludes = 'Definition Update for Windows Defender'

foreach ($computer in $wsus.GetComputerTargets($computerScope)) {
    $key = $computer.FullDomainName
    $computer.GetUpdateInstallationInfoPerUpdate($updateScope) | foreach-object {
        $title = $_.GetUpdate().Title
        if ($report.ContainsKey($key)) {
            $report[$key] += 1
            #$report[$key] += $_.GetUpdate().Title
        } else {
            $report[$key] = 1
            #$report[$key] = @($_.GetUpdate().Title)

8 August, 2017

How to remove users from AD groups using PowerShell

I recently cleaned up permissions on our AD, and found that working methodically though the Department/Company attributes first paid dividends in removing extraneous groups that users had access to:

$group = 'GroupToBeCleaned'
$users = Get-ADGroupMember $group | Get-ADUser -Properties * | ? { $_.company -Like "*Contractor*"}
foreach ($user in $users){
    Remove-ADGroupMember -Identity $group -Members $user -Confirm:$false
10 January, 2017

PulseSecure Client on Fedora 25

Just installed the PulseSecure 5.2R6 client on Fedora 25; while it isn't officially supported (support is only for fc23), it works well, and as all the dependancies are in the Fedora repo, you don't have to do any dodgy downloads (like you do with the officially supported fc23...).

This will only give you VPN access to a PCS appliance using their proprietary SSL VPN; it doesn't support IPSec connections, you'll still need to patch vpnc for that.

sudo rpm -Uvh ps-pulse-linux-5.2r6.0-b977-centos-rhel-installer.rpm
sudo dnf install -y glibc.i686 nss.i686 zlib.i686 glib-networking.i686 \
  webkitgtk.i686 xulrunner.i686 libproxy.i686 libproxy-mozjs \
  libproxy-gnome.i686 webkitgtk3.i686
sudo ln -s /usr/local/pulse/libpulseui.so /usr/lib/libpulseui.so

After that, the Pulse UI client is available in GNOME; just add the details and you're away.

3 August, 2015

Android Stagefright: Exchange PowerShell Snippets

So, this Stagefright thing sounds bad. We'll obviously know more later this week about how bad it is. If you're a BYOD happy organisation, right now you need to know how your organisation will be affected; how many Androids do you have, what versions, etc, and then what steps you'll need to take to remotely wipe/quarantine/block devices. And you don't want to be spending all night doing it. PowerShell to the rescue!

First, create an export directory, and assign Full Control permissions to the Exchange Trusted Subsystem group - I've used D:\ExchangeExport in these examples. Next, get a list of all your Android devices:

    | where {$_.DeviceOs -like "*Android*" -or $_.DeviceOs -match ""}
    | Export-Csv -path D:\ExchangeExport\android.csv

Some devices don't provide a DeviceOs, and devices that have been migrated from Exchange 2010 to 2013 seem to sometimes have an empty DeviceOs string, so you'll have to manually filter these. Next, get their last sync time (to determine devices which haven't been seen in a while) with:

    | where {$_.DeviceOs -like "*Android*" -or $_.DeviceOs -match ""}
    | foreach { Get-MobileDeviceStatistics -Identity $_.Identity }
    | Export-Csv -path D:\ExchangeExport\android-stats.csv
21 September, 2014

Juniper SRX Dynamic VPN with Fedora 20

Update 2014-12-02: I've updated the below process for vpnc-0.5.3-svn550, which hit Fedora 20 a few weeks ago and will be present in Fedora 21.

Hot on the heels of the work I've done with Ubuntu, I've also done the same for Fedora 20 vpnc...

sudo yum install rpm-build libgcrypt-devel gnutls-devel gtk3-devel dbus-devel NetworkManager-devel \
 NetworkManager-glib-devel intltool libgnome-keyring-devel perl-LWP-Protocol-https perl-Data-UUID -y

yumdownloader --source vpnc
yumdownloader --source NetworkManager-vpnc

rpm -Uvh vpnc-0.5.3-21.svn550.fc20.src.rpm
rpm -Uvh NetworkManager-vpnc-

cd ~/rpmbuild

curl https://gist.githubusercontent.com/jlaundry/0c9f32176924c7486762/raw/631c0ac68c611ed7ba519252bf769f75466cbd25/build.patch > build.patch
#curl https://github.com/ndpgroup/vpnc/commit/8f005fefbc8713535d59f95e3abee8a45b05399a.patch \
 > SOURCES/vpnc-0.5.3-juniper.patch
curl https://gist.githubusercontent.com/jlaundry/e54c6a152eafd7c2bb97/raw/707286377f01d66b066ddaaa36e5c57784134af8/vpnc-0.5.3-juniper.patch \
 > SOURCES/vpnc-0.5.3-juniper.patch
curl https://gist.githubusercontent.com/jlaundry/036ed1719a4dda561fc2/raw/224bf3f605fac6a6df191cfe14cdad95d7400830/NetworkManager-vpnc- \
 > SOURCES/NetworkManager-vpnc-

patch -p1 < build.patch

rpmbuild -ba SPECS/vpnc.spec
rpmbuild -ba SPECS/NetworkManager-vpnc.spec

sudo rpm -Uvh RPMS/x86_64/vpnc-0.5.3-20.svn550.juniper.fc20.x86_64.rpm
sudo rpm -Uvh RPMS/x86_64/NetworkManager-vpnc-*juniper*
21 September, 2014

Juniper SRX Dynamic VPN with vpnc Ubuntu 13.10

Pretty simple really; we use Juniper SRXes (running Junos 11.4) with Dynamic VPN at work, and I use an Ubuntu laptop. All the patches to make vpnc work with the SRX are available, but for some reason haven’t made it into official source yet…

Step 1: Patch vpnc

apt-get source vpnc
sudo apt-get build-dep vpnc
wget https://github.com/ndpgroup/vpnc/commit/8f005fefbc8713535d59f95e3abee8a45b05399a.patch
cd vpnc-0.5.3r512
patch < ../8f005fefbc8713535d59f95e3abee8a45b05399a.patch
dpkg-buildpackage -rfakeroot -uc -b
sudo dpkg -i ../vpnc_0.5.3r512-2ubuntu1_amd64.deb

Step 2: Patch network-manager-vpnc, because it’s not Ubuntu if it’s not a GUI! 😉

I’ve adapted the patch for NetworkManager from the version found here.

apt-get source network-manager-vpnc
sudo apt-get build-dep network-manager-vpnc
wget https://gist.githubusercontent.com/jlaundry/cbf79311bc46fcf6c626/raw/f142f086032c66b19fb182c2933ced139271275f/network-manager-vpnc_0.9.6.0-0ubuntu2-juniper.patch
patch < network-manager-vpnc_0.9.8.2-1ubuntu1-juniper.patch
cd network-manager-vpnc-
dpkg-buildpackage -rfakeroot -uc -b
sudo dpkg -i ../network-manager-vpnc-gnome_0.9.8.2-1ubuntu1_amd64.deb

Step 3: Create a NetworkManager script

wget https://raw.github.com/ndpgroup/juniper-srx-linux/master/jam-config
chmod u+x jam-config
./jam-config addr vpn.example.com user joe pass joespwd | sudo tee /etc/NetworkManager/system-connections/MyVPN

Finally, reboot your machine to flush out the old, non-Juniper-friendly NetworkManager, and you’re away!

8 September, 2014

Junos 12.1X44 Dynamic VPN with FreeRADIUS

One of the features Juniper added to the SRX Dynamic VPN starting with Junos 12.1X44 is the ability to set the VPN client group via RADIUS (eliminating the need to specify the client username).

What Juniper don’t tell you is how to do it; using the Juniper-Local-Group-Name VSA (vendor 2636 option 46). So, after some trial and error, here’s how:

Step 1: Configure the access profile and create the Dynamic VPN client group:

set security dynamic-vpn clients dynclient-testing remote-protected-resources
set security dynamic-vpn clients dynclient-testing remote-exceptions
set security dynamic-vpn clients dynclient-testing ipsec-vpn vpn-dynamic
set security dynamic-vpn clients dynclient-testing user-groups dynvpn_testing

Step 2: Add the following line to /usr/share/freeradius/dictionary.juniper:

ATTRIBUTE   Juniper-Local-Group-Name        46  string

Step 3: Assign the user the group through /etc/raddb/users (or however you do it):

testuser    Cleartext-Password := "Testing123"
            Juniper-Local-Group-Name = dynvpn_testing

And… well, test!