Making an offline mirror for Elixir Hex

I work in an offline development environment, which has many challenges as so much software just assumes it will be connected at all times to the Internet.

For software development on an offline network it is very handy to have locally hosted mirrors of various software repositories. Some of these are as easy as just using rsync, while others provide tools to create a mirror.

I recently needed to create an offline mirror of Hex (the Elixir language package repository). There did not appear to be any official way to create a mirror, and the few solutions already out there were dated and no longer seemed to work.

After some poking around and experimentation I realised it wouldn’t be that hard to write something that would do it. The main thing was to get the list of packages. And fortunately there was an API for that.

So I wrote a small Python script that would download the repo.hex.pm repository for offline mirroring. On subsequent runs the script will only download new files, so it is very quick.

As of June 2022 the whole repo is under 9 GB, so it doesn’t take very long to download the entire thing. The script uses parallel download jobs to speed it up.

The script is available on GitHub at https://github.com/WaterJuice/download-hexpm

To download simply run

./download-hex.pm download   

This will download the entire repo to a directory ./repo.hex.pm

The best way to run a mirror on an offline network is to host a webserver serving the files from the download directory as https://repo.hex.pm

In order allow hex to accept the TLS certificate either set environment variable `HEX_CACERTS_PATH` to a PEM file containing your root CA. Or configure hex with `mix hex.config cacerts_path`

Alternatively it can be served on a different address and just use HTTP. In which case set HEX_MIRROR` to point to the server. Or configure hex with `mix hex.config mirror_url`

To test on local computer you can use

python3 -m http.server ./repo.hex.pm 8000

Then set environment variable HEX_MIRROR = http://localhost:8000

Advertisement

Creating a Docker image to build for Linux

The great thing about Docker is that you can put your build environment inside a Docker image and not have to mess around with settings on your computer. Additionally, any computer can build exactly the same just by using the same Docker image. So how do you go about creating a Docker image in the first place?

There is a huge repository of images already built and maintained on hub.docker.com. There is a good chance that there is already an image out there that has the build environment you want. But its still nice to be able to create your own so you know exactly what goes in it.

A Docker image is built by using a Dockerfile. By default the file is simply called Dockerfile. The syntax is simple and basically a list of instructions that are executed in order within a docker container and the result saved as an image. A Dockerfile starts with a FROM instruction which is the name of an existing Docker image that you want to use as the base. There is a built on one called scratch that is simply an entirely empty file system. In general you don’t use this one, this is how the operating system base images are created. Typically you’ll use a standard linux base. Every major distribution now provides Docker images on Docker Hub. The image you use for your base has nothing to do with the host operating system you are running on (Note this is not the case for Windows images, but that is a completely separate situation).

I like to start my images with Ubuntu as that is an easy to use distribution and only 73MB (as of 20.04). A very popular one to use is Alpine, but I don’t particularly like it as I’m not familiar with its package manager. I am going to demonstrate building a Docker image that is capable of building the WjCryptLib for Linux x64.

Save the following as Dockerfile

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y cmake
RUN apt-get install -y ninja-build
RUN apt-get install -y build-essential

ENV CMAKE_GENERATOR=Ninja
ENV CFLAGS=-Wno-deprecated-declarations

The first line says use the existing image ubuntu:20.04 as the starting base. This will be automatically downloaded from hub.docker.com the first time you try to use it.

Then ENV DEBIAN_FRONTEND=noninteractive line sets an environment variable. Annoyingly the Ubuntu images don’t already have this set, and if its not set then using apt-get install on some packages will cause a prompt to come up to ask for timezone information. This prevents the build from jamming.

The next 4 RUN lines are command line instructions run directly in the temporary container created to build the image. The ubuntu images don’t come with the apt-get cache prepulated, so you have to start with an apt-get update

Finally there are two more environment variables which will be set in the image. One sets the CMAKE generator to automatically be Ninja seen we have install that. The second one add a default C compile flag which is currently needed for building WjCryptLib without warnings.

This Dockerfile is all that is needed. Often you have extra files that you want to put into the image. But in this case all we needed to do from a fresh Ubuntu was to do a few apt-get installs.

To build the image run the following command

docker build -t build-linux .

The -t build-linux gives the image a name. In this case I’ve called it build-linux. The '.' at the end says to use the current working directory as the directory to send to Docker as the build context. This will include any files that are in the dir. In this case there aren’t any.

You will see the output from the execution in the output. When its finished there will be a docker image called build-linux in your images list in Docker. If you run docker images you’ll see something like

REPOSITORY                   TAG          IMAGE ID       CREATED          SIZE
build-linux                  latest       5e29d818504f   10 seconds ago   413MB

This image is now ready to use to build WjCryptLib. You can downloadWjCryptLib with the following command (assuming you have git installed)

git clone https://github.com/WaterJuice/WjCryptLib.git

This will download the repo to a directory called WjCryptLib. With in that directory run the following commands to build it.

docker run --rm -v $PWD:$PWD -w $PWD build-linux cmake -H. -Bbuild -DCMAKE_INSTALL_PREFIX=bin
docker run --rm -v $PWD:$PWD -w $PWD build-linux cmake --build build --target install

And that is it!

Now the great thing about Docker is that you don’t need to recreate the Docker image all the time. Once you have made it you can then share it with other people who don’t ever need to worry creating it. They can then use it without any effort. Note you generally would share images by publishing them to Docker hub, in which case the name of the image would have your account name in it.

For example I have the build image wjxx/ubuntu20.04-build which is similar to the one described here. It is also capable of building WjCryptLib and can be used immediately by simply using it instead of the locally build build-linux

docker run --rm -v $PWD:$PWD -w $PWD wjxx/ubuntu20.04-build cmake -H. -Bbuild -DCMAKE_INSTALL_PREFIX=bin
docker run --rm -v $PWD:$PWD -w $PWD wjxx/ubuntu20.04-build cmake --build build --target install

The docker run --rm -v $PWD:$PWD -w $PWD is a very common thing to use. So I have an alias for it called drun so I don’t have to type it in each time.

Now when building I always use --rm which says to create a temporary container and then throw it away when finished. We have no interest in any file changes that occur within the docker container. The files we are interested in are all done in the volume mounted from the host current directory. In fact you can run it with a readonly file system by adding the flag --read-only. Another useful flag to add is --network=none as this build requires no network access you can feel comfortable knowing the container won’t allow it.

The Dockerfile used in this example is simple and can be improved somewhat to make the image a bit smaller. However the following change should only be done once you’ve got your image working otherwise you’ll lose out the benefit of the cache when modifying it.

FROM ubuntu:20.04

ENV \
 DEBIAN_FRONTEND=noninteractive \
 CMAKE_GENERATOR=Ninja \
 CFLAGS=-Wno-deprecated-declarations

RUN set -ex ;\
 apt-get update ;\
 apt-get install -y cmake ;\
 apt-get install -y ninja-build ;\
 apt-get install -y build-essential ;\
 rm -rf /var/lib/apt/lists/*

Each command in a Dockerfile creates a layer which can be useful, and often not wasteful in space. If all you are doing is adding things to the filesystem them no space is wasted by splitting it over layers. If however you delete files in a command you don’t save any space because each layer is a diff from the previous one. In this Dockerfile the line rm -rf /var/lib/apt/lists/* has been added at the end. This deletes the apt-get cache that is downloaded with apt-get update. This can save a bit of space, however it is only useful if it all happens within the one layer. So it is important to have the rm occur in the same RUN command as the apt-get update.

In this case the end image is 384 MB versus the original 413 MB. Whether that is important or not is a matter of opinion.

Docker

Its been a while since I’ve written anything here. The latest thing of interest to me is Docker. This has revolutionised how I build things and I plan to write a few articles detailing how to perform some tasks using it.

What is Docker? A quick summary of Docker is small user mode OS images (typically Linux) containing just the programs required to perform a particular task. The images can easily be built and then shared. Once built an image can be run on any computer that has Docker installed.

How is Docker useful for building software? One of the more annoying things with software development is setting up a build environment. The instructions for doing so can get fiddly and often forgotten when trying to setup another computer. Docker eliminates this problem by keeping the build environment inside static docker images that will operate the same wherever they are run. A development environment need only consist of the OS of your choice (Linux, Windows, or MacOS), Docker installed, your favourite editor/IDE (eg VSCode). And for convenience its a good idea to have python installed so you can script the building.

As an example here is a set of instructions you can run in order to build the WjCryptLib for Linux using any desktop platform.

  1. Install Docker on your OS. (Download)
  2. Download WjCryptLib. (Download)
    Or you can use the command:
    curl -O https://github.com/WaterJuice/WjCryptLib/archive/master.zip
  3. Extract the WjCryptLib zip file and go to its root directory
  4. docker run --rm -v $PWD:$PWD -w $PWD wjxx/ubuntu20.04-build cmake -H. -G Ninja -Bbuild -DCMAKE_INSTALL_PREFIX=bin/linux -DCMAKE_BUILD_TYPE=release
  5. docker run --rm -v $PWD:$PWD -w $PWD wjxx/ubuntu20.04-build cmake --build build --target install

That is all! This will have built the binaries and put them in bin/linux. If you are running on a Linux system you can run it. Otherwise you can run them in another Linux docker container.

docker run --rm -v $PWD:$PWD -w $PWD ubuntu:20.04 bin/linux/WjCryptLibTest 

This should show the output

WjCryptLibTest
------------

Test MD5     - Pass
Test SHA1    - Pass
Test SHA256  - Pass
Test SHA512  - Pass
Test RC4     - Pass
Test AES     - Pass
Test AES CBC - Pass
Test AES CTR - Pass
Test AES OFB - Pass

All tests passed.

You would have noticed that the first time you ran the docker run commands it would download an image from the Internet. Once you have downloaded it, it remains in Docker and can be reused. Running an image that is already downloaded is extremely quick.

So what actually happened?

The first part of the command:
docker run --rm -v $PWD:$PWD -w $PWD wjxx/ubuntu20.04-build
This tells docker to run the image wjxx/ubuntu20.04-build which is an image I built that contains the linux gcc compiler, CMake, and Ninja. I will describe in a later article how this image was made, but for now just know that there are many many images available on hub.docker.com that people have created for all sorts of tasks. This is just one of them.

The --rm means don’t keep the container around after use (as we are not running this as an on going service).

The -v $PWD:$PWD means mount the current working directory into the same location inside the docker container, this allows the docker container to access the source files that are on the host, and to write out the binaries.

The -w $PWD means set the working directory inside the Docker container as the same as the current one in the host.

The rest of the command line that follows is simply the command line to be run inside the container. In this case:
cmake -H. -G Ninja -Bbuild -DCMAKE_INSTALL_PREFIX=bin/linux -DCMAKE_BUILD_TYPE=release
Which says to generate a cmake build using ninja and to set the output to bin/linux and set build type to release mode.

The commands I have listed will work on MacOS and Linux, its slightly different on Windows simply because $PWD does not exist on Windows. On Windows you would use:
-v %CD%:%CD% -w %CD%
Though note this only works if you are working from the C: directory. Otherwise you’ll have to map things slightly differently.

Note: You don’t have to have the directory inside the Docker container the same as the host one, it just makes it easier when getting error messages because the file paths match. But there may be times you don’t want to have the name exist inside Container (for example you may not want the names in your path to show up in the debug binaries, or in the case of Windows you maybe using a drive other than C:). In this case you could use something like
-v $PWD:/tmp -w /tmp
or (for Windows):
-v %CD%:/tmp -w /tmp
In this case we have said to mount the current directory on the host into the /tmp directory of the image.

Okay this was just a quick sample of how Docker can be used for building. This example will work anywhere without anything more than Docker itself being installed. All the settings for the build environment are inside the docker image that you didn’t have to worry about. This makes building in teams and CIs very convenient as everyone can guarantee they are using the same build system settings, but without having to compromise on what platform they develop on.

The other huge advantage of using Docker images is that you can build using all sorts of exotic build environments without having them conflict with each other.

C library for JSON/JSON5 parsing and writing

I have recently been using JSON in various projects and quite like the format. It is very easy to use in Python and is one of the standard libraries that ship with Python. I decided I wanted to use JSON as configuration files in some of my C projects. Previously I have used XML for configuration and have some convenient libraries for marshalling and unmarshalling the data in an XML file to C structures. I wanted to write a library that would let me swap over to using JSON files without adding complexity to those projects.

In particular I want to be able to have a C structure that contains configuration information and be able to save it to JSON and read it back out without much work. In Python this is very easy as a lot of python objects can just be written to JSON with a single function. As C doesn’t have anyway of inspecting its own C structure there needs to be some mapping provided.

So over the past few months I’ve worked on creating “JsonLib“. This is a C library that can marshall and unmarshall data to and from JSON. Originally I was just going to write a JSON parser with an extension to allow comments (because having comments is very convenient in configuration files). While looking up what the most standard extensions were for handling comments I came across JSON5. I decided to make my library JSON5 compliant rather than simply JSON with non official extensions.

By default JsonLib will accept and parse JSON5, however it has a mode which will only accept strict JSON if desired. In general, unless you are writing something to validate JSON, it is better to always accept JSON5 as input. For output the library defaults to strict JSON unless JSON5 is explicity enabled. This provdes greatest compatibility. By default it can parse anything and for output it always makes strict JSON which anything else will be able to read. It should be noted that JSON5 is a strict superset of JSON. So not enabling JSON5 simply means rejecting input that otherwise could unambiguously be parsed.

The library is now available on GitHub

This is free and unencumbered software released into the public domain.

Documentation is rather sparse at the moment, but I wanted to release it anyway. Hopefully there is enough examples in the unit tests to demonstrate its use.

This a link to the current user guide

GUIDs to OIDs

I previously talked about OIDs and how they are used as globally unique identifiers through coordination. GUIDs on the other hand require no coordination, anyone can generate a GUID at any time they need. However there are situations where you don’t get a choice and you must use an OID. Getting an OID can be difficult and time consuming if you don’t already have a tree (although I set up a service to get a free instant one here).

The creators of the OID system came up with a solution for creating an instant OID that did not require coordination. Create a GUID and turn it into an OID. They set up a very nice spot right up high in the tree for it. 2.25 is the root OID for the tree containing GUIDS. A GUID is a 128 bit number and this 128 bit number can be represented as a decimal and then simply appended to 2.25.

For example the GUID {53c08bb6-b2eb-5038-bf28-ad41a08c50ef} can be made into the following OID: 2.25.111325678376819997685911819737516232943

This seems initially like a nice scheme however it has a big problem. Converting a 128 bit number into decimal is not something that is trivial in all computer languages. In Python it is very simple and can be done with in one line:

myOid = '2.25.%u' % int( myGuid.hex, 16 )

However this is not very easy in C as most compilers don’t have native 128 bit math. Additionally some implementations handling OIDs will store them internally using an array of integers, which will also not be able to handle a 128 bit number. Apart from the 2.25 branch, pretty much no other OID subcomponent will be a enormous number. So you can easily get away with just using 32 bit numbers for each oid part and you’ll be able to handle almost any OID presented. Except of course the enormous 2.25 OID.

Microsoft came up with a solution to this problem and reserved a branch in their OID space of 1.2.840.113556.1.8000.2554. Appended onto this is the GUID but broken down into several smaller sub parts. Again with my example GUID of {53c08bb6-b2eb-5038-bf28-ad41a08c50ef} the MS OID is: 1.2.840.113556.1.8000.2554.21440.35766.45803.20536.48936.11354528.9195759

MS provided a VBScript to convert a GUID into an OID if this type. Each component of the OID first within 32bits so is easy to handle in any language.

In Python you can create the MS OID with the following code

oidParts = [None] * 7
oidParts[0] = str( int( myGuid.hex[0:4], 16 ) )
oidParts[1] = str( int( myGuid.hex[4:8], 16 ) )
oidParts[2] = str( int( myGuid.hex[8:12], 16 ) )
oidParts[3] = str( int( myGuid.hex[12:16], 16 ) )
oidParts[4] = str( int( myGuid.hex[16:20], 16 ) )
oidParts[5] = str( int( myGuid.hex[20:26], 16 ) )
oidParts[6] = str( int( myGuid.hex[26:32], 16 ) )
myOid = '1.2.840.113556.1.8000.2554.%s' % '.'.join(oidParts)

The MS solution has one big problem as well. Because they gave it such a large prefix the entire OID string is pretty long. And some implementations handling OIDs are known to have a 64 character limit. This is not part of the standard, but an implementation limit. However you won’t be able to use the OID in these systems.

What would have been good is if the prefix could have been a tiny one like the 2.25 one. However only the ISO OID committee can allocate that. However I decided to do what I could to help the situation.

I have reserved the following OID for the use of converting a GUID to an OID: 1.3.6.1.4.1.54392.1

My scheme is simpler than the MS one. The GUID is broken into two 64 bit numbers and then appended. So again my example GUID {53c08bb6-b2eb-5038-bf28-ad41a08c50ef} becomes the OID: 1.3.6.1.4.1.54392.1.6034977117478539320.13774449957690691823

This OID is less than 64 characters long and it also can be created easily with any language that can handle 64 bit numbers.

Python code for this OID is:

oidParts = [None] * 2
oidParts[0] = str( int( myGuid.hex[0:16], 16 ) )
oidParts[1] = str( int( myGuid.hex[16:32], 16 ) )
myOid = '1.3.6.1.4.1.54392.1.%s' % '.'.join(oidParts)

This OID might still have the problem with systems that store the OID using 32 bit numbers for the parts. In which case I have also reserved the OID 1.3.6.1.4.1.54392.2 for breaking the GUID into 4 32 bit numbers. So again my example GUID {53c08bb6-b2eb-5038-bf28-ad41a08c50ef} becomes the OID: 1.3.6.1.4.1.54392.2.1405127606.3001765944.3207114049.2693550319 This is also under 64 characters in length and each sub part can be stored in 32 bit.

Python code for making the 32 bit compatible OID:

oidParts = [None] * 4
oidParts[0] = str( int( myGuid.hex[0:8], 16 ) )
oidParts[1] = str( int( myGuid.hex[8:16], 16 ) )
oidParts[2] = str( int( myGuid.hex[16:24], 16 ) )
oidParts[3] = str( int( myGuid.hex[24:32], 16 ) )
myOid = '1.3.6.1.4.1.54392.2.%s' % '.'.join(oidParts)

Finally for completeness I decided to reserve the OID 1.3.6.1.4.1.54392.3 for representing a GUID using 8 16 bit parts. This time my example GUID {53c08bb6-b2eb-5038-bf28-ad41a08c50ef} becomes the OID: 1.3.6.1.4.1.54392.3.21440.35766.45803.20536.48936.44353.41100.20719

Note however this final form is also over 64 characters in length so offers no advantage over the Microsoft version.

I have created a script that creates all 5 forms of OIDs from a GUID.

This script is public domain and you are free to use it how you wish. I’m hopeful that this will mean people can create OIDs from GUIDs that are fully useable in any implementation.

 

Quantum GUIDs!

I recently discovered the following web site operated by the Australian National University (ANU): https://qrng.anu.edu.au

This is a random number generator using quantum physics. This should produce the purest of random numbers. From their web site you can get random numbers in various formats and they also provide a simple API that returns the values in JSON format.

I wrote a python script to collect random values from the site and produce Type 4 GUIDs from it. These GUIDs will have 122 bits of pure random. Quantum random!

The python file will generate between 1 and 1000 GUIDs specified on the command line.

python3 QuantumGuid.py 10
{9b747ca1-f809-44e6-af35-7041b790ddf4}
{6b5c5703-cb16-41c9-9101-c889fb31a93e}
{3f4f6eb7-0ca1-4ffb-a685-1b1afc824235}
{36e75e7c-3317-4a10-b781-d4376decf937}
{e2e85ad6-8a6d-4de1-b145-455c2bead457}
{5a0784aa-a302-40e9-87f3-d151abec44f5}
{7596d675-2e8c-4dbd-8e5e-ca9282222421}
{cdc57458-de8f-49af-811a-29c80ae76107}
{165565bb-1038-4381-b6ba-b64c3e64fe98}
{77d631dc-ef63-465a-acb7-b22b4e8e92c4}

These GUIDs feel so much more random than a regular type 4 version that I really feel they should be in their own type space so as to not be contaminated with regular non quantum GUIDs ;-). Perhaps they could be type 8 GUIDs? (type 6 is already adhoc used, and I already have an idea for type 7!)

If you want to get your own, then download the script

This is free and unencumbered software released into the public domain.

Personal GUIDs – Fixed

Six years ago I wrote the article Personal GUIDs. This introduced a technique for assigning a unique unchanging type 5 GUID to each person on the planet. It made a string of the format:

"Surname;GivenNames;Sex;CountryOfBirth;PlaceOfBirth;DateOfBirth,"

which in theory should be unique for everyone, and turned it into a type 5 GUID using the Namespace GUID:

 {5b390b3f-9a62-508a-b235-6e6e8d270720}

(Read https://waterjuiceweb.wordpress.com/2013/06/16/type-3-and-5-guids/ for a description on how a type 5 GUID is made). I wrote a C program to generate the GUIDs.

Unfortunately, I made a mistake!

The C code did not use the DateOfBirth value in creating the string to be hashed. This meant only the first 5 fields were used. As a result the GUIDs produced did not match the description. Also it would mean two people born in the same place with same name but at different times could generate the same GUID.

I only just noticed the error while I was including the code into a new library I’m producing (Stay tuned for WjGuid coming soon!) and I was testing the results against an online type 5 GUID generator.

I have fixed the code and released PersonalGuid 1.0.1. The latest version is available here:

 >PersonalGuid.exe Doe John m Australia Sydney 19700101
ID: DOE;JOHN;M;AUSTRALIA;SYDNEY;19700101
{897469e0-c10b-5829-bd15-86309b1523cf}

To manually produce the personal guid without this tool you can use a site such as https://www.toolswow.com/generator/uuid-v5

For NameSpace mode use UUID and use the value ” {{5b390b3f-9a62-508a-b235-6e6e8d270720}”.

ScreenShot.png
Screenshot from https://www.toolswow.com/generator/uuid-v5

I additionally made a Python version that will work in Python 2 or 3. The python file is inside the .zip, but I have included it here as well.

 

 

 

Making a basic web site look better

TL;DR: Here is a repo https://github.com/WaterJuice/BottleMaterializeTemplate which contains all you need to use Materialize and Bottle in an self contained host using only Python.

Recently I wrote about OIDs and made a basic website that would issue out a new OID to anyone who wanted one without any registration required. (Here)

I don’t pretend to be a web designer but I think it is safe to say the original site was pretty ugly:

Screen Shot 2019-08-23 at 6.08.10 pm.png

I wanted something simple to “make it look good” without requiring me to go to any particular great effort or learn a bunch of CSS. Materialize was recommended to me, so I had a look. Using one of the samples provided I was quickly able to make my site look considerably better. The end result:

Screen Shot 2019-08-24 at 2.51.30 pm.png

I think this was considerably better. Additionally it works well on mobile devices.

While making the site I decided to make a template that I could easily use to start any future projects. In particular I wanted to make have all the files local so it could be self hosted without Internet access (for example on an internal intranet). I used bottle which is a web server framework written in python. This can be installed with pip, or simply copy the single file (bottle.py). Materialize just required two files (materialize.min.css and materialize.min.js), plus the additional Material fonts which are normally hosted on google fonts. If you want to host them yourself you just need a few file MaterialIcons-Regular.ttf, MaterialIcons-Regular.woff, and MaterialIcons-Regular.woff2. Plus an additional css file that maps them.

I put all the necessary files together and made a very simple starting example called “bottle_app.py”

All of the files are in a github repo: https://github.com/WaterJuice/BottleMaterializeTemplate

All that is required is python (2 or 3). Simply run

python bottle_app.py

This will start a simple web server running on port 8080. Then browse to http://localhost:8080 (or ip address of your computer) to see the sample page. This is what it looks like on an iPhone.

IMG_8601

The template also contains error pages for common HTTP errors. Eg try browsing to an invalid page and you’ll get a 404 error.

Perhaps this will be useful to you.

Materialize and Bottle are both released using the MIT License, so I have retained the same for my template.

Get the template at https://github.com/WaterJuice/BottleMaterializeTemplate

 

 

 

Oh my OID!

TL;DR: Get a free OID online instantly at https://freeoid.pythonanywhere.com

I have long been interested in GUIDs (UUIDs) which provide a mechanism for unique IDs across multiple domains without any central authority required. The version 1 GUID partitioned a huge 128 bit number space up into unique computer, and also time domains. This meant GUIDs could be generated anywhere anytime at  extremely fast rates without any worry of collision. The downside to version 1 GUIDs are that the computer network address (MAC address) is generally encoded into GUID which leads to privacy issues. Version 4 GUIDs take the approach that 128 bits (122 actually) is so huge that if everyone just picks random numbers the chance of collision is close to 0 anyway. Version 4 GUIDs are now the most common form and they work great as unique IDs. It is not, however, the only approach provide globally unique IDs. I recently stumbled across a different scheme called the OID.

I had been revisiting information on version 3 and 5 GUIDs and looking to see what pre-made namespaces were provided. Disappointingly there are only 4 default namespaces: NameSpace_DNS, NameSpace_URL,NameSpace_OID, and NameSpace_X500.I was curious as to what the NameSpace_OID was for and that led me to discover the Object Identifier (OID).

What is an Object Identifier (OID)

The OID is a globally unique identifier that is guaranteed to be unique by using a tree structure where each part of the tree is responsible for assigning the elements directly under it. This provides a controlled, yet distributed mechanism for assigning OIDs so that they will never collide.

The most common way to display an OID is in “dotted decimal” form. Such as:

1.3.6.1.4.1.37476.9000.79 

This OID is registered to this website. I can subdelegate this anyway I wish by appending a futher dot and number. I am responsible for assigning any futher oids from this one. I can also give one of my sub OIDs or range of sub OIDs to someone else to manage.

So how did this ID get formed?

The first level is controlled by ITU-T and ISO organisations. They have assigned only 3 values. 0 for ITU-T, 1 for ISO, and 2 for joint ITU-T and ISO things. The number follows a path down a tree, which in the case of our example is

  • 1 – ISO
  • 1.3 – Identified organization
  • 1.3.6 – DoD (Department of Defence)
  • 1.3.6.1 – Internet
  • 1.3.6.1.4 – Private
  • 1.3.6.1.4.1 – IANA enterprise numbers
  • 1.3.6.1.4.1.37476 – ViaThinkSoft (IANA number 37476)
  • 1.3.6.1.4.1.37476.9000 – FreeOid
  • 1.3.6.1.4.1.37476.9000.79 – WaterJuice

Because any OIDs I create will be appended on as 1.3.6.1.4.1.37476.9000.79.* there is no chance that someone else will create the same IDs as they will be starting theirs from somewhere else in the tree.

The majority of all OIDs are hanging off the 1.3.6.1.4.1 branch. Anyone can (with a bit of effort) get an IANA “Enterprise Number” for free. You have to register for it and provide details and it takes about a week. Once you have your number you can assign sub OIDs however you want. Obviously no one is supposed to just arbitrarily use someone else’s branch and assume they can just add some branches that they don’t think are being used. The whole scheme breaks down if people do that.

There are other OID branches further up the tree but there are much harder to get attached to one of those. They tend to be used for ISO things, or reserved for countries etc. There are some places that will charge you a fee to get an OID. So generally most organisations get themselves a free IANA number. It is disappointing that the IANA numbers can’t start much higher up, ideally even at the top level. It could have been 3.* for the IANA enterprise numbers. However the biggest problem with the OIDs are how precious people who control the early ones tend to be. ViaThinkSoft provide free ones attached to a sub OID from their IANA number. You just need an email address to get one. One will be assigned and sent in a reasonably short amount of time.

There are also two methods to get an OID instantly by converting a GUID into an OID. This requires no registration or server interaction and can be done at anytime by anyone. However neither of the techniques are particularly satisfactory (I’ll discuss why in a later article) and you have a rather ugly looking OID such as

2.25.324293003071620890871350535745399440863

or

1.2.840.113556.1.8000.2554.62456.36735.23508.16633.39838.4613307.1590751

(These incidentally both represent the GUID {f3f88f7f-5bd4-40f9-9b9e-4664bb1845df} )

These might be okay for automatically generated OIDs that stand alone, but they don’t make a particularly good branch to add a whole sub tree to. I could sub-delegate out that tree and give, for example, 1.2.840.113556.1.8000.2554.62456.36735.23508.16633.39838.4613307.1590751.2 to someone to start their own tree, but its not a great starting point, they would generally prefer to be much higher up the tree.

So the methods available to get an OID seemed to be either generate an obnoxiously large one of your own instantly from a GUID, or wait around for someone to give you a nicer one from their tree. The IANA branch is the best one you can realistically get, but it can be an annoying process getting one from them (and you will start getting spam immediately after you get one as they publish your email address!).

Given that OIDs are just numbers and not precious gems, it seems ridiculous that its not possible just to easily get one without any hassle. The only point of the scheme is to avoid collisions, the whole registering for one and providing identification etc is unnecessary. So I decided I would provide a free service to give a decent OID to anyone and everyone who wants one!

Get a free OID

https://freeoid.pythonanywhere.com

This is a very simple website I setup to issue unique OIDs at the press of a button. There is no registration, email address, or anything required. Simply press the button and a new OID will be generated. I was originally going to attach it to a branch off my 1.3.6.1.4.1.37476.9000.79 OID, but I felt that it would be too far down the tree and there is no reason why anyone shouldn’t be able to easily have a better one. So I registered an IANA number for the purpose and have assigned the following OID to be the branch for these free OIDs

1.3.6.1.4.1.54392.5

If you go to the site you can get an instant OID that will be a sub OID from this one. You might be wondering what the .5 part is for. I have already assigned the earlier ones for a different use that I will detail in a future article).

There are no limits on generating OIDs with this site. It will simply increment an internal counter and give you the next one available. There is no practical limit as to how large the number can grow. I don’t imagine the service will be particularly busy so the numbers aren’t likely to get to enormous.

So feel free to get yourself your very own OID for free right now, or get several if you want. I don’t mind how many you want or what you want to do with them!

 

AES-CBC

The previous two cipher modes of AES I wrote into WjCryptLib were AES-CTR and AES-OFB. Both of these turn AES into stream ciphers. In both cases only the AES block encrypt function is used. So today I add AES-CBC (Cipher Block Chaining) mode to the library. I don’t particularly like CBC as a mode personally, however it is one of the most common modes used so I wanted to include it in the library.

Cipher Block Chaining mode works by XORing the previous cipher block onto the plaintext before performing the block encrypt. An IV is used as the first “previous cipher block”. A change in a byte of plaintext will cause all the following cipher text to be different. A disadvantage of the mode is that it has to work with whole number of blocks (16 bytes in the case of AES). This limitation is usually overcome by padding the last block and keeping a count value of the actual data. There is also a fancier technique called cipher text stealing which reduces the limitation to only requiring a minimum of a a single block. I have not included this technique I my implementation.

CBC is not a stream cipher mode, as in it does not generate a parallel stream of bytes that are then applied (usually with XOR) onto the input stream. CBC uses the block encrypt and decrypt block functions on the input data.

I have released WjCryptLib 2.3.0 which contains AES-CBC.

The relevant source files needed are:

This is free and unencumbered software released into the public domain.