rbx-bufferize

A tool to losslessly encode / decode roblox data types to buffers.

local mail = {
	email = "john.doe@email.com",
	street = "321 Road City Country",
	unit = 123,
}

local tbl = {
	name = "John Doe",
	age = 603,
	contact = mail,
	mail = mail,
}

local b: buffer = Bufferize.encode("Hello world!", 123, true, tbl)
print(Bufferize.decode(b)) -- "Hello world!", 123, true, tbl

Instances​

Since instances can be converted to roblox data types it's possible for Bufferize to encode and decode them. To help with this Bufferize contains two functions to help with this process.

local b = Bufferize.encode(Bufferize.serializeInstance(workspace.Baseplate))
local baseplateCopy = Bufferize.deserializeInstance(Bufferize.decode(b))

Sometimes properties are references to other instances (i.e. ObjectValue.Value). In order for a reference to be maintained it must be pointing to an instance that is also in the instance tree being serialized.

local objV = Instance.new("ObjectValue")
local folder = Instance.new("Folder")
folder.Parent = objV

objV.Value = folder
local b = Bufferize.encode(Bufferize.serializeInstance(folder))
local objVCopy = Bufferize.deserializeInstance(Bufferize.decode(b))
-- valid: objVCopy.Value == folderCopy

objV.Value = workspace.Terrain
local b = Bufferize.encode(Bufferize.serializeInstance(folder))
local objVCopy = Bufferize.deserializeInstance(Bufferize.decode(b))
-- not valid: objVCopy.Value == nil

Custom Encoding​

Bufferize attempts to store all data types losslessly by default. This is helpful if precision is important for you, but depending on your project it may not be needed. In order to remain flexible in this regard Bufferize supports the ability to define custom encodings for any specific data type (except tables).

local inHouseEncoder = Bufferize.custom()

-- CFrame override that stores rotation euler angles XYZ rounded to nearest degree
-- this is not lossless, but depending on your use case it may be good enough and it results
-- in a smaller buffer size
inHouseEncoder:override("CFrame", {
	read = function(b: buffer)
		local x = buffer.readf32(b, 0)
		local y = buffer.readf32(b, 4)
		local z = buffer.readf32(b, 8)
		local rx = math.rad(buffer.readi16(b, 12))
		local ry = math.rad(buffer.readi16(b, 14))
		local rz = math.rad(buffer.readi16(b, 16))
		return CFrame.new(x, y, z) * CFrame.fromEulerAngles(rx, ry, rz, Enum.RotationOrder.XYZ)
	end,
	write = function(cf: CFrame)
		local b = buffer.create(18)
		local rx, ry, rz = cf:ToEulerAngles(Enum.RotationOrder.XYZ)
		buffer.writef32(b, 0, cf.X)
		buffer.writef32(b, 4, cf.Y)
		buffer.writef32(b, 8, cf.Z)
		buffer.writei16(b, 12, math.round(math.deg(rx)))
		buffer.writei16(b, 14, math.round(math.deg(ry)))
		buffer.writei16(b, 16, math.round(math.deg(rz)))
		return b
	end,
})


local complexRotation = CFrame.new(0, 0, 0, 1, 2, 3, 4)
local lengthA = buffer.len(Bufferize.encode(complexRotation))
local lengthB = buffer.len(inHouseEncoder:encode(complexRotation))
print(lengthA > lengthB) -- true

Raw Read / Write​

The encoding for each data type is also exposed directly via Bufferize.read and Bufferize.write. These skip the version header and type ids, so they're useful when you already know the layout of your data and want to pack values as tightly as possible.

Readers take a buffer and offset and return the value along with the offset immediately after it. Writers take a buffer, offset, and value and return the buffer along with the offset immediately after the written value. If the value doesn't fit, a larger copy of the buffer is returned, so always use the returned buffer for subsequent writes.

local b, offset = Bufferize.write.CFrame(buffer.create(0), 0, CFrame.new(1, 2, 3))
b, offset = Bufferize.write.Vector3(b, offset, Vector3.new(4, 5, 6))

local cf, nextOffset = Bufferize.read.CFrame(b, 0)
local v3 = Bufferize.read.Vector3(b, nextOffset)

Raw readers and writers always use the default encoding and ignore any overrides registered on a custom encoder. Since no version header is written, buffers produced this way are not protected by the versioning checks described below.

Versioning​

Bufferize strictly adheres to semantic versioning.

When a buffer is encoded the current version of bufferize is included. That way when a buffer is decoded it's possible to ensure we're not attempting to decode with an incompatible version of bufferize.

For example, say you used bufferize v1.0.0 to store data in a datastore and then v2.0.0 is released. If you tried to read the v1.0.0 data with v2.0.0 bufferize then you'd get an error.

If you don't need these guarantees (e.g. for short-lived data sent over the network where both ends are always running the same version) you can use Bufferize.encodeHeaderless and Bufferize.decodeHeaderless to skip writing the version header and save a few bytes. No version checks are performed when decoding these buffers. Custom encoders have the matching encoder:encodeHeaderless and encoder:decodeHeaderless methods.

This means in practice that when a major version of bufferize releases old data will not be compatible. When the minor or patch version changes you can decode your old data with the new version of bufferize, but you can't decode your new data with the old version.

If you'd rather branch on compatibility than handle an error you can check a buffer before decoding it. Bufferize.isCompatible runs the same version check that decode does, so a false result means decode would throw.

if Bufferize.isCompatible(b) then
	local value = Bufferize.decode(b)
else
	print(`cannot decode a buffer from version {Bufferize.readVersion(b)} with version {Bufferize.VERSION}`)
end

Only the header is read, so a true result doesn't promise the rest of the buffer is well-formed. Buffers that weren't produced by encode at all return false.

Supported DataTypes​

✔ Implemented | ❌ Unimplemented | ⛔ Never