Translator config section

A documentation page for the TRANSLATOR section of config manifest.

An example configuration piece:

TRANSLATOR: 
  ConstCharIsString: true
  ConstUCharIsString: false
  ConstRules: 
    defines: eval
  PtrTips:
    function:
      - {target: "vorbis_synthesis_pcmout$", tips: [ref,arr]}
      - {target: ^vorbis_, tips: [ref,ref,ref]}
      - {target: ^ogg_, self: arr, tips: [ref,ref]}
  Rules: 
    global: 
      - {transform: lower}
      - {action: accept, from: "^vorbis_"}
      - {action: accept, from: "^ogg_"}
      - {action: replace, from: "^vorbis_", to: _}
      - {transform: export}
    const:
      - {action: accept, from: "^OV_"}
      - {action: replace, from: "^ov_", to: _}
    type: 
      - {action: replace, from: "_t$"}
    private:
      - {transform: unexport}
    post-global: 
      - {action: doc, from: "^ogg_u?int[0-9]+_t"} # types like ogg_uint32_t
      - {action: doc, from: "^ogg_", to: "https://xiph.org/ogg/doc/libogg/$name.html"}
      - {action: doc, from: "^vorbis_", to: "https://xiph.org/vorbis/doc/libvorbis/$name.html"}
      - {action: replace, from: _$}
      - {load: snakecase}

ConstCharIsString / ConstUCharIsString

These boolean flags have been added recently to explicitly turn off const char * (also const unsigned char *) as string in Go, thus asserting its immutability. The problem of such translation was in treating \x00 within the data passed to Go land. With string being asserted, data could be trimmed too early. So for some projects ConstCharIsString and ConstUCharIsString allow to disable that heuristic rule and treat const char * as char * i.e. []byte.

ConstCharIsString is set to true by default, ConstUCharIsString is set to false by default.

ConstRules

Specifies the rules of constant value unfolding. There are two const scopes: enum and defines, so they are used to match the origin of value, whether it has been defined in #define or has been declared as enumeration.

N.B. As an example of using c-for-go just to generate constant definitions check the -nocgo executable flag and the spv SPIR-V Go package.

There are three unfold rules: eval, cgo and expand. By default eval is being used by translator.

Example:

  ConstRules: 
    defines: expand
    enum: cgo

Consider this C snippet:

#define VK_LOD_CLAMP_NONE                 1000.0f
#define VK_REMAINING_MIP_LEVELS           (~0U)
#define VK_WHOLE_SIZE                     (~0ULL)

typedef enum VkResult {
    VK_SUCCESS = 0,
    VK_NOT_READY = 1,
    VK_TIMEOUT = 2,
}

#define paNoDevice       ((PaDeviceIndex)-1)
#define paFloat32        ((PaSampleFormat) 0x00000001) /**< @see PaSampleFormat */
#define paInt16          ((PaSampleFormat) 0x00000008) /**< @see PaSampleFormat */

eval constants

Eval unfold method simply substitutes the values of constants as evaluated by cznic/cc. If the value cannot be evaluated, then its value being expanded (see expand constants below). The latter might introduce compilation errors, so either skip these constants or use cgo constant aliases (see below).

const (
	// LodClampNone as defined in vulkan/vulkan.h:95
	LodClampNone = 1000
	// RemainingMipLevels as defined in vulkan/vulkan.h:96
	RemainingMipLevels = 4294967295
	// WholeSize as defined in vulkan/vulkan.h:98
	WholeSize = 18446744073709551615
)

// Result enumeration from https://www.khronos.org/registry/vulkan/specs/1.0/man/html/VkResult.html
const (
	Success                   Result = iota
	NotReady                  Result = 1
	Timeout                   Result = 2
)

const (
	// PaNoDevice as defined in portaudio/portaudio.h:169
	PaNoDevice = -1
	// PaFloat32 as defined in portaudio/portaudio.h:436
	PaFloat32 = ((SampleFormat)(0x00000001))
	// PaInt16 as defined in portaudio/portaudio.h:439
	PaInt16 = ((SampleFormat)(0x00000008))
)

All constants except PaFloat32 and PaInt16 were set to their values, and hopefully for us a type casting expression has a typed constant value in Go (in C they are no more than text substitutions). Note that RemainingMipLevels and WholeSize has no sense in this context now.

expand constants

Expanding is an unfold method when an ad-hoc state-machine implemented in a few hours can be useful for transferring constant expressions from C to Go with minimal modifications. Remember PaFloat32 from the last section? Here we are pursuing this idea because we like that behaviour.

const (
	// LodClampNone as defined in vulkan/vulkan.h:95
	LodClampNone = 1000.0
	// RemainingMipLevels as defined in vulkan/vulkan.h:96
	RemainingMipLevels = (^uint32(0))
	// WholeSize as defined in vulkan/vulkan.h:98
	WholeSize = (^uint64(0))
)

// Result enumeration from https://www.khronos.org/registry/vulkan/specs/1.0/man/html/VkResult.html
const (
	Success                   Result = iota
	NotReady                  Result = 1
	Timeout                   Result = 2
)

const (
	// PaNoDevice as defined in portaudio/portaudio.h:169
	PaNoDevice = ((DeviceIndex)(-1))
	// PaFloat32 as defined in portaudio/portaudio.h:436
	PaFloat32 = ((SampleFormat)(0x00000001))
	// PaInt16 as defined in portaudio/portaudio.h:439
	PaInt16 = ((SampleFormat)(0x00000008))
)

This implementation has a very naive backend so could not always deliver, but when it does, results are looking smooth. Note that RemainingMipLevels and WholeSize have an obvious reason being here and LodClampNone is expected to be a floating point value.

cgo alias

The most simple unfold that writes an alias to the original const, this implies that you'll need to keep CGo and headers in place to compile that code, even there is nothing except the constants (the spirv case). If the target is a define then it will be evaluated or expanded (whenever the evaluation fails).

// Result enumeration from https://www.khronos.org/registry/vulkan/specs/1.0/man/html/VkResult.html
const (
	Success                   Result = C.VK_SUCCESS
	NotReady                  Result = C.VK_NOT_READY
	Timeout                   Result = C.VK_TIMEOUT
)

Rules

Rules are the translator's main engine, this part makes all the difference between c-for-go and SWIG. In general, this section defines a set of rules for each target scope, the scopes correspond to processing stages when rules are being taken into account, the names of entities being processed are matched agains regular expressions and the specified actions are being executed for that entity. Now let's talk more specific.

There are several rule scope groups: (global, post-global), (const, type, function), (public, private).

• global sets actions and accept/reject rules that take precedence before specific type scopes;
• post-global sets rules that are considered at last;
• public and private sets rules that being applied depending on the visibility of entity, e.g. function args are considered private since they cannot be referenced;
• const, type and function target the specific groups of entities.

Every rule conforms this specification:

type RuleSpec struct {
	From, To  string
	Action    RuleAction
	Transform RuleTransform
	Load      string
}

There a several kinds of rules that have different subsets of fields used.

Accept/Reject

These are simple rules whose from field specifies a regular expression the original C name should match and an action to take when one does match. Every name is checked against each expression until it's clear whether we should accept or ignore the name. By default everything is ignored.

Example:

  Rules: 
    global: 
      - {action: accept, from: "^vorbis_"}
      - {action: accept, from: "^ogg_"}

Accepts any name staring with either vorbis_ or ogg_ prefixes.

  Rules: 
    const:
      - {action: accept, from: "^OV_"}
      - {action: accept, from: "(?i)^(vpx|vp8|vp9)"}

Accepts constants whose name starts with OV_ prefix. Note that only the original names are matched against patterns, all transformations from the replace/transform pipeline are not affecting this. Accept names that start from one of three prefixes in any case - vpx, vp8 and vp9.

Replace

Rules with action replace are forming the replace/transform pipeline. These rules are being applied in order of occurrence and are capable of substring transformation. Having from filled but to empty means making a cut out.

Example:

  Rules:
    global:
      - {action: replace, from: "^vorbis_", to: _}
      - {action: replace, from: "^ov_", to: _}
      - {action: replace, from: "_t$"}
    function:
      - {action: replace, from: PFN_vkDebugReportCallback, to: DebugReportCallbackFunc}

Here we cut prefixes but leave the fact that there was a prefix, and cut _t suffix entirely. Rename a function.

Transform

In order to keep rules as simple as possible, there is a set of predefined transforms that are commonly used in name normalisation.

Before	Transform	After
helloWorld	lower	helloworld
hello_world	title	Hello_world
helloWorld	export	HelloWorld
HelloWorld	unexport	helloWorld
hello	upper	HELLO

Example transform pipeline:

- {transform: title, from: "_partitions"}
- {transform: title, from: "_resilient"}
- {transform: title, from: "_error"}
- {transform: export}

Applied to vpx_error_resilient_partitions results in

• vpx_error_resilientPartitions
• vpx_errorResilientPartitions
• vpxErrorResilientPartitions
• VpxErrorResilientPartitions

The most popular transforms are those that are global or scope-global, e.g. have no from filter and being applied to the whole range of entities. Consider this example:

  Rules: 
    global: 
      - {transform: lower}
      # other transforms skipped
      - {transform: export}
    private:
      - {transform: unexport}

Documentation

The special action doc allows to define the documentation source. By default that is the position in the originating C header file, but this may be overridden. The from filter allows to narrow the scope of the target entities and to specifier defines a template. Supported template variables: path, file, line, name, goname. The latter two represent C entity name and Go entity name (after transforms). The default template is $path:$line.

Example:

  Rules: 
    post-global: 
      - {action: doc, from: "^ogg_u?int[0-9]+_t"} # types like ogg_uint32_t
      - {action: doc, from: "^ogg_", to: "https://xiph.org/ogg/doc/libogg/$name.html"}
      - {action: doc, from: "^vorbis_", to: "https://xiph.org/vorbis/doc/libvorbis/$name.html"}

Here we ignore some types that have no docs, any other entity with either ogg_ or vorbis_ prefix is known to have an official specification that can be located using the C entity name. Check out this beauty:

// OggSyncState as declared in https://xiph.org/ogg/doc/libogg/ogg_sync_state.html
type OggSyncState struct {
	// omitted
}

// Block as declared in https://xiph.org/vorbis/doc/libvorbis/vorbis_block.html
type Block struct {
	// omitted
}

// OggStreamPacketin function as declared in https://xiph.org/ogg/doc/libogg/ogg_stream_packetin.html
func OggStreamPacketin(os *OggStreamState, op *OggPacket) int32 {
	// omitted
}

Built-in presets

There is a special field in the rule spec that allows to load a predefined rule specification, at this moment there are several predefined presets and I have plans to add more after receiving the feedback. That's definitely a good idea to add a preset list based on golint substitutions (ID instead of Id and so on).

var builtinRules = map[string]RuleSpec{
	"snakecase":  RuleSpec{Action: ActionReplace, From: "_([^_]+)", To: "$1", Transform: TransformTitle},
	"doc.file":   RuleSpec{Action: ActionDocument, To: "$path:$line"},
	"doc.google": RuleSpec{Action: ActionDocument, To: "https://google.com/search?q=$file+$name"},
}

Note the snakecase preset that participates in every c-for-go manifest since then. The very common case:

  Rules: 
    post-global: 
      - {action: replace, from: _$}
      - {load: snakecase}

Makes _stuff_like_that shine as StuffLikeThis. Also using doc.google preset is a good idea when the official docs are poorly organised and there is no comments in the original C header code either.

The real world example

  Rules: 
    global:
      - {action: accept, from: ^A}
      - {action: ignore, from: ^ABS}
      - {action: accept, from: ^android_Log}
      - {action: replace, from: ^android_Log, to: Log}
      - {action: replace, from: "(?i)^Android"}
      - {action: replace, from: ^A}
    function:
      - {action: accept, from: ^__android_log_write}
      - {action: replace, from: ^__android}
      - {action: ignore, from: JNI_OnLoad}
      - {action: ignore, from: JNI_OnUnload}
      - {action: ignore, from: ANativeActivity_onCreate}
      - {action: ignore, from: ASensorManager_getSensorList}
    type:
      - {action: accept, from: ^J}
      - {action: accept, from: jobject$}
      - {action: replace, from: "_t$"}
    const:
      - {action: ignore, from: TTS_H$}
      - {transform: lower}
    private:
      - {transform: unexport}
    post-global: 
      - {transform: export}
      - {load: snakecase}

PtrTips

Specifies the translation tips for pointer types. There are three possible tips: ref, sref and arr. If your type in C looks like int *, then using ref tip against it will leave *int32 in Go, using arr will yield []int32 and will use special conversion options (unpack/pack) to pass that through in both directions. The sref option is a special one for multiple pointer cases in C, when you are expected to provide a pointer to a pointer, etc.

Example 1:

C	Hint	Go
Object**	ref	[]*Object
Object**	arr	[][]Object
Object**	sref	**Object

Example 2:

C	Hint	Go
Object***	ref	[][]*Object
Object***	arr	[][][]Object
Object***	sref	***Object

Supported scopes: function, struct, any. A pointer tip can be targeted to function args, to function return value, to struct fields, to any name matching pattern.

Tip specification

type TipSpec struct {
	Target  string
	Tips    Tips
	Self    Tip
	Default Tip
}

Where target is any supported regular expression. Self tip targets the function return value. Default tip for missing cases, however I advise to fill all tips, default ones mark as 0 or size. The latter does nothing because it's not a valid tip, but it helps to maintain these tips and avoiding human errors.

The real world example:

  PtrTips:
    struct:
      - {target: VkGraphicsPipelineCreateInfo, tips: [0,0,0,0,arr,ref,ref,ref,ref,ref,ref,ref,ref,ref]}
      - {target: VkInstanceCreateInfo, tips: [0,0,0,ref]}
    function:
      - {target: ^callVkEnumeratePhysicalDevices$, tips: [0,ref,arr]}
      - {target: ^callVkGetPhysicalDeviceQueueFamilyProperties$, tips: [0,ref,arr]}
      - {target: ^callVkEnumerateInstanceExtensionProperties$, tips: [0,ref,arr]}
      - {target: ^callVkEnumerateDeviceExtensionProperties$, tips: [0,0,ref,arr]}
      - {target: ^callVkEnumerateInstanceLayerProperties$, tips: [ref,arr]}
      - {target: ^callVkEnumerateDeviceLayerProperties$, tips: [0,ref,arr]}
      - {target: ^callVkQueueSubmit$, tips: [0,size,arr]}
      - {target: ^callVkFlushMappedMemoryRanges$, tips: [0,size,arr]}
      - {target: ^callVkInvalidateMappedMemoryRanges$, tips: [0,size,arr]}
      - {target: ^callVkGetImageSparseMemoryRequirements$, tips: [0,0,size,arr]}
      - {target: ^callVkGetPhysicalDeviceSparseImageFormatProperties$, tips: [0,0,0,0,0,0,size,arr]}
      - {target: ^callVkQueueBindSparse$, tips: [0,size,arr]}
      - {target: ^callVkResetFences$, tips: [0,size,arr]}
      - {target: ^callVkWaitForFences$, tips: [0,size,arr]}
      - {target: ^callVkMergePipelineCaches$, tips: [0,0,size,arr]}
      - {target: ^callVkCreateGraphicsPipelines$, tips: [0,0,size,arr,ref,arr]}
      - {target: ^callVkCreateComputePipelines$, tips: [0,0,size,arr,ref,arr]}
      - {target: ^callVkUpdateDescriptorSets$, tips: [0,size,arr,size,arr]}
      - {target: ^callVkAllocateCommandBuffers$, tips: [0,ref,arr]}
      - {target: ^callVkFreeCommandBuffers$, tips: [0,0,size,arr]}
      - {target: ^callVkMapMemory, tips: [0,0,0,0,0,ref]}
      - {target: ^callVkCmdSetViewport$, tips: [0,0,size,arr]}
      - {target: ^callVkCmdSetScissor$, tips: [0,0,size,arr]}
      - {target: ^callVkCmdBindDescriptorSets$, tips: [0,0,0,0,size,arr,size,arr]}
      - {target: ^callVkCmdBindVertexBuffers$, tips: [0,0,size2,arr,arr]}
      - {target: ^callVkCmdCopyBuffer$, tips: [0,0,0,size,arr]}
      - {target: ^callVkCmdCopyImage$, tips: [0,0,0,0,0,size,arr]}
      - {target: ^callVkCmdBlitImage$, tips: [0,0,0,0,0,size,arr]}
      - {target: ^callVkCmdCopyBufferToImage$, tips: [0,0,0,0,size,arr]}
      - {target: ^callVkCmdCopyImageToBuffer$, tips: [0,0,0,0,size,arr]}
      - {target: ^callVkCmdClearColorImage$, tips: [0,0,0,ref,size,arr]}
      - {target: ^callVkCmdClearDepthStencilImage$, tips: [0,0,0,ref,size,arr]}
      - {target: ^callVkCmdClearAttachments$, tips: [0,size,arr,size,arr]}
      - {target: ^callVkCmdResolveImage$, tips: [0,0,0,0,0,size,arr]}
      - {target: ^callVkCmdWaitEvents$, tips: [0,size,arr,0,0,size,arr,size,arr,size,arr]}
      - {target: ^callVkCmdPipelineBarrier$, tips: [0,0,0,0,size,arr,size,arr,size,arr]}
      - {target: ^callVkCmdExecuteCommands$, tips: [0,size,arr]}
      - {target: ^callVkGetPhysicalDeviceSurfaceFormatsKHR$, tips: [0,0,ref,arr]}
      - {target: ^callVkGetPhysicalDeviceSurfacePresentModesKHR$, tips: [0,0,ref,arr]}
      - {target: ^callVkGetSwapchainImagesKHR$, tips: [0,0,ref,arr]}
      - {target: ^callVkGetPhysicalDeviceDisplayPropertiesKHR$, tips: [0,ref,arr]}
      - {target: ^callVkGetPhysicalDeviceDisplayPlanePropertiesKHR$, tips: [0,ref,arr]}
      - {target: ^callVkGetDisplayPlaneSupportedDisplaysKHR$, tips: [0,0,ref,arr]}
      - {target: ^callVkGetDisplayModePropertiesKHR$, tips: [0,0,ref,arr]}
      - {target: ^callVkCreateSharedSwapchainsKHR$, tips: [0,size,arr,ref,ref]}
      # this covers all other cases
      - {target: ^callVk, tips: [sref,sref,sref,sref,sref,sref,sref,sref]}

TypeTips

Specifies the translation tips that can help to distinguish between UODawgSpecial32 and float32 plain type that might be the same in terms of memory and only a developer may know that. Especially useful in cases when there are deeply nested typedefs, to the base plain type is not so easy to exact automatically.

Similar to PtrTips and the tip spec is the same, the only allowed tips are named and plain respectively. By default all types are considered as named.

Example:

  TypeTips:
    function:
      - {target: ^glBindAttribLocation$, tips: [0,0,plain]}
      - {target: ^glGetAttribLocation$, tips: [0,plain]}
      - {target: ^glGetUniformLocation$, tips: [0,plain]}
      - {target: ^glGetProgramInfoLog$, tips: [0,0,0,plain]}
      - {target: ^glShaderSource$, tips: [0,0,plain,0]}
      - {target: ^glGetString$, self: plain}

MemTips

Specifies the translation tips regarding the handling of struct memory and fields. Similar to PtrTips and the tip spec is the same, but the difference is that it does not target any scope, it's just a list that turns off any struct introspection of C types, thus avoiding unsafe conversions and disables bindings generation for a type or a field. Possible tips: raw.

Example:

  MemTips:
    - {target: VkAllocationCallbacks, self: raw}
    - {target: ANativeWindow, self: raw}

And now these two types are just aliases to CGo types and can be handled in "opaque" mode as "raw" values.

// ANativeWindow as declared in android/native_window.h:36
type ANativeWindow C.ANativeWindow

// AllocationCallbacks as declared in https://www.khronos.org/registry/vulkan/specs/1.0/man/html/VkAllocationCallbacks.html
type AllocationCallbacks C.VkAllocationCallbacks

Later, a custom method:

func (w *ANativeWindow) X() {
	win := (*C.ANativeWindow)(w)
	// work with C.ANativeWindow pointer directly, pass it around
}