Luma's Workshop - User contributions [en]

BCSV (File format)

2025-01-11T17:28:23Z

Lord-Giganticus: /* Field Order & Entry Size */

[[Category:File formats]]
{{Finished}}
'''BCSV''' stands for '''B'''inary '''C'''omma '''S'''eparated '''V'''alues and is the most common data format used in both Super Mario Galaxy games. Some older GameCube titles, such as ''Luigi's Mansion'' and ''Donkey Kong Jungle Beat'', use this data format as well. As the name suggests, BCSV is a binary variant of comma-separated values (CSV). This means that the data is laid out in a table-like structure. The column names are hashed for faster access. The data is flatbuffer-like and is loaded directly into memory, meaning that it does not have to be deserialized first. The game supports reading data as signed and unsigned integers (8, 16 and 32 bit), single-precision floats and strings.
All BCSV files are padded to the nearest 32 byte boundary with '@' (0x40). There is no consistent file extension for BCSV data. Instead, the game contains various BCSV, BANMT, BCAM, PA and TBL files. BCSV files that use ''TBL'' as their file extension are expected to be sorted in ascending order by some specific field. Each string is a null-terminated ''SHIFT-JIS (Codepage 932)'' encoded string.

==Header==
Each BCSV file starts with a header:

{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x00 || u32 || Entry count
|-
| 0x04 || u32 || Field count
|-
| 0x08 || u32 || Offset to the entry data section
|-
| 0x0C || u32 || The size of each entry in bytes
|}

==Fields Section==
Right after the header comes the list of fields. The structure of a single field is as follows:

{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x00 || u32 || Name hash
|-
| 0x04 || u32 || Bitmask (often the data type's max value)
|-
| 0x08 || u16 || Offset to the data under this field in an individual entry
|-
| 0x0A || u8 || Data shift amount
|-
| 0x0B || u8 || The type of data that this field uses
|}

===Data types===
Fields may cover one of the following data types:

{| class="wikitable"
! Name !! ID !! Size (in bytes) !! Description
|-
| LONG || 0x00 || 4 || 32-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| STRING || 0x01 || 32 || Embedded string. '''Deprecated'''. Use ''STRING_OFFSET'' instead.
|-
| FLOAT || 0x02 || 4 || Single-precision floating-point value.
|-
| LONG_2 || 0x03 || 4 || 32-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| SHORT || 0x04 || 2 || 16-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| CHAR || 0x05 || 1 || 8-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| STRING_OFFSET || 0x06 || 4 || 32-bit offset into string table.
|}

It's debated that there may be a NULL data type (ID of 7) but there is no way of confirming this info at the time.

===Field Order & Entry Size===
For efficiency and hardware limitations, the field offsets and total entry size are calculated depending on a special ordering of the fields. This only affects the order of the data in an entry and not the order of the fields in this section. When saving, the tool should ensure that the field offsets and total entry size are calculated depending on this order: STRING < FLOAT < LONG < LONG_2 < SHORT < CHAR < STRING_OFFSET. A sample implementation from [https://github.com/SunakazeKun/pyjmap/blob/main/pyjmap/jmap.py#L750 pyjmap can be found on Github] which shows how to calculate these properly.
Another sample would be [https://github.com/Lord-G-INC/libbcsv/blob/main/src/types.rs#L65 from libbcsv] which shows the order of the data types.

==Data Section==
Contains the individual data entries. The structure of their data is specified by the BCSV's fields. Each entry is aligned to four bytes. When saving, the tool should ensure that they are written based off the Field Order. The amount of data entries should be <code>header.entrycount * header.fieldcount</code>.

==String Pool==
Right after the data comes the string pool which contains all strings used within the BCSV. Here are some rules regarding the string pool:
* ALWAYS starts directly after the Data Section.
* String entries are UNIQUE.
* String entries are NUL terminated.
* It should always be possible to find the table's start by using <code>header.entrydataoff + header.entrycount * header.entrysize</code>.

BMG (File Format)

2024-07-10T21:00:11Z

Lord-Giganticus: /* Header */ Fix red link

[[Category:File formats]]

'''BMG''' is a file format used to hold text in SMG1. This was replaced by the [[MSBT]] file format in SMG2. Unlike the messages in SMG2, SMG1 uses a single, region specific BMG file (Region/message.arc/message.bmg) to hold ''ALL'' text.

= Format Specifications =
Below you'll find helpful tables on how the file is structured.

'''THIS IS SPECIFIC TO SMG1/3D All Stars. Other games that use BMG might be different!'''

''All Char[] types are in ASCII encoding.''
== Header ==
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || Char[4] || File magic. "MESG" for SMG1, "GSEM" for 3D All Stars.
|-
| 0x4 || Char[4] || File magic. "bmg1" for SMG1. "1gmb" for 3D All Stars.
|-
| 0x8 || UInt32 || [[BMG_(File_Format)#FLW1|FLW1 Section]] Offset.
|-
| 0x0C || UInt32 || Section Number.
|-
| 0x10 || UInt8 || Unknown. Believed to be a definition of encoding in some games.
|-
| 0x11 || UInt8[15] || Padding.
|}

== Section Base ==
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || Char[4] || Magic. Relates to the Section Type it is.
|-
| 0x4 || UInt32 || Section Size.
|}
== Section Types ==
This is where things get a little weird. Depending on the Magic from the [[BMG (File Format)#Section_Base|Section Base]], it can be one of the following types below.
=== INF1 Section ===
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || UInt16 || Section Number.
|-
| 0x2 || UInt16 || Section Size.
|-
| 0x4 || UInt8[4] || Padding.
|}
=== DAT1 ===
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || UInt8[] || Section Data, determined by Section Size. Contains wchar_t strings.
|}
=== FLW1 ===
{| class="wikitable"
|-
! Offset !! Type !! Description
|-
| 0x0 || UInt16 || Node Number.
|-
| 0x2 || UInt16 || Brach Node Number.
|-
| 0x4 || UInt8[4] || Padding.
|}
=== FLI1 ===
''' Not used/supported in SMG1/3D All Stars.'''

= Tools =
The following tools can handle SMG1's BMG (some may have 3D All Stars support, but most don't.)
* [http://wiki.tockdom.com/wiki/Wiimms_SZS_Tools Wiimms SZS Tools (wbmgt)] (Can convert a BMG to a TXT file. High chance of crashing if you edit with this.)
* [https://github.com/ZyphronG/Tomato Tomato] (Can convert a BMG + the [[BCSV|TBL]] file to a XML file and vise versa. '''Has 3D All Stars Support''')

BMG (File Format)

2024-07-10T20:59:15Z

Lord-Giganticus: /* Section Types */ Document DAT1

[[Category:File formats]]

'''BMG''' is a file format used to hold text in SMG1. This was replaced by the [[MSBT]] file format in SMG2. Unlike the messages in SMG2, SMG1 uses a single, region specific BMG file (Region/message.arc/message.bmg) to hold ''ALL'' text.

= Format Specifications =
Below you'll find helpful tables on how the file is structured.

'''THIS IS SPECIFIC TO SMG1/3D All Stars. Other games that use BMG might be different!'''

''All Char[] types are in ASCII encoding.''
== Header ==
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || Char[4] || File magic. "MESG" for SMG1, "GSEM" for 3D All Stars.
|-
| 0x4 || Char[4] || File magic. "bmg1" for SMG1. "1gmb" for 3D All Stars.
|-
| 0x8 || UInt32 || [[BMG#FLW1|FLW1 Section]] Offset.
|-
| 0x0C || UInt32 || Section Number.
|-
| 0x10 || UInt8 || Unknown. Believed to be a definition of encoding in some games.
|-
| 0x11 || UInt8[15] || Padding.
|}

== Section Base ==
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || Char[4] || Magic. Relates to the Section Type it is.
|-
| 0x4 || UInt32 || Section Size.
|}
== Section Types ==
This is where things get a little weird. Depending on the Magic from the [[BMG (File Format)#Section_Base|Section Base]], it can be one of the following types below.
=== INF1 Section ===
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || UInt16 || Section Number.
|-
| 0x2 || UInt16 || Section Size.
|-
| 0x4 || UInt8[4] || Padding.
|}
=== DAT1 ===
{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x0 || UInt8[] || Section Data, determined by Section Size. Contains wchar_t strings.
|}
=== FLW1 ===
{| class="wikitable"
|-
! Offset !! Type !! Description
|-
| 0x0 || UInt16 || Node Number.
|-
| 0x2 || UInt16 || Brach Node Number.
|-
| 0x4 || UInt8[4] || Padding.
|}
=== FLI1 ===
''' Not used/supported in SMG1/3D All Stars.'''

= Tools =
The following tools can handle SMG1's BMG (some may have 3D All Stars support, but most don't.)
* [http://wiki.tockdom.com/wiki/Wiimms_SZS_Tools Wiimms SZS Tools (wbmgt)] (Can convert a BMG to a TXT file. High chance of crashing if you edit with this.)
* [https://github.com/ZyphronG/Tomato Tomato] (Can convert a BMG + the [[BCSV|TBL]] file to a XML file and vise versa. '''Has 3D All Stars Support''')

BCSV (File format)

2024-04-04T20:11:36Z

Lord-Giganticus: update this a bit sense it was a tad old

[[Category:File formats]]
{{Finished}}
'''BCSV''' stands for '''B'''inary '''C'''omma '''S'''eparated '''V'''alues and is the most common data format used in both Super Mario Galaxy games. Some older GameCube titles, such as ''Luigi's Mansion'' and ''Donkey Kong Jungle Beat'', use this data format as well. As the name suggests, BCSV is a binary variant of comma-separated values (CSV). This means that the data is laid out in a table-like structure. The column names are hashed for faster access. The data is flatbuffer-like and is loaded directly into memory, meaning that it does not have to be deserialized first. The game supports reading data as signed and unsigned integers (8, 16 and 32 bit), single-precision floats and strings.
All BCSV files are padded to the nearest 32 byte boundary with '@' (0x40). There is no consistent file extension for BCSV data. Instead, the game contains various BCSV, BANMT, BCAM, PA and TBL files. BCSV files that use ''TBL'' as their file extension are expected to be sorted in ascending order by some specific field. Each string is a null-terminated ''SHIFT-JIS (Codepage 932)'' encoded string.

==Header==
Each BCSV file starts with a header:

{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x00 || u32 || Entry count
|-
| 0x04 || u32 || Field count
|-
| 0x08 || u32 || Offset to the entry data section
|-
| 0x0C || u32 || The size of each entry in bytes
|}

==Fields Section==
Right after the header comes the list of fields. The structure of a single field is as follows:

{| class="wikitable"
! Offset !! Type !! Description
|-
| 0x00 || u32 || Name hash
|-
| 0x04 || u32 || Bitmask (often the data type's max value)
|-
| 0x08 || u16 || Offset to the data under this field in an individual entry
|-
| 0x0A || u8 || Data shift amount
|-
| 0x0B || u8 || The type of data that this field uses
|}

===Data types===
Fields may cover one of the following data types:

{| class="wikitable"
! Name !! ID !! Size (in bytes) !! Description
|-
| LONG || 0x00 || 4 || 32-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| STRING || 0x01 || 32 || Embedded string. '''Deprecated'''. Use ''STRING_OFFSET'' instead.
|-
| FLOAT || 0x02 || 4 || Single-precision floating-point value.
|-
| LONG_2 || 0x03 || 4 || 32-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| SHORT || 0x04 || 2 || 16-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| CHAR || 0x05 || 1 || 8-bit integer. Signedness is not specified. ANDed with the bitmask and shifted right by the field's shift amount.
|-
| STRING_OFFSET || 0x06 || 4 || 32-bit offset into string table.
|}

It's debated that there may be a NULL data type (ID of 7) but there is no way of confirming this info at the time.

===Field Order & Entry Size===
For efficiency and hardware limitations, the field offsets and total entry size are calculated depending on a special ordering of the fields. This only affects the order of the data in an entry and not the order of the fields in this section. When saving, the tool should ensure that the field offsets and total entry size are calculated depending on this order: STRING < FLOAT < LONG < LONG_2 < SHORT < CHAR < STRING_OFFSET. A sample implementation from [https://github.com/SunakazeKun/pyjmap/blob/main/pyjmap/jmap.py#L750 pyjmap can be found on Github] which shows how to calculate these properly.
Another sample would be [https://github.com/Lord-G-INC/libbcsv/blob/main/src/types.rs#L56 from libbcsv] which shows the order of the data types.

==Data Section==
Contains the individual data entries. The structure of their data is specified by the BCSV's fields. Each entry is aligned to four bytes. When saving, the tool should ensure that they are written based off the Field Order. The amount of data entries should be <code>header.entrycount * header.fieldcount</code>.

==String Pool==
Right after the data comes the string pool which contains all strings used within the BCSV. Here are some rules regarding the string pool:
* ALWAYS starts directly after the Data Section.
* String entries are UNIQUE.
* String entries are NUL terminated.
* It should always be possible to find the table's start by using <code>header.entrydataoff + header.entrycount * header.entrysize</code>.